Python pipeline utility library

zendesk, updated 🕥 2022-12-09 05:19:10

PAKKR

Description

In the process of building machine learning things at Zendesk, we have noticed that a lot of the steps are sequential where later steps rely on outputs of previous steps. Because Python functions only return a single value (return with multiple values are returned as a tuple), deconstructing and keeping track of return values becomes tedious for long sequences of steps, especially when inputs are not returned from the immediately previous step.

PAKKR is an utility created to remediate these pain points; it provides the user with a way to specify how return values should be interpreted and optionally caches results and injects them in later steps automatically.

Install from PyPi

bash pip install pakkr

Install from source

bash git clone [email protected]:zendesk/pakkr.git cd pakkr python setup.py install

Usage

```python from pakkr import Pipeline, returns

@returns(int, original_num_as_string=str) # this function returns an integer and insert original_num_as_string into the meta cache def times_two(n): return n*2, {'original_num_as_string': str(n)}

@returns(int, int) # this functions returns two integers and will be passed on as two arguments def plus_five_and_three(n): return n + 5, n + 3

@returns(str) def summary(a, b, original_num_as_string): # a and b are passed in as positional arguments, # but original_num_as_string would be injected from the meta cache return f'Original input was {original_num_as_string} and it became {str(a)} and {str(b)} after processing'

pipeline = Pipeline(times_two, plus_five_and_three, summary, _name='process_int') print(pipeline(3)) Running the above code should print: Original input was 3 and it became 11 and 9 after processing ```

What's going on?

returns is used to indicate how the return values should be interpreted; @returns(int, str, x=bool) means the Callable should be returning something like return 10, 'hello', {'x': True} and the 10 and 'hello' will be passed as two positional arguments into the next Callable while x would be cached in the meta space and be injected if any following Callables require x but not being given as positional argument from the previous Callable.

Development

This project uses tox to manage testing on multiple Python versions assuming the required Python versions are available. git clone [email protected]:zendesk/pakkr.git cd pakkr pip install tox tox

Optionally, uses pyenv and pipenv to manage Python installation and development dependencies. ```bash git clone [email protected]:zendesk/pakkr.git cd pakkr

Install pyenv, see instructions in https://github.com/pyenv/pyenv

Install Python versions supported by pakkr if not available locally

pyenv install 3.6.10

pyenv install 3.7.6

pyenv install 3.8.1

Set available Python verions

pyenv local 3.6.10 3.7.6 3.8.1

Install pipenv

pip install pipenv pipenv sync --dev

Run tests

pipenv run tox ```

Reporting Bugs

Please raise an issue via GitHub.

Contributing

Improvements are always welcome. Please follow these steps to contribute

Submit a Pull Request with a detailed explanation of changes
Receive approval from maintainers
Maintainers will merge your changes

License

Use of this software is subject to important terms and conditions as set forth in the LICENSE file.

Issues

Bump certifi from 2021.5.30 to 2022.12.7

opened on 2022-12-09 05:19:10 by dependabot[bot]

Bumps certifi from 2021.5.30 to 2022.12.7.

Commits

9e9e840 2022.12.07
b81bdb2 2022.09.24
939a28f 2022.09.14
aca828a 2022.06.15.2
de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...
b8eb5e9 2022.06.15.1
47fb7ab Fix deprecation warning on Python 3.11 (#199)
b0b48e0 fixes #198 -- update link in license
9d514b4 2022.06.15
4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/zendesk/pakkr/network/alerts).

Bump ipython from 7.25.0 to 7.31.1

opened on 2022-01-21 19:12:02 by dependabot[bot]

Bumps ipython from 7.25.0 to 7.31.1.

Commits

e321e76 release 7.31.1
67ca2b3 Merge pull request from GHSA-pq7m-3gw7-gq5x
2794330 back to dev
be343e7 release 7.31.0
0fcf2c4 Merge pull request #13428 from meeseeksmachine/auto-backport-of-pr-13427-on-7.x
b8db9b1 Backport PR #13427: wn 731
7f253dc Merge pull request #13412 from bnavigator/backport-inspect
4f26796 fix xxlimited_35 import name
77ca4a6 don't run nose-based iptest on py310, only pytest
533e509 back to decorator skip
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Supporting return type annotation

opened on 2019-07-18 03:12:08 by ePak

Rather than adding an attribute onto the function/class being decorated, it would be great if we could specify the return type, i.e. @returns(...) directly as type annotation like def do_something(x: int) -> returns(int, some_meta=str) etc.

Releases

Version 0.1.0 2020-01-10 05:29:18

Initial release to PyPi

Zendesk

GitHub Repository