Async IPython Magic for Asynchronous Notebook Cell Execution

leriomaggio, updated 🕥 2022-08-23 17:10:28

%%async_run: an IPython notebook magic for asynchronous cell execution

Description (Talk Abstract @EuroScipy 2016)

The IPython notebook project is one the preferred tools of data scientists, and it is nowadays the bastion for Reproducible Research.

In fact, notebooks are now used as in-browser IDE (Integrated Development Environment) to implement the whole data analysis process, along with the corresponding documentation.

However, since this kind of processes usually include heavy-weight computations, it may likely happen that execution results get lost if something wrong happens, e.g. the connection to the notebook server hangs or an accidental page refresh is issued.

To this end, [%]%async_run notebook line/cell magic to the rescue.

In this talk, I would like to talk about some of the technologies I played with since I decided to develop this extension. These technologies include asynchronous I/O libraries (e.g. asyncio, tornado.websocket), multiprocessing, along with IPython kernels and notebooks.

During the talk, I would like to discuss pitfalls, failures, and adopted solutions (e.g. namespace management among processes) , aiming at getting as many feedbacks as possible from the community.

A general introduction to the actual state-of-the-art of the Jupyter projects (an libraries) will be presented as well, in order to help those who are willing to know some more details about the internals of IPython.

Slides

Slides of the talk are available on my SpeakerDeck profile.

Enabling the Magic(s)

Enabling the magic is simple as copying files into a directory. Open the terminal and:

cp -R async_run_ipython_magic.py run_async/ ~/.ipython/profile_default/startup/

After that, the magic will be enabled by default at the startup of each Jupyter/IPython sessions.

Requirements

The only two main requirements for this Magic are notebook and tornado (which will be indeed installed by the jupyter notebook itself). Moreover, the psutil module is needed.

To ease the installation of all the requirements, a requirements.txt file is provided in the repo for pip installing:

pip install -f requirements.txt

Python 2 Users

So far, this magic works only with Python 3. For example, it relies on the concurrent.futures module to allow for the multiprocessing execution.

This module is only available in Python 3 standard library. For Python 2, you have to pip install the futures

A couple of fix could be included in the code to easily support Python 2 as well, leveraging on the six module. These fixes will be soon included in the code base.

Usage

Three [%]%async_* magics are provided within this package:

  • %async_run_server : Spawns the AsyncRunServer process, which is in charge of handling the async cell execution inside a Tornado WebApplication and IOLoop.

  • %async_stop_server : Stops the AsyncRunServer running process, if any.

  • [%]%async_run : Line/Cell Magic to asynchronously execute the content of the line/cell, respectively.

Examples

Please, check out the examples folder for examples and hints for usage (so far, very few examples available. More to come!)

Note:

If you want to run the server in a terminal and get the log output, move to the startup folder and execute:

  • python -m run_async.run_server

Issues

Any chance you can make this work modern tornado (>=6.2)

opened on 2023-02-18 22:18:05 by JohnOmernik

I stumbled across this and I think this could be a huge step forward for many things in Jupyter. Specifically the framework to allow OTHER magic functions to operate with async io and do things like submit queries to remote servers without blocking. I want to get into understanding things, but it appears some of the ways you call IOloop are deprecated. Any chance you can take a look at things and it working with Tornado 6.2.

From there I want to start learning how things work and I will obviously share back what I do! Thank

Bump nbconvert from 4.2.0 to 6.5.1

opened on 2022-08-23 17:10:25 by dependabot[bot]

Bumps nbconvert from 4.2.0 to 6.5.1.

Release notes

Sourced from nbconvert's releases.

Release 6.5.1

No release notes provided.

6.5.0

What's Changed

New Contributors

Full Changelog: https://github.com/jupyter/nbconvert/compare/6.4.5...6.5

6.4.3

What's Changed

New Contributors

Full Changelog: https://github.com/jupyter/nbconvert/compare/6.4.2...6.4.3

6.4.0

What's Changed

New Contributors

... (truncated)

Commits


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/leriomaggio/async-ipython-magic/network/alerts).

Bump notebook from 6.1.5 to 6.4.12

opened on 2022-06-16 23:12:40 by dependabot[bot]

Bumps notebook from 6.1.5 to 6.4.12.

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/leriomaggio/async-ipython-magic/network/alerts).

Bump ipython from 5.1.0 to 7.16.3

opened on 2022-01-21 19:28:37 by dependabot[bot]

Bumps ipython from 5.1.0 to 7.16.3.

Commits


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/leriomaggio/async-ipython-magic/network/alerts).

UnicodeDecodeError on %%async_run

opened on 2017-04-24 09:09:39 by bofm

Hi,

I tried to play with this and got the error. The notebook is the one from the examples.

After executing %%async_run N = 10**s print(N) the error appeared in the console.

ERROR:tornado.application:Uncaught exception in /ws/PYTHON---9c4c6b81-49b3-4977-9093-b04f5bbf3328 Traceback (most recent call last): File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tornado/websocket.py", line 412, in _run_callback callback(*args, **kwargs) File "/Users/sergey/.ipython/profile_default/startup/run_async/run_server.py", line 181, in on_message data = json.loads(message) File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 349, in loads s = s.decode(detect_encoding(s), 'surrogatepass') UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

I used Jupyter Notebook 5.0.0 on Python 3.6 on MacOS

Valerio Maggio

Developer Advocate @anaconda - Data Scientist and Researcher

GitHub Repository