Cell-by-cell testing for production Jupyter notebooks in JupyterLab

jpmorganchase, updated 🕥 2023-03-14 15:38:37

Overview

nbcelltests is designed for writing tests for linearly executed notebooks. Its primary use is for unit testing reports.

Installation

Python package installation: pip install nbcelltests

To use in JupyterLab, you will also need the lab and server extensions. Typically, these are automatically installed alongside nbcelltests, so you should not need to do anything special to use them. The lab extension will require a rebuild of JupyterLab, which you'll be prompted to do on starting JupyterLab the first time after installing celltests (or you can do manually with jupyter lab build). Note that you must have node.js installed (as for any lab extension).

To see what extensions you have, check the output of jupyter labextension list (look for jupyterlab_celltests), and jupyter serverextension list (look for nbcelltests). If for some reason you need to manually install the extensions, you can do so as follows:

bash jupyter labextension install jupyterlab_celltests jupyter serverextension enable --py nbcelltests

(Note: if using in an environment, you might wish to add --sys-prefix to the serverextension command.)

"Linearly executed notebooks?"

When converting notebooks into html/pdf/email reports, they are executed top-to-bottom one time, and are expected to contain as little code as reasonably possible, focusing primarily on the plotting and markdown bits. Libraries for this type of thing include Papermill, JupyterLab Emails, etc.

Doesn't this already exist?

Nbval is a great product (we leverage it in this project) and I recommend using it for notebook regression tests. But it only allows for testing for unexpected failures or simple output equality tests.

So why do I want this again?

This doesn't necessarily help you if your data sources go down, but its likely you'll notice this anyway. Where this comes in handy is:

when the environment (e.g. package versions) are changing in your system
when you play around in the notebook (e.g. nonlinear execution) but aren't sure if your reports will still generate
when your software lifecycle systems have a hard time dealing with notebooks (can't lint/audit them as code unless integrated nbdime/nbconvert to script, tough to test, tough to ensure what works today works tomorrow)

So what does this do?

Given a notebook, you can write mocks and assertions for individual cells. You can then generate a testing script for this notebook, allowing you to hook it into your testing system and thereby provide unittests of your report.

Writing tests

When you write tests for a cell, we create a new method on a unittest class corresponding to the index of your cell, and including the cumulative tests for all previous cells (to mimic what has happened so far in the notebook's linear execution). You can write whatever mocking and asserts you like, and can call %cell to inject the contents of the cell into your test. The tests themselves are stored in the cell metadata, similar to celltags, slide information, etc.

Running tests

You can run the tests offline from an .ipynb file, or you can execute them from the browser and view the results of pytest-html's html plugin.

Extra Tests

Max number of lines per cell
Max number of cells per notebook
Max number of function definitions per notebook
Max number of class definitions per notebook
Percentage of cells tested

Example

In the committed examples/Example.ipynb notebook, but modified so that cell 0 has its import statement copied 10 times (to trigger test and lint failures):

Tests

The following output is generated by running nbcelltests test examples/Example.ipynb bash examples/_Example_test.py::TestNotebook::test_cell_coverage PASSED [ 20%] examples/_Example_test.py::TestNotebook::test_code_cell_1 PASSED [ 40%] examples/_Example_test.py::TestNotebook::test_code_cell_2 PASSED [ 60%] examples/_Example_test.py::TestNotebook::test_code_cell_3 PASSED [ 80%] examples/_Example_test.py::TestNotebook::test_code_cell_4 PASSED [100%]

Lint

The following output is generated by running nbcelltests lint examples/Example.ipynb

bash PASSED: Checking lines in cell (max=10; actual=2) (Cell 1) PASSED: Checking lines in cell (max=10; actual=1) (Cell 2) PASSED: Checking lines in cell (max=10; actual=1) (Cell 3) PASSED: Checking lines in cell (max=10; actual=1) (Cell 4) PASSED: Checking cells per notebook (max=10; actual=4) PASSED: Checking functions per notebook (max=10; actual=0) PASSED: Checking classes per notebook (max=10; actual=0) FAILED: Checking lint: examples/Example.ipynb (in /var/folders/s3/1mjw0y192zg3450tkkn1yfnm0000gn/T/tmpp91li59p.py):32:1: F821 undefined name 'test3' examples/Example.ipynb (in /var/folders/s3/1mjw0y192zg3450tkkn1yfnm0000gn/T/tmpp91li59p.py):32:6: W291 trailing whitespace

NB: In jupyterlab, notebooks will be lint checked in-process using the version of python that is running jupyter lab itself. A notebook intended to be run with a Python 2 kernel could therefore generate syntax errors during lint checking.

Development

See CONTRIBUTING.md for guidelines.

License

This software is licensed under the Apache 2.0 license. See the LICENSE and AUTHORS files for details.

Issues

Bump webpack from 5.28.0 to 5.76.1 in /js

opened on 2023-03-14 15:38:36 by dependabot[bot]

Bumps webpack from 5.28.0 to 5.76.1.

Release notes

Sourced from webpack's releases.

v5.76.1

Fixed

Added assert/strict built-in to NodeTargetPlugin

Revert

Improve performance of hashRegExp lookup by @ryanwilsonperkin in webpack/webpack#16759

v5.76.0

Bugfixes

Avoid cross-realm object access by @Jack-Works in webpack/webpack#16500

Improve hash performance via conditional initialization by @lvivski in webpack/webpack#16491

Serialize generatedCode info to fix bug in asset module cache restoration by @ryanwilsonperkin in webpack/webpack#16703

Improve performance of hashRegExp lookup by @ryanwilsonperkin in webpack/webpack#16759

Features

add target to LoaderContext type by @askoufis in webpack/webpack#16781

Security

CVE-2022-37603 fixed by @akhilgkrishnan in webpack/webpack#16446

Repo Changes

Fix HTML5 logo in README by @jakebailey in webpack/webpack#16614

Replace TypeScript logo in README by @jakebailey in webpack/webpack#16613

Update actions/cache dependencies by @piwysocki in webpack/webpack#16493

New Contributors

@Jack-Works made their first contribution in webpack/webpack#16500

@lvivski made their first contribution in webpack/webpack#16491

@jakebailey made their first contribution in webpack/webpack#16614

@akhilgkrishnan made their first contribution in webpack/webpack#16446

@ryanwilsonperkin made their first contribution in webpack/webpack#16703

@piwysocki made their first contribution in webpack/webpack#16493

@askoufis made their first contribution in webpack/webpack#16781

Full Changelog: https://github.com/webpack/webpack/compare/v5.75.0...v5.76.0

v5.75.0

Bugfixes

experiments.* normalize to false when opt-out

avoid NaN%

show the correct error when using a conflicting chunk name in code

HMR code tests existance of window before trying to access it

fix eval-nosources-* actually exclude sources

fix race condition where no module is returned from processing module

fix position of standalong semicolon in runtime code

Features

add support for @import to extenal CSS when using experimental CSS in node

... (truncated)

Commits

21be52b Merge pull request #16804 from webpack/chore-patch-release
1cce945 chore(release): 5.76.1
e76ad9e Merge pull request #16803 from ryanwilsonperkin/revert-16759-real-content-has...
52b1b0e Revert "Improve performance of hashRegExp lookup"
c989143 Merge pull request #16766 from piranna/patch-1
710eaf4 Merge pull request #16789 from dmichon-msft/contenthash-hashsalt
5d64468 Merge pull request #16792 from webpack/update-version
67af5ec chore(release): 5.76.0
97b1718 Merge pull request #16781 from askoufis/loader-context-target-type
b84efe6 Merge pull request #16759 from ryanwilsonperkin/real-content-hash-regex-perf
Additional commits viewable in compare view

Maintainer changes

This version was pushed to npm by evilebottnawi, a new releaser for webpack since your current version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/jpmorganchase/nbcelltests/network/alerts).

Bump http-cache-semantics from 4.1.0 to 4.1.1 in /js

opened on 2023-02-03 21:56:33 by dependabot[bot]

Bumps http-cache-semantics from 4.1.0 to 4.1.1.

Commits

2449650 Update mocha
560b2d8 Don't use regex to trim whitespace
b1bdb92 Remove linting package zoo
c20dc7e Cache 308
See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

JupyterLab 3.0+ Support

opened on 2022-04-29 15:01:11 by traviscancode604

Is your feature request related to a problem? Please describe. We are running JupyterLab 3.2.8 and cannot use nbcelltests because of an extension installation error related to jupyterlab_celltests.

Error message: The extension "jupyterlab_celltests" does not yet support the current version of JupyterLab.

Describe the solution you'd like Would like to be able to run pip install nbcelltests per the documentation and have the package work for JupyterLab 3.2.8.

Describe alternatives you've considered Alternative of downgrading JupyterLab version is not feasible.

Additional context Related to https://github.com/jpmorganchase/nbcelltests/issues/215

``` Conflicting Dependencies: JupyterLab Extension Package

=3.2.8 <3.3.0 >=2.0.0 <3.0.0 @jupyterlab/application =3.2.8 <3.3.0 >=2.0.0 <3.0.0 @jupyterlab/apputils =3.2.8 <3.3.0 >=2.0.0 <3.0.0 @jupyterlab/codeeditor =3.2.8 <3.3.0 >=2.0.0 <3.0.0 @jupyterlab/codemirror =5.2.8 <5.3.0 >=4.0.0 <5.0.0 @jupyterlab/coreutils =3.2.8 <3.3.0 >=2.0.0 <3.0.0 @jupyterlab/docmanager =3.2.8 <3.3.0 >=2.0.0 <3.0.0 @jupyterlab/filebrowser =3.2.8 <3.3.0 >=2.0.0 <3.0.0 @jupyterlab/launcher =3.2.8 <3.3.0 >=2.0.0 <3.0.0 @jupyterlab/mainmenu =3.2.8 <3.3.0 >=2.0.0 <3.0.0 @jupyterlab/notebook ```

Add support for prebuilt extension

opened on 2021-03-28 02:15:56 by timkpaine None

Increase default timeout for running cell tests

opened on 2020-11-05 17:57:01 by ceball

Currently, once you click to run cell tests in jupyter lab, the result needs to come back within 5 seconds or it'll be cancelled.

But even if all heavy parts of a notebook are mocked out, many a notebook's celltests will not complete in 5 seconds.

We should probably up the default to 15 seconds or so.

https://github.com/jpmorganchase/nbcelltests/blob/fa86221569181e9ebdac9322cdd6dcc64b833923/js/src/run.ts#L15

https://github.com/jpmorganchase/nbcelltests/blob/fa86221569181e9ebdac9322cdd6dcc64b833923/js/src/run.ts#L35

https://github.com/timkpaine/requests-helper/blob/master/src/index.ts#L21

That should be easy.

We could also consider making that configurable.

However, it might be better to focus on improving how celltests are launched: #210

Improve launching and reporting of tests and lint

opened on 2020-11-05 17:56:28 by ceball

Placeholder: fill in some content about a nice design that fits in with jupyter lab :) Are there other extensions we could be inspired by/steal from?

Currently: * You launch a test run or lint check from the Commands box (which seems fine). * You get a modal dialog asking if you want to run. * After clicking OK, you get no feedback until the result appears. * If the tests or lint don't complete within 5 seconds, you get an error (#211). * If the tests or lint do complete within 5 seconds, you get an html report in a modal dialog.

Releases

Release v0.2.3 2020-10-05 14:16:09

Add auth
Fixup CLI w/ argparse
UI Overhaul, lots of changes and improvements
Omit raw cells before running external linter
Find noqa comments in empty ast code cells
Some README tweaks and new logo
Fixes for forked testing

Release v0.2.2 2020-06-17 06:40:53

Minor change from v0.2.1: support running nbcelltests.lint on python 2.7

Release v0.2.1 2020-06-15 19:52:52

See the 0.2.1 milestone for a list of changes included in this release.

Release v0.2.0 2020-05-28 13:36:37

See the 0.2 milestone for a list of changes.

Release v0.1.3 2020-03-09 15:35:24

Release v0.1.2 2019-12-10 18:38:47

Test and lint object #27
Linter cleanup
Packaging cleanups
Py3.6+
Windows path fixes
Better parsing (uses AST rather than string parsing)
Autopep
Kernel checks

J.P. Morgan Chase & Co.

Welcome to the open source project repositories for JPMorgan Chase

GitHub Repository

jupyter jupyterlab jupyterlab-extension testing jupyter-notebook jupyter-notebooks papermill pytest unittesting jpmorganchase