Tools to run Jupyter notebooks as jobs in Amazon SageMaker - ad hoc, on a schedule, or in response to events

aws-samples, updated 🕥 2023-03-15 06:12:19

SageMaker Run Notebook

Note: sagemaker_run_notebook is experimental software designed for trial use. It may change significantly in the future and there is no guarantee of support. Please do use it and give us feedback on what we could improve, but take its experimental nature into account.

This is a library and a JupyterLab extension that lets you run your Jupyter Notebooks in AWS using SageMaker processing jobs. Notebooks can be run on a schedule, triggered by an event, or called ad hoc. Notebooks are executed using papermill which allows you to specify parameters for each notebook run.

In addition to running notebooks, the library has tools to visualize runs and download the output notebooks. Notebooks can mark output data using scrapbook and that data can be retrieved from a single run or across several runs.

Getting Started

There are several ways to use the tools provided here, each with a slightly different set up.

We provide a convenience library to configure your infrastructure, build executable environments, and execute notebooks. With this library, you have three ways to run, schedule, and monitor notebook execution:

You can perform operations from the shell using a command-line interface designed explicitly for running notebooks (e.g., $ run-notebook run weather.ipynb -p place="Seattle, WA").
You can perform operations from a Jupyter notebook or Python program using a special Python library (e.g., run.invoke(notebook="weather.ipynb", parameters={"place": "Seattle, WA"}))
You can use the JupyterLab extension to run, schedule, and monitor notebooks interactively in any JupyterLab environment (including SageMaker Studio and SageMaker notebook instances)

To install and configure these tools, see the Quick Start.

Alternatively, you can create the infrastructure you need with the provided CloudFormation template and then use standard AWS APIs to schedule, run, and monitor your notebooks. This is a good route when you don't want to add unsupported dependencies or you want to perform these actions from a language other than Python. For instructions on this path, see Build Your Own Notebook Execution Environment.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.

Issues

Bump webpack from 5.74.0 to 5.76.1 in /labextension

opened on 2023-03-15 06:12:18 by dependabot[bot]

Bumps webpack from 5.74.0 to 5.76.1.

Release notes

Sourced from webpack's releases.

v5.76.1

Fixed

Added assert/strict built-in to NodeTargetPlugin

Revert

Improve performance of hashRegExp lookup by @ryanwilsonperkin in webpack/webpack#16759

v5.76.0

Bugfixes

Avoid cross-realm object access by @Jack-Works in webpack/webpack#16500

Improve hash performance via conditional initialization by @lvivski in webpack/webpack#16491

Serialize generatedCode info to fix bug in asset module cache restoration by @ryanwilsonperkin in webpack/webpack#16703

Improve performance of hashRegExp lookup by @ryanwilsonperkin in webpack/webpack#16759

Features

add target to LoaderContext type by @askoufis in webpack/webpack#16781

Security

CVE-2022-37603 fixed by @akhilgkrishnan in webpack/webpack#16446

Repo Changes

Fix HTML5 logo in README by @jakebailey in webpack/webpack#16614

Replace TypeScript logo in README by @jakebailey in webpack/webpack#16613

Update actions/cache dependencies by @piwysocki in webpack/webpack#16493

New Contributors

@Jack-Works made their first contribution in webpack/webpack#16500

@lvivski made their first contribution in webpack/webpack#16491

@jakebailey made their first contribution in webpack/webpack#16614

@akhilgkrishnan made their first contribution in webpack/webpack#16446

@ryanwilsonperkin made their first contribution in webpack/webpack#16703

@piwysocki made their first contribution in webpack/webpack#16493

@askoufis made their first contribution in webpack/webpack#16781

Full Changelog: https://github.com/webpack/webpack/compare/v5.75.0...v5.76.0

v5.75.0

Bugfixes

experiments.* normalize to false when opt-out

avoid NaN%

show the correct error when using a conflicting chunk name in code

HMR code tests existance of window before trying to access it

fix eval-nosources-* actually exclude sources

fix race condition where no module is returned from processing module

fix position of standalong semicolon in runtime code

Features

add support for @import to extenal CSS when using experimental CSS in node

... (truncated)

Commits

21be52b Merge pull request #16804 from webpack/chore-patch-release
1cce945 chore(release): 5.76.1
e76ad9e Merge pull request #16803 from ryanwilsonperkin/revert-16759-real-content-has...
52b1b0e Revert "Improve performance of hashRegExp lookup"
c989143 Merge pull request #16766 from piranna/patch-1
710eaf4 Merge pull request #16789 from dmichon-msft/contenthash-hashsalt
5d64468 Merge pull request #16792 from webpack/update-version
67af5ec chore(release): 5.76.0
97b1718 Merge pull request #16781 from askoufis/loader-context-target-type
b84efe6 Merge pull request #16759 from ryanwilsonperkin/real-content-hash-regex-perf
Additional commits viewable in compare view

Maintainer changes

This version was pushed to npm by evilebottnawi, a new releaser for webpack since your current version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/aws-samples/sagemaker-run-notebook/network/alerts).

Bump http-cache-semantics from 4.1.0 to 4.1.1 in /labextension

opened on 2023-02-04 17:52:26 by dependabot[bot]

Bumps http-cache-semantics from 4.1.0 to 4.1.1.

Commits

2449650 Update mocha
560b2d8 Don't use regex to trim whitespace
b1bdb92 Remove linting package zoo
c20dc7e Cache 308
See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Enable and disable Rules

opened on 2023-01-30 12:58:26 by yosefbs

Issue #, if available:

Description of changes: 1. make schedule state clickable and allow user to enable and disable rule:

add china support in build and push docker script
fix hard-coded value in load_params method (handlers.py)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Is there a way to monitor output during code running?

opened on 2023-01-28 00:45:36 by susuleung

Can we only see outputs from S3 bucket after code completely run. Is there a way to monitor and watch outputs while the code is running?

Lifecycle configuration failure

opened on 2023-01-23 16:00:33 by jeremyallenjacobson

Created a new domain SageMaker Studio domain today. I can install the run-notebook extension in studio if I run the commands of the install-run-notebook.sh script myself in a studio lab terminal.

However, when I add that script as a Lifecycle configuration following instructions provided in this repo I can't start studio as it keeps cycling back to a "restart server" screen. When I do click to restart on that screen, a couple of minutes pass and it comes back to the same screen. I had colleagues test it as well and it was the same for them.

UPDATE: Wanted to update this post to say I just noticed in my new SageMaker Domain the new "Notebook Jobs" feature which seems to integrate sageMaker-run-notebook directly into the SageMaker Studio UI. I tested it and it accomplishes what I was using the library for. Therefore this issue is no longer of high importance to me, but I did want to raise the issue in case someone else is spending time on this.

Also, thank you for making this awesome Jupyter Extension and tooling! I was raving to people about it last year, so couldn't be more pleased to see it is now integrated into SageMaker.

the scheduled job failed when the scheduled notebook include modules from same folder

opened on 2023-01-10 03:23:32 by sunnydgm

The Scheduled job failed to produce results if the scheduled notebook need to import modules from same folder.

Folder 1 scheduled_notebook module1.py module2.py

if my scheduled_notebook needs to read modlue1 and module2 from the same folder, the run will fault with 'Execution using papermill encountered an exception here and stopped: ModuleNotFoundError:.."

Releases

v0.23.0 2022-11-10 20:30:43

Fixes the lifecycle scripts for SageMaker Studio and SageMaker Notebook Instances to point to the right release artifact.

v0.22.0 2022-10-22 00:46:36

JupyterLab 3.x support

The GUI plugin is now updated to support JupyterLab 3.x and, therefore, supports the current versions of SageMaker Notebook Instances and SageMaker Studio notebooks.

As of this release, the JupyterLab extension is no longer compatible with JupyterLab 2.x. If you're using JupyterLab 2.x, please use release v0.21.0.

Additional small features

Add a real lifecycle script for SageMaker Studio notebooks so you can install the plugin permanently in your domain.
Support for China regions (thanks to @yosefbs).
Update default container environment to Python 3.10.

v0.21.0 2022-10-05 02:51:25

This release is just security patches in the dependencies.

v0.20.0 2021-12-10 03:54:36

This release provides support for JupyterLab 2.x.

As of this release, the JupyterLab extension is no longer compatible with JupyterLab 1.x. If you're using JupyterLab 1.x, please use release v0.19.0.

v0.19.0 2021-12-08 07:14:08

This release has a number of small improvements and bug fixes.

After upgrading to this release, we recommend that you delete and recreate your infrastructure with the following commands:

aws cloudformation delete-stack --stack-name sagemaker-run-notebook
run-notebook create-infrastructure

This will pick up the new managed policies (see under [Bug Fixes])(#bug-fixes) below) which can not be updated with the --update option.

Bug fixes

Fixed an install failure on versions >= 3.10. (Fixes issue #35)
Use managed policies so that we can include the policies in other roles (See Change policies to managed policies by @dmoser04).
Use a single permission statement on the Lambda function for all EventBridge rules. This prevents us overflowing the number of separate permissions when we have many scheduled notebook runs. This also means that we're not creating permissions at schedule time. (Fixes issue #9)
Correctly handle scheduled runs with no supplied parameters. (Fixes issue #25)
Update various JS dependencies for security fixes.

v0.18.0 2020-11-06 06:30:02

This is a documentation only release:

It adds the documentation tarball (docs.tar.gz) to the release.
It adds reasonable docstrings to all the exported functions in sagemaker_run_notebook.

AWS Samples

GitHub Repository

jupyter jupyterlab jupyterlab-extension aws sagemaker cloudformation