Build a Python Jupyter Notebook Server with Docker & Heroku.

codingforentrepreneurs, updated 🕥 2023-01-17 04:58:50

Original Post

Jupyter x Docker on Heroku Post

Jupyter is a tool for running interactive notebooks; basically add Python with Markdown and you've got Jupyter. if you haven't used it before, I recommend you do.

In this post, I'm going to show you how to deploy a Jupyter Notebook server on Heroku using Docker.

The big caveat

Jupyter has the ability to create new notebooks and they will 100% save on your deployed docker-based Jupyter server... but they will disappear as soon as you deploy a new version. That's because containers, by their very nature, are ephemeral by default.

This caveat doesn't mean we shouldn't do this... it just means it is a HUGE consideration when using this guide over something like http://colab.research.google.com.

Near the bottom, I'll show you how to package all your Jupyter contents, download it, and unpackage it again when you deploy.

Final Project Structure

cfe_jupyter | Dockerfile │ Pipfile │ Pipfile.lock │ └───conf │ │ jupyter.py | └───nbs │ │ notebook.tar.gz │ └───scripts │ Dockerfile │ d_build.sh | d_run.sh | deploy.sh | entrypoint.sh

How it's done.

1. Use pipenv and install jupyter

pip install pipenv cd path/to/your/project/ pipenv install jupyter --python 3.8

2. Create Jupyter Configuration

Generate Default Config jupyter notebook --generate-config This command creates the default jupyter_notebook_config.py file on your local machine. Mine was stored on ~/.jupyter/jupyter_notebook_config.py

Create conf/jupyter.py mkdir conf echo "" > conf/jupyter.py In conf/jupyter.py add:

```python import os c = get_config()

Kernel config

c.IPKernelApp.pylab = 'inline' # if you want plotting support always in your notebook

Notebook config

c.NotebookApp.notebook_dir = 'nbs' c.NotebookApp.allow_origin = u'cfe-jupyter.herokuapp.com' # put your public IP Address here c.NotebookApp.ip = '*' c.NotebookApp.allow_remote_access = True c.NotebookApp.open_browser = False

ipython -c "from notebook.auth import passwd; passwd()"

c.NotebookApp.password = u'sha1:8da45965a489:86884d5b174e2f64e900edd129b5ef0d2f784a65' c.NotebookApp.port = int(os.environ.get("PORT", 8888)) c.NotebookApp.allow_root = True c.NotebookApp.allow_password_change = True c.ConfigurableHTTPProxy.command = ['configurable-http-proxy', '--redirect-port', '80'] ``` A few noteable setup items here:

  • c.NotebookApp.notebook_dir I set as nbs which means you should create a directory as nbs for your default notebooks directory. In my case, jupyter will open right to this directory ignoring all others.
  • c.NotebookApp.password - this has to be a hashed password. To create a new one, just run ipython -c "from notebook.auth import passwd; passwd()" on your command line.
  • c.NotebookApp.port - Heroku sets this value in our environment variables thus int(os.environ.get("PORT", 8888)) as our default.

Test your new configuration locally with: jupyter notebook --config=./conf/jupyter.py

3.Create a notebook under -> nbs/Load_Unload.ipynb

This will be how you can handle the ephemeral nature of Docker containers with Jupyter notebooks. Just create a new notebook called Load_Unload.ipynb, and add the following:

```python mode = "unload"

if mode == 'unload': # Zip all files in the current directory !tar chvfz notebook.tar.gz *

elif mode == 'load: # Unzip all files in the current directory !!tar -xv -f notebook.tar.gz ```

4. Add your Dockerfile

This is the absolute minimum setup here. You might want to add additional items as needed. Certain packages, especially the ones for data science, require additional installs for our docker-based linux server.

```dockerfile FROM python:3.8.2-slim

ENV APP_HOME /app WORKDIR ${APP_HOME}

COPY . ./

RUN pip install pip pipenv --upgrade RUN pipenv install --skip-lock --system --dev

CMD ["./scripts/entrypoint.sh"] ```

The most noteable part of this all is that (1) I'm using pipenv locally and in docker and (2) I both install pipenv and run pipenv install --system to install all pipenv dependancies to the entire docker container (instead of in a virtual environment within the container as well).

5. Create scripts/entrypoint.sh

I perfer using a entrypoint.sh script for the CMD in Dockerfiles.

```bash

!/bin/bash

/usr/local/bin/jupyter notebook --config=./conf/jupyter.py ```

6. Build & Run Docker Locally

``` docker build -t cfe-jupyter -f Dockerfile .

docker run --env PORT=8888 -it -p 8888:8888 cfe-jupyter ```

7. Heroku Setup

1. Create heroku app

heroku create cfe-jupyter - Change cfe-jupyter to your app name

2. Login to Heroku Container Registry

``` heroku container:login

```

7. Push & Release To Heroku

bash heroku container:push web heroku container:release web

  • web is the default for our Dockerfile.
  • On the commands above, you might have to append -a <your-app-name> like `heroku container:push web -a cfe-jupyter

8. That's it

heroku open This should allow you to open up your project.

Full Reference

Dockerfile

```dockerfile FROM python:3.8.2-slim

ENV APP_HOME /app WORKDIR ${APP_HOME}

COPY . ./

RUN pip install pip pipenv --upgrade RUN pipenv install --skip-lock --system --dev

CMD ["./scripts/entrypoint.sh"] ```

Pipfile

``` [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true

[dev-packages]

[packages] jupyter = "*"

[requires] python_version = "3.8" ```

scripts/d_build.sh

bash docker build -t cfe-jupyter -f Dockerfile .

scripts/d_run.sh

bash docker run --env PORT=8888 -it -p 8888:8888 cfe-jupyter

scripts/deploy.sh

bash heroku container:push web heroku container:release web

scripts/entrypoint.sh

```bash

!/bin/bash

/usr/local/bin/jupyter notebook --config=./conf/jupyter.py ```

conf/jupyter.py

```python import os c = get_config()

Kernel config

c.IPKernelApp.pylab = 'inline' # if you want plotting support always in your notebook

Notebook config

c.NotebookApp.notebook_dir = 'nbs' c.NotebookApp.allow_origin = u'cfe-jupyter.herokuapp.com' # put your public IP Address here c.NotebookApp.ip = '*' c.NotebookApp.allow_remote_access = True c.NotebookApp.open_browser = False

ipython -c "from notebook.auth import passwd; passwd()"

c.NotebookApp.password = u'sha1:8da45965a489:86884d5b174e2f64e900edd129b5ef0d2f784a65' c.NotebookApp.port = int(os.environ.get("PORT", 8888)) c.NotebookApp.allow_root = True c.NotebookApp.allow_password_change = True c.ConfigurableHTTPProxy.command = ['configurable-http-proxy', '--redirect-port', '80'] ```

Create a notebook under -> nbs/Load_Unload.ipynb

```python mode = "unload"

if mode == 'unload': # Zip all files in the current directory !tar chvfz notebook.tar.gz *

elif mode == 'load: # Unzip all files in the current directory !!tar -xv -f notebook.tar.gz ```

Bonus Installs

You might need additional packages (like numpy or pandas or opencv) in your project. Here's what you need to do in your Dockerfile, (on our repo the final docker file is listed as Dockerfile.Bonus) just update it to the following:

```dockerfile FROM python:3.8.2-slim

ENV APP_HOME /app WORKDIR ${APP_HOME}

COPY . ./

Install Ubuntu dependencies

libopencv-dev = opencv dependencies

RUN apt-get update && apt-get install -y --no-install-recommends \ tzdata \ libopencv-dev \ build-essential \ libssl-dev \ libpq-dev \ libcurl4-gnutls-dev \ libexpat1-dev \ gettext \ unzip \ supervisor \ python3-setuptools \ python3-pip \ python3-dev \ python3-venv \ python3-urllib3 \ git \ && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*

Upgrade PIP

RUN pip install pip pipenv --upgrade

sklearn opencv, numpy, and pandas

RUN pip install scikit-learn opencv-contrib-python numpy pandas

tensorflow (including Keras)

RUN pip install tensorflow keras

pytorch (cpu)

RUN apt-get update && apt-get -y install gcc mono-mcs && rm -rf /var/lib/apt/lists/* RUN pip install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

fastai

RUN pip install fastai

Project installs

RUN pipenv install --skip-lock --system --dev

CMD ["./scripts/entrypoint.sh"] ```

Issues

Create heroku.yml

opened on 2023-01-17 04:58:49 by oktest145 None

Bump nbconvert from 5.6.1 to 6.5.1

opened on 2022-08-23 18:23:50 by dependabot[bot]

Bumps nbconvert from 5.6.1 to 6.5.1.

Release notes

Sourced from nbconvert's releases.

Release 6.5.1

No release notes provided.

6.5.0

What's Changed

New Contributors

Full Changelog: https://github.com/jupyter/nbconvert/compare/6.4.5...6.5

6.4.3

What's Changed

New Contributors

Full Changelog: https://github.com/jupyter/nbconvert/compare/6.4.2...6.4.3

6.4.0

What's Changed

New Contributors

... (truncated)

Commits


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/codingforentrepreneurs/Jupyter-x-Docker-on-Heroku/network/alerts).

Bump mistune from 0.8.4 to 2.0.3

opened on 2022-07-29 23:18:52 by dependabot[bot]

Bumps mistune from 0.8.4 to 2.0.3.

Release notes

Sourced from mistune's releases.

Version 2.0.2

Fix escape_url via lepture/mistune#295

Version 2.0.1

Fix XSS for image link syntax.

Version 2.0.0

First release of Mistune v2.

Version 2.0.0 RC1

In this release, we have a Security Fix for harmful links.

Version 2.0.0 Alpha 1

This is the first release of v2. An alpha version for users to have a preview of the new mistune.

Changelog

Sourced from mistune's changelog.

Changelog

Here is the full history of mistune v2.

Version 2.0.4


Released on Jul 15, 2022

    

  • Fix url plugin in &lt;a&gt; tag
    • 

  • Fix * formatting
    • 



Version 2.0.3

Released on Jun 27, 2022

  • Fix table plugin
  • Security fix for CVE-2022-34749

Version 2.0.2


Released on Jan 14, 2022

Fix escape_url


Version 2.0.1

Released on Dec 30, 2021

XSS fix for image link syntax.

Version 2.0.0


Released on Dec 5, 2021

This is the first non-alpha release of mistune v2.


Version 2.0.0rc1

Released on Feb 16, 2021

Version 2.0.0a6


</tr></table>

... (truncated)

Commits
  • 3f422f1 Version bump 2.0.3
  • a6d4321 Fix asteris emphasis regex CVE-2022-34749
  • 5638e46 Merge pull request #307 from jieter/patch-1
  • 0eba471 Fix typo in guide.rst
  • 61e9337 Fix table plugin
  • 76dec68 Add documentation for renderer heading when TOC enabled
  • 799cd11 Version bump 2.0.2
  • babb0cf Merge pull request #295 from dairiki/bug.escape_url
  • fc2cd53 Make mistune.util.escape_url less aggressive
  • 3e8d352 Version bump 2.0.1
  • Additional commits viewable in compare view


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/codingforentrepreneurs/Jupyter-x-Docker-on-Heroku/network/alerts).

Bump notebook from 6.0.3 to 6.4.12

opened on 2022-06-16 23:58:40 by dependabot[bot]

Bumps notebook from 6.0.3 to 6.4.12.

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/codingforentrepreneurs/Jupyter-x-Docker-on-Heroku/network/alerts).

Bump ipython from 7.16.1 to 7.16.3

opened on 2022-01-21 20:42:01 by dependabot[bot]

Bumps ipython from 7.16.1 to 7.16.3.

Commits


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/codingforentrepreneurs/Jupyter-x-Docker-on-Heroku/network/alerts).

Bump pygments from 2.6.1 to 2.7.4

opened on 2021-03-30 00:59:16 by dependabot[bot]

Bumps pygments from 2.6.1 to 2.7.4.

Release notes

Sourced from pygments's releases.

2.7.4

  • Updated lexers:

    • Apache configurations: Improve handling of malformed tags (#1656)

    • CSS: Add support for variables (#1633, #1666)

    • Crystal (#1650, #1670)

    • Coq (#1648)

    • Fortran: Add missing keywords (#1635, #1665)

    • Ini (#1624)

    • JavaScript and variants (#1647 -- missing regex flags, #1651)

    • Markdown (#1623, #1617)

    • Shell

      • Lex trailing whitespace as part of the prompt (#1645)
      • Add missing in keyword (#1652)

    • SQL - Fix keywords (#1668)

    • Typescript: Fix incorrect punctuation handling (#1510, #1511)

  • Fix infinite loop in SML lexer (#1625)

  • Fix backtracking string regexes in JavaScript/TypeScript, Modula2 and many other lexers (#1637)

  • Limit recursion with nesting Ruby heredocs (#1638)

  • Fix a few inefficient regexes for guessing lexers

  • Fix the raw token lexer handling of Unicode (#1616)

  • Revert a private API change in the HTML formatter (#1655) -- please note that private APIs remain subject to change!

  • Fix several exponential/cubic-complexity regexes found by Ben Caller/Doyensec (#1675)

  • Fix incorrect MATLAB example (#1582)

Thanks to Google's OSS-Fuzz project for finding many of these bugs.

2.7.3

... (truncated)

Changelog

Sourced from pygments's changelog.

Version 2.7.4

(released January 12, 2021)

  • Updated lexers:

    • Apache configurations: Improve handling of malformed tags (#1656)

    • CSS: Add support for variables (#1633, #1666)

    • Crystal (#1650, #1670)

    • Coq (#1648)

    • Fortran: Add missing keywords (#1635, #1665)

    • Ini (#1624)

    • JavaScript and variants (#1647 -- missing regex flags, #1651)

    • Markdown (#1623, #1617)

    • Shell

      • Lex trailing whitespace as part of the prompt (#1645)
      • Add missing in keyword (#1652)

    • SQL - Fix keywords (#1668)

    • Typescript: Fix incorrect punctuation handling (#1510, #1511)

  • Fix infinite loop in SML lexer (#1625)

  • Fix backtracking string regexes in JavaScript/TypeScript, Modula2 and many other lexers (#1637)

  • Limit recursion with nesting Ruby heredocs (#1638)

  • Fix a few inefficient regexes for guessing lexers

  • Fix the raw token lexer handling of Unicode (#1616)

  • Revert a private API change in the HTML formatter (#1655) -- please note that private APIs remain subject to change!

  • Fix several exponential/cubic-complexity regexes found by Ben Caller/Doyensec (#1675)

  • Fix incorrect MATLAB example (#1582)

Thanks to Google's OSS-Fuzz project for finding many of these bugs.

Version 2.7.3

(released December 6, 2020)

... (truncated)

Commits
  • 4d555d0 Bump version to 2.7.4.
  • fc3b05d Update CHANGES.
  • ad21935 Revert "Added dracula theme style (#1636)"
  • e411506 Prepare for 2.7.4 release.
  • 275e34d doc: remove Perl 6 ref
  • 2e7e8c4 Fix several exponential/cubic complexity regexes found by Ben Caller/Doyensec
  • eb39c43 xquery: fix pop from empty stack
  • 2738778 fix coding style in test_analyzer_lexer
  • 02e0f09 Added 'ERROR STOP' to fortran.py keywords. (#1665)
  • c83fe48 support added for css variables (#1633)
  • Additional commits viewable in compare view


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/codingforentrepreneurs/Jupyter-x-Docker-on-Heroku/network/alerts).
Coding For Entrepreneurs

Build real projects and learn to code. Step by step. By @jmitchel3

GitHub Repository Homepage

jupyter python notebooks server heroku docker tutorial