The purpose of this project is to demonstrate how to automate the testing and deployment of a simple Flask-based (RESTful) micro-service to a production-like environment on AWS. The deployment pipeline is handled by Travis-CI, that has been granted access to this GitHub repository and configured to run upon a pull request or a merge to the master branch. The pipeline is defined in the .travis.yaml
file and consists of the following steps:
Pipenv
package using pip
;Pipenv
to install the project dependencies defined in Pipfile.lock
;pipenv run python -m unittest tests/*.py
; and,master
branch - e.g. if a pull request has been merged - then start Docker and run the deploy_to_aws.py
script.The deploy_to_aws.py
script defines the deployment process, which performs the following steps without any manual intervention:
It is reliant on the definition of three environment variables: AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, and AWS_REGION
. For security reasons, these are kept out of the .travis.yml
and are instead defined using the Travis-CI UI.
Although the micro-service used in this example - as defined in microservice/api.py
module - only returns a simple message upon a simple GET
request, it could just as easily be a Machine Learning (ML) model-scoring service that receives the values of feature variables and returns a prediction - the overall pattern is the same.
Currently, the initial setup of the required AWS infrastructure is entirely manual (although this could also be scripted in the future). What's required, is an ECS cluster that is capable hosting multiple groups of Docker containers (or 'tasks' - i.e. web applications or in our case just a single micro-service), that sit behind a load balances that accepts incoming traffic and routes it to different containers in the cluster. Collectively,this constitutes a 'service' that is highly available. At a high-level, the steps required to setup this infrastructure using the AWS management console, are as follows (assuming the existence of a repository in ECR, containing our docker image):
t2.medium
;/microservice
, otherwise it won't get 200s and and will try to re-register hosts;daemon
mode - i.e. assume there is only one container per-task;We use pipenv for managing project dependencies and Python environments (i.e. virtual environments). All of the direct packages dependencies required to run the code (e.g. docker and boto3), as well as all the packages used during development (e.g. IPython for interactive console sessions), are described in the Pipfile
. Their precise downstream dependencies are described in Pipfile.lock
.
To get started with Pipenv, first of all download it - assuming that there is a global version of Python available on your system and on the PATH, then this can be achieved by running the following command,
bash
pip3 install pipenv
Pipenv is also available to install from many non-Python package managers. For example, on OS X it can be installed using the Homebrew package manager, with the following terminal command,
bash
brew install pipenv
For more information, including advanced configuration options, see the official pipenv documentation.
Make sure that you're in the project's root directory (the same one in which Pipfile
resides), and then run,
bash
pipenv install --dev
This will install all of the direct project dependencies as well as the development dependencies (the latter a consequence of the --dev
flag).
In order to continue development in a Python environment that precisely mimics the one the project was initially developed with, use Pipenv from the command line as follows,
bash
pipenv run python3
The python3
command could just as well be ipython3
or the Jupyter notebook server, for example,
bash
pipenv run jupyter notebook
This will fire-up a Jupyter notebook server where the default Python 3 kernel includes all of the direct and development project dependencies. This is how we advise that the notebooks within this project are used.
All test have been written using the unittest package from the Python standard library. Tests are kept in the tests
folder and can be run from the command line by - e.g. by evoking,
bash
pipenv run python -m unittest tests/test_*.py
This can be started via the command line, from the root directory using,
bash
pipenv run python -m microservice.api
Which will start the server at http://localhost:5000/microservice
.
Bumps werkzeug from 0.14.1 to 0.15.5.
Sourced from werkzeug's releases.
0.15.5
0.15.4
- Blog: https://palletsprojects.com/blog/werkzeug-0-15-3-released/
- Changes: https://werkzeug.palletsprojects.com/en/0.15.x/changes/#version-0-15-4
0.15.3
- Blog: https://palletsprojects.com/blog/werkzeug-0-15-3-released/
- Changes: https://werkzeug.palletsprojects.com/en/0.15.x/changes/#version-0-15-3
0.15.2
- Blog: https://palletsprojects.com/blog/werkzeug-0-15-2-released/
- Changes: https://werkzeug.palletsprojects.com/en/0.15.x/changes/#version-0-15-2
0.15.1
- Blog: https://palletsprojects.com/blog/werkzeug-0-15-1-released/
- Changes: https://werkzeug.palletsprojects.com/en/0.15.x/changes/
0.15.0
Sourced from werkzeug's changelog.
Version 0.15.5
Released 2019-07-17
- Fix a
TypeError
due to changes toast.Module
in Python 3.8. :issue:1551
- Fix a C assertion failure in debug builds of some Python 2.7 releases. :issue:
1553
- :class:
~exceptions.BadRequestKeyError
adds theKeyError
message to the description ife.show_exception
is set toTrue
. This is a more secure default than the original 0.15.0 behavior and makes it easier to control without losing information. :pr:1592
- Upgrade the debugger to jQuery 3.4.1. :issue:
1581
- Work around an issue in some external debuggers that caused the reloader to fail. :issue:
1607
- Work around an issue where the reloader couldn't introspect a setuptools script installed as an egg. :issue:
1600
- The reloader will use
sys.executable
even if the script is marked executable, reverting a behavior intended for NixOS introduced in 0.15. The reloader should no longer causeOSError: [Errno 8] Exec format error
. :issue:1482
, :issue:1580
SharedDataMiddleware
safely handles paths with Windows drive names. :issue:1589
Version 0.15.4
Released 2019-05-14
- Fix a
SyntaxError
on Python 2.7.5. (:issue:1544
)Version 0.15.3
Released 2019-05-14
- Properly handle multi-line header folding in development server in Python 2.7. (:issue:
1080
)- Restore the
response
argument to :exc:~exceptions.Unauthorized
. (:pr:1527
)- :exc:
~exceptions.Unauthorized
doesn't add theWWW-Authenticate
header ifwww_authenticate
is not given. (:issue:1516
)- The default URL converter correctly encodes bytes to string rather than representing them with
b''
. (:issue:1502
)- Fix the filename format string in
... (truncated)
8cfab30
release version 0.15.5d1f5fab
update project links726b25b
Merge pull request #1613 from pallets/revert-nixos-reloader-check1f532b8
don't detect executable file for reloadera8d26bf
Merge pull request #1612 from pallets/shared-dataacc999e
SharedDataMiddleware uses safe_join8afe4eb
Merge pull request #1582 from Cerebus/fix/1581-jquery-3.4.1e0de4a4
Merge pull request #1601 from pokoli/reloader_packageec68771
update to jQuery 3.4.17a01660
explain reloader workaround for egg scriptDependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps certifi from 2018.4.16 to 2022.12.7.
9e9e840
2022.12.07b81bdb2
2022.09.24939a28f
2022.09.14aca828a
2022.06.15.2de0eae1
Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...b8eb5e9
2022.06.15.147fb7ab
Fix deprecation warning on Python 3.11 (#199)b0b48e0
fixes #198 -- update link in license9d514b4
2022.06.154151e88
Add py.typed to MANIFEST.in to package in sdist (#196)Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps ipython from 6.4.0 to 7.16.3.
d43c7c7
release 7.16.35fa1e40
Merge pull request from GHSA-pq7m-3gw7-gq5x8df8971
back to dev9f477b7
release 7.16.2138f266
bring back release helper from master branch5aa3634
Merge pull request #13341 from meeseeksmachine/auto-backport-of-pr-13335-on-7...bcae8e0
Backport PR #13335: What's new 7.16.28fcdcd3
Pin Jedi to <0.17.2.2486838
release 7.16.120bdc6f
fix conda buildDependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps urllib3 from 1.23 to 1.26.5.
Sourced from urllib3's releases.
1.26.5
:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap
- Fixed deprecation warnings emitted in Python 3.10.
- Updated vendored
six
library to 1.16.0.- Improved performance of URL parser when splitting the authority component.
If you or your organization rely on urllib3 consider supporting us via GitHub Sponsors
1.26.4
:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap
- Changed behavior of the default
SSLContext
when connecting to HTTPS proxy during HTTPS requests. The defaultSSLContext
now setscheck_hostname=True
.If you or your organization rely on urllib3 consider supporting us via GitHub Sponsors
1.26.3
:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap
Fixed bytes and string comparison issue with headers (Pull #2141)
Changed
ProxySchemeUnknown
error message to be more actionable if the user supplies a proxy URL without a scheme (Pull #2107)If you or your organization rely on urllib3 consider supporting us via GitHub Sponsors
1.26.2
:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap
- Fixed an issue where
wrap_socket
andCERT_REQUIRED
wouldn't be imported properly on Python 2.7.8 and earlier (Pull #2052)1.26.1
:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap
- Fixed an issue where two
User-Agent
headers would be sent if aUser-Agent
header key is passed asbytes
(Pull #2047)1.26.0
:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap
Added support for HTTPS proxies contacting HTTPS servers (Pull #1923, Pull #1806)
Deprecated negotiating TLSv1 and TLSv1.1 by default. Users that still wish to use TLS earlier than 1.2 without a deprecation warning should opt-in explicitly by setting
ssl_version=ssl.PROTOCOL_TLSv1_1
(Pull #2002) Starting in urllib3 v2.0: Connections that receive aDeprecationWarning
will failDeprecated
Retry
optionsRetry.DEFAULT_METHOD_WHITELIST
,Retry.DEFAULT_REDIRECT_HEADERS_BLACKLIST
andRetry(method_whitelist=...)
in favor ofRetry.DEFAULT_ALLOWED_METHODS
,Retry.DEFAULT_REMOVE_HEADERS_ON_REDIRECT
, andRetry(allowed_methods=...)
(Pull #2000) Starting in urllib3 v2.0: Deprecated options will be removed
... (truncated)
Sourced from urllib3's changelog.
1.26.5 (2021-05-26)
- Fixed deprecation warnings emitted in Python 3.10.
- Updated vendored
six
library to 1.16.0.- Improved performance of URL parser when splitting the authority component.
1.26.4 (2021-03-15)
- Changed behavior of the default
SSLContext
when connecting to HTTPS proxy during HTTPS requests. The defaultSSLContext
now setscheck_hostname=True
.1.26.3 (2021-01-26)
Fixed bytes and string comparison issue with headers (Pull #2141)
Changed
ProxySchemeUnknown
error message to be more actionable if the user supplies a proxy URL without a scheme. (Pull #2107)1.26.2 (2020-11-12)
- Fixed an issue where
wrap_socket
andCERT_REQUIRED
wouldn't be imported properly on Python 2.7.8 and earlier (Pull #2052)1.26.1 (2020-11-11)
- Fixed an issue where two
User-Agent
headers would be sent if aUser-Agent
header key is passed asbytes
(Pull #2047)1.26.0 (2020-11-10)
NOTE: urllib3 v2.0 will drop support for Python 2.
Read more in the v2.0 Roadmap <https://urllib3.readthedocs.io/en/latest/v2-roadmap.html>
_.Added support for HTTPS proxies contacting HTTPS servers (Pull #1923, Pull #1806)
Deprecated negotiating TLSv1 and TLSv1.1 by default. Users that still wish to use TLS earlier than 1.2 without a deprecation warning
... (truncated)
d161647
Release 1.26.52d4a3fe
Improve performance of sub-authority splitting in URL2698537
Update vendored six to 1.16.007bed79
Fix deprecation warnings for Python 3.10 ssl moduled725a9b
Add Python 3.10 to GitHub Actions339ad34
Use pytest==6.2.4 on Python 3.10+f271c9c
Apply latest Black formatting1884878
[1.26] Properly proxy EOF on the SSLTransport test suitea891304
Release 1.26.48d65ea1
Merge pull request from GHSA-5phf-pp7p-vc2rDependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps pygments from 2.2.0 to 2.7.4.
Sourced from pygments's releases.
2.7.4
Updated lexers:
Fix infinite loop in SML lexer (#1625)
Fix backtracking string regexes in JavaScript/TypeScript, Modula2 and many other lexers (#1637)
Limit recursion with nesting Ruby heredocs (#1638)
Fix a few inefficient regexes for guessing lexers
Fix the raw token lexer handling of Unicode (#1616)
Revert a private API change in the HTML formatter (#1655) -- please note that private APIs remain subject to change!
Fix several exponential/cubic-complexity regexes found by Ben Caller/Doyensec (#1675)
Fix incorrect MATLAB example (#1582)
Thanks to Google's OSS-Fuzz project for finding many of these bugs.
2.7.3
... (truncated)
Sourced from pygments's changelog.
Version 2.7.4
(released January 12, 2021)
Updated lexers:
Fix infinite loop in SML lexer (#1625)
Fix backtracking string regexes in JavaScript/TypeScript, Modula2 and many other lexers (#1637)
Limit recursion with nesting Ruby heredocs (#1638)
Fix a few inefficient regexes for guessing lexers
Fix the raw token lexer handling of Unicode (#1616)
Revert a private API change in the HTML formatter (#1655) -- please note that private APIs remain subject to change!
Fix several exponential/cubic-complexity regexes found by Ben Caller/Doyensec (#1675)
Fix incorrect MATLAB example (#1582)
Thanks to Google's OSS-Fuzz project for finding many of these bugs.
Version 2.7.3
(released December 6, 2020)
... (truncated)
4d555d0
Bump version to 2.7.4.fc3b05d
Update CHANGES.ad21935
Revert "Added dracula theme style (#1636)"e411506
Prepare for 2.7.4 release.275e34d
doc: remove Perl 6 ref2e7e8c4
Fix several exponential/cubic complexity regexes found by Ben Caller/Doyenseceb39c43
xquery: fix pop from empty stack2738778
fix coding style in test_analyzer_lexer02e0f09
Added 'ERROR STOP' to fortran.py keywords. (#1665)c83fe48
support added for css variables (#1633)Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps jinja2 from 2.10 to 2.11.3.
Sourced from jinja2's releases.
2.11.3
This contains a fix for a speed issue with the
urlize
filter.urlize
is likely to be called on untrusted user input. For certain inputs some of the regular expressions used to parse the text could take a very long time due to backtracking. As part of the fix, the email matching became slightly stricter. The various speedups apply tourlize
in general, not just the specific input cases.
- PyPI: https://pypi.org/project/Jinja2/2.11.3/
- Changes: https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-3
2.11.2
2.11.1
This fixes an issue in async environment when indexing the result of an attribute lookup, like
{{ data.items[1:] }}
.2.11.0
- Changes: https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-0
- Blog: https://palletsprojects.com/blog/jinja-2-11-0-released/
- Twitter: https://twitter.com/PalletsTeam/status/1221883554537230336
This is the last version to support Python 2.7 and 3.5. The next version will be Jinja 3.0 and will support Python 3.6 and newer.
2.10.3
2.10.2
2.10.1
Sourced from jinja2's changelog.
Version 2.11.3
Released 2021-01-31
- Improve the speed of the
urlize
filter by reducing regex backtracking. Email matching requires a word character at the start of the domain part, and only word characters in the TLD. :pr:1343
Version 2.11.2
Released 2020-04-13
- Fix a bug that caused callable objects with
__getattr__
, like :class:~unittest.mock.Mock
to be treated as a :func:contextfunction
. :issue:1145
- Update
wordcount
filter to trigger :class:Undefined
methods by wrapping the input in :func:soft_str
. :pr:1160
- Fix a hang when displaying tracebacks on Python 32-bit. :issue:
1162
- Showing an undefined error for an object that raises
AttributeError
on access doesn't cause a recursion error. :issue:1177
- Revert changes to :class:
~loaders.PackageLoader
from 2.10 which removed the dependency on setuptools and pkg_resources, and added limited support for namespace packages. The changes caused issues when using Pytest. Due to the difficulty in supporting Python 2 and :pep:451
simultaneously, the changes are reverted until 3.0. :pr:1182
- Fix line numbers in error messages when newlines are stripped. :pr:
1178
- The special
namespace()
assignment object in templates works in async environments. :issue:1180
- Fix whitespace being removed before tags in the middle of lines when
lstrip_blocks
is enabled. :issue:1138
- :class:
~nativetypes.NativeEnvironment
doesn't evaluate intermediate strings during rendering. This prevents early evaluation which could change the value of an expression. :issue:1186
Version 2.11.1
Released 2020-01-30
- Fix a bug that prevented looking up a key after an attribute (
{{ data.items[1:] }}
) in an async template. :issue:1141
... (truncated)
cf21539
release version 2.11.315ef8f0
Merge pull request #1343 from pallets/urlize-speedupef658dc
speed up urlize matchingeeca0fe
Merge pull request #1207 from mhansen/patch-12dd7691
Merge pull request #1209 from mhansen/patch-34892940
do_dictsort: update example ready to copy/paste7db7d33
api.rst: bugfix in docs, import PackageLoader9ec465b
fix changelog header737a4cd
release version 2.11.2179df6b
Merge pull request #1190 from pallets/native-evalDependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
machine learning engineer - (data)scientist - reformed quant - habitual coder - PhD
GitHub Repositorypython microservice docker aws ecs travis-ci continuous-integration continuous-deployment boto3