Sample project with Redis and plot.ly Dash

imankulov, updated 🕥 2023-02-16 07:16:42

Visualizing Wikipedia updates stream with Redis

Wikipedia provides a real-time SSE feed for updates for all wikimedia projects.

Documentation about event streams from Wikipedia: https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams

Event stream source is available at https://stream.wikimedia.org/v2/stream/recentchange

The easiest way to visualize it:

curl https://stream.wikimedia.org/v2/stream/recentchange

Running Redis server

To run these examples you need to run a Redis server.

docker-compose up

Installing dependencies

Dependencies installed with pipenv.

pipenv install

Redis pub-sub from SSE

The file redis_publisher.py shows how to publish messages to the channel. The script reads messages from SSE and publishes them to the channel "wiki".

Run the script:

pipenv run ./redis_publisher.py

It immediately starts publishing events to the channel. You can visualize them with a simple Redis client

redis-cli subscribe wiki

Type Ctrl+C to quit.

Time-series events with Redis

There is a script redis_subscriber.py which consumes the events from the pub-sub channel and saves them to a time-series events.

Resulting data structure has following properties:

Events aggregated by domain name.
Hardcoded value expiration_timeout_sec defines how long events are stored in Redis
Another hardcoded value aggregation_interval_sec defines the event aggregation interval (in seconds).

Run the script:

pipenv run ./redis_subscriber.py

As a result of the script work, a number of keys will be created in Redis:

$ redis-cli 127.0.0.1:6379> keys * ... 7306) "ev:www.wikidata.org:1572640220" 7307) "ev:no.wikipedia.org:1572642080" 7308) "ev:cs.wiktionary.org:1572640550"

Each key contains a integer value: the number of events for the specific domain name, happening within aggregation_interval_sec after the time moment, encoded as the timestamp in the key.

Visualizing time-series events with Dash

Dash is an open source web framework to build interactive data-driven applications, primarily data visualizations and dashboards.

File dash_app.py contains a sample application which lets you pick one or more domain names to get te update statistics for it.

Run the script

pipenv run ./dash_app.py

As a result, you should see:

More information

Issues

Bump werkzeug from 2.2.2 to 2.2.3

opened on 2023-02-16 07:16:42 by dependabot[bot]

Bumps werkzeug from 2.2.2 to 2.2.3.

Release notes

Sourced from werkzeug's releases.

2.2.3

This is a fix release for the 2.2.x release branch.

Changes: https://werkzeug.palletsprojects.com/en/2.2.x/changes/#version-2-2-3

Milestone: https://github.com/pallets/werkzeug/milestone/26?closed=1

This release contains security fixes for:

https://github.com/pallets/werkzeug/security/advisories/GHSA-xg9f-g7g7-2323

https://github.com/pallets/werkzeug/security/advisories/GHSA-px8h-6qxv-m22q

Changelog

Sourced from werkzeug's changelog.

Version 2.2.3

Released 2023-02-14

Ensure that URL rules using path converters will redirect with strict slashes when the trailing slash is missing. :issue:2533

Type signature for get_json specifies that return type is not optional when silent=False. :issue:2508

parse_content_range_header returns None for a value like bytes */-1 where the length is invalid, instead of raising an AssertionError. :issue:2531

Address remaining ResourceWarning related to the socket used by run_simple. Remove prepare_socket, which now happens when creating the server. :issue:2421

Update pre-existing headers for multipart/form-data requests with the test client. :issue:2549

Fix handling of header extended parameters such that they are no longer quoted. :issue:2529

LimitedStream.read works correctly when wrapping a stream that may not return the requested size in one read call. :issue:2558

A cookie header that starts with = is treated as an empty key and discarded, rather than stripping the leading ==.

Specify a maximum number of multipart parts, default 1000, after which a RequestEntityTooLarge exception is raised on parsing. This mitigates a DoS attack where a larger number of form/file parts would result in disproportionate resource use.

Commits

22a254f release version 2.2.3
517cac5 Merge pull request from GHSA-xg9f-g7g7-2323
babc8d9 rewrite docs about request data limits
09449ee clean up docs
fe899d0 limit the maximum number of multipart form parts
cf275f4 Merge pull request from GHSA-px8h-6qxv-m22q
8c2b4b8 don't strip leading = when parsing cookie
7c7ce5c [pre-commit.ci] pre-commit autoupdate (#2585)
19ae03e [pre-commit.ci] auto fixes from pre-commit.com hooks
a83d3b8 [pre-commit.ci] pre-commit autoupdate
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/imankulov/wikipedia-playground/network/alerts).

Bump ipython from 7.16.3 to 8.10.0

opened on 2023-02-10 23:11:36 by dependabot[bot]

Bumps ipython from 7.16.3 to 8.10.0.

Release notes

Sourced from ipython's releases.

See https://pypi.org/project/ipython/

We do not use GitHub release anymore. Please see PyPI https://pypi.org/project/ipython/

Commits

15ea1ed release 8.10.0
560ad10 DOC: Update what's new for 8.10 (#13939)
7557ade DOC: Update what's new for 8.10
385d693 Merge pull request from GHSA-29gw-9793-fvw7
e548ee2 Swallow potential exceptions from showtraceback() (#13934)
0694b08 MAINT: mock slowest test. (#13885)
8655912 MAINT: mock slowest test.
a011765 Isolate the attack tests with setUp and tearDown methods
c7a9470 Add some regression tests for this change
fd34cf5 Swallow potential exceptions from showtraceback()
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Roman Imankulov

Full-stack Python web developer. Software development is not about code. It's about people.

GitHub Repository