Python implementation of the HydRand protocol

PhilippSchindler, updated 🕥 2022-10-18 19:03:43

HydRand

This is a python implementation of the HydRand protocol for generating publicly-verifiable randomness in a distributed fashion. The corresponding research paper is accepted for the IEEE S&P conference and available online at https://doi.ieeecomputersociety.org/10.1109/SP40000.2020.00003. This work is a joint research project by my team members Aljosha Judmayer and Nicholas Stifter and myself, conducted at SBA Research and Vienna University of Technology under the supervision of Edgar Weippl.

Security notice

Currently, the open source implementation provided in this repository is a proof-of-concept implementation and mainly used for performance evaluation. While the main functionally, including signatures checks and verification procedures of the messages, have been fully implemented the code is still considered a prototype and NOT SAFE TO USE IN PRODUCTION. Please contact me or one of my team members at SBA research if you are interested in running HydRand in production.

Flowchart

(Somewhat simplified) figure of the internals of the Hydrand protocol, from the view of an individual Hydrand node.

throughput

Message Format

Overview of format of the messages exchanged by HydRand nodes.

throughput

Performance evaluation

We tested the performance of the protocol using Amazon Web Services (AWS) with up to 128 EC2 instances of type t2.micro (1GB of RAM, 1 virtual CPU core, and a 60-80 Mbit/s internet connection) in different globally distributed datacenters. We discuss our evaluation in the research paper https://eprint.iacr.org/2018/319. In the following we provide additional metrics obtained during our performance tests. The corresponding raw data is provided upon request (the collected files are bigger than the limit of this free git repository).

Throughput

The following figure shows the troughput of the HydRand protocol benchmark runs for different configurations. The round duration was experimentally derived. We tested the protocol in two settings: for the first setting we run the HydRand software as is and ensured that every single node was able to complete the protocol run successfully. For the second settings we deliberately stopped f nodes during the execution of the protocol. In this case all remaining node are able to finished the protocol run successfully.

throughput

Network bandwidth

The following figure shows the average bandwidth used by the HydRand nodes, again with and without simulated failures.

bandwidth

CPU utilization

It can be observed, that the limiting resource in our evaluation was the vCPU of the AWS instances. The following figure shows the overall CPU utilization in percent for different runs with the respective number of nodes on the x axis.

cpuruns

The following figure shows a normal run with 128 nodes.

cpuall

Memory utilization

The following figure shows the memory utilization of the AWS instances in MiB for different runs with the respective number of nodes on the x axis.

memruns

The following figure shows a normal run with 128 nodes.

memall

Network utilization

The following figures shows the sent data in Mbit/s for different runs with the respective number of nodes on the x axis.

sendruns

The following figure shows the detailed data received in a normal run with 128 nodes for every node. The bursts show that almost all nodes have been selected as leader and broadcasted a propose message.

sendall

The following figure shows the data received in a normal run with 128 nodes for every node.

recvall

Running HydRand on a single machine

To try HydRand and for development purposes it is useful to run the protocol on a single machine. In the following, we give additional instructions on running Hydrand for this purpose.

Dependencies

Ensure you have python3.7 and the package pyzmq installed.
You can install pyzmq using the command python3.7 -m pip install pyzmq.

Network configuration

HydRand automatically determines which network configuration to use. It uses the config/network/localhost.txt configuration file per default. Without modification of this file the ports 5000, 5001, 5002 ... are used by the nodes with id 0, 1, 2, ..., respectively. Ensure that these ports are not used by any other software running on the system or change the configuration file to use different ports. (The file config/network/amazon.txt is used if HydRand is executed on a Amazon EC2 instance, all nodes are listening on port 5000 per default).

Cryptographic Keys and Initial PVSS Shares

In the config/005, config/007, config/008 and config/016, we provide a pre generated set of keypairs used to sign and verfify the sent messages as well as the initial pvss shares for scenarios with 5, 7, 8 and 16 nodes.

If you want to test the software with a different number of nodes (e.g. 4) nodes you need to generate these files for your specifiy setting. For this purpose have a look at the file config.py, it contains a function which creates a sample configuration. From the root directory of the respositity you can invoke it direclty by executing
python3.7 -c "from hydrand.config import generate_sample_config; generate_sample_config(n=4, write_to_disk=True)".
It generates the folder config/004 with all required files.

Starting HydRand nodes

To test the protocol with e.g. 5 nodes locally, open up 5 terminal windows and switch the root directory of this reposity.

In terminal window A execute python3.7 -m hydrand 0 -n 5 --start-at "2019-04-17 09:09:50" --num-rounds 20
In terminal window B execute python3.7 -m hydrand 1 -n 5 --start-at "2019-04-17 09:09:50" --num-rounds 20
...
In terminal window E execute python3.7 -m hydrand 4 -n 5 --start-at "2019-04-17 09:09:50" --num-rounds 20

The first argument (i.e. 0, 1, ..., 4 in the example) specifies the HydRand node id (a zero-based index).
The parameter -n 5 tells all nodes to use a configuration with 5 nodes in total.

Finally, --start-at 2019-04-17 09:09:50 is used to specifiy the common time the protocol should start. This timestamp MUST BE SPECIFIED IN UTC TIME. Ensure that the timestamp is sufficiently far in the future so that you can start all 5 nodes before the start time is reached. (Failing to do so triggers a recovery of the unstarted nodes.)

--num-rounds 20 instructs the nodes to automatically shutdown after completing 20 protocol rounds.

You call also execute python3.7 -m hydrand --help to get additional inforamation on the available command line options.

Notes

Depencencies

For running the software, python3.7 and with the package pyzmq is required. For executing all test cases we additionally require the pytest and sympy packages. For developement we provide dependency files for the use with pipenv.

See also ./aws/setup-instance.sh for a script which installs all the required dependencies to run HydRand on an Amazon EC2 instance.

Configuration

Configuration files FOR TESTING ONLY can be generated for differnent number of nodes using the pyhton file ./hydrand/config.py. Notice that the script generates key files for all nodes and stores them in the ./config directory. It is NOT SECURE to use the script for setting up the protocol IN PRODUCTION.

In addition the scripts provided in the ./aws folder automatically generate a network configuration file.

Tesing locally

The available unit tests can be executed using pytest as a test runner.

For testing a high number of nodes on a single local machine the ulimit has to be increased, e.g. ulimit -n 4096.
See also /etc/security/limits.conf.

Acknowlegdements

I would like to express my very great appreciation to my co-authors Aljosha Judmayer and Nicholas Stifter for the excellent collabortion and support throughout the design and implementation of this project, a variety of critical discussions, and their valuable contributions to the paper.

Issues

Bump py from 1.8.0 to 1.10.0

opened on 2022-10-18 19:03:42 by dependabot[bot]

Bumps py from 1.8.0 to 1.10.0.

Changelog

Sourced from py's changelog.

1.10.0 (2020-12-12)

Fix a regular expression DoS vulnerability in the py.path.svnwc SVN blame functionality (CVE-2020-29651)

Update vendored apipkg: 1.4 => 1.5

Update vendored iniconfig: 1.0.0 => 1.1.1

1.9.0 (2020-06-24)

Add type annotation stubs for the following modules:

py.error

py.iniconfig

py.path (not including SVN paths)

py.io

py.xml

There are no plans to type other modules at this time.

The type annotations are provided in external .pyi files, not inline in the code, and may therefore contain small errors or omissions. If you use py in conjunction with a type checker, and encounter any type errors you believe should be accepted, please report it in an issue.

1.8.2 (2020-06-15)

On Windows, py.path.locals which differ only in case now have the same Python hash value. Previously, such paths were considered equal but had different hashes, which is not allowed and breaks the assumptions made by dicts, sets and other users of hashes.

1.8.1 (2019-12-27)

Handle FileNotFoundError when trying to import pathlib in path.common on Python 3.4 (#207).

py.path.local.samefile now works correctly in Python 3 on Windows when dealing with symlinks.

Commits

e5ff378 Update CHANGELOG for 1.10.0
94cf44f Update vendored libs
5e8ded5 testing: comment out an assert which fails on Python 3.9 for now
afdffcc Rename HOWTORELEASE.rst to RELEASING.rst
2de53a6 Merge pull request #266 from nicoddemus/gh-actions
fa1b32e Merge pull request #264 from hugovk/patch-2
887d6b8 Skip test_samefile_symlink on pypy3 on Windows
e94e670 Fix test_comments() in test_source
fef9a32 Adapt test
4a694b0 Add GitHub Actions badge to README
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/PhilippSchindler/HydRand/network/alerts).

Bump nbconvert from 5.5.0 to 6.5.1

opened on 2022-08-23 17:49:19 by dependabot[bot]

Bumps nbconvert from 5.5.0 to 6.5.1.

Release notes

Sourced from nbconvert's releases.

Release 6.5.1

No release notes provided.

6.5.0

What's Changed

Drop dependency on testpath. by @anntzer in jupyter/nbconvert#1723

Adopt pre-commit by @blink1073 in jupyter/nbconvert#1744

Add pytest settings and handle warnings by @blink1073 in jupyter/nbconvert#1745

Apply Autoformatters by @blink1073 in jupyter/nbconvert#1746

Add git-blame-ignore-revs by @blink1073 in jupyter/nbconvert#1748

Update flake8 config by @blink1073 in jupyter/nbconvert#1749

support bleach 5, add packaging and tinycss2 dependencies by @bollwyvl in jupyter/nbconvert#1755

[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in jupyter/nbconvert#1752

update cli example by @leahecole in jupyter/nbconvert#1753

Clean up pre-commit by @blink1073 in jupyter/nbconvert#1757

Clean up workflows by @blink1073 in jupyter/nbconvert#1750

New Contributors

@pre-commit-ci made their first contribution in jupyter/nbconvert#1752

Full Changelog: https://github.com/jupyter/nbconvert/compare/6.4.5...6.5

6.4.3

What's Changed

Add section to customizing showing how to use template inheritance by @stefanv in jupyter/nbconvert#1719

Remove ipython genutils by @rgs258 in jupyter/nbconvert#1727

Update changelog for 6.4.3 by @blink1073 in jupyter/nbconvert#1728

New Contributors

@stefanv made their first contribution in jupyter/nbconvert#1719

@rgs258 made their first contribution in jupyter/nbconvert#1727

Full Changelog: https://github.com/jupyter/nbconvert/compare/6.4.2...6.4.3

6.4.0

What's Changed

Optionally speed up validation by @gwincr11 in jupyter/nbconvert#1672

Adding missing div compared to JupyterLab DOM structure by @SylvainCorlay in jupyter/nbconvert#1678

Allow passing extra args to code highlighter by @yuvipanda in jupyter/nbconvert#1683

Prevent page breaks in outputs when printing by @SylvainCorlay in jupyter/nbconvert#1679

Add collapsers to template by @SylvainCorlay in jupyter/nbconvert#1689

Fix recent pandoc latex tables by adding calc and array (#1536, #1566) by @cgevans in jupyter/nbconvert#1686

Add an invalid notebook error by @gwincr11 in jupyter/nbconvert#1675

Fix typos in execute.py by @TylerAnderson22 in jupyter/nbconvert#1692

Modernize latex greek math handling (partially fixes #1673) by @cgevans in jupyter/nbconvert#1687

Fix use of deprecated API and update test matrix by @blink1073 in jupyter/nbconvert#1696

Update nbconvert_library.ipynb by @letterphile in jupyter/nbconvert#1695

Changelog for 6.4 by @blink1073 in jupyter/nbconvert#1697

New Contributors

... (truncated)

Commits

7471b75 Release 6.5.1
c1943e0 Fix pre-commit
8685e93 Fix tests
0abf290 Run black and prettier
418d545 Run test on 6.x branch
bef65d7 Convert input to string prior to escape HTML
0818628 Check input type before escaping
b206470 GHSL-2021-1017, GHSL-2021-1020, GHSL-2021-1021
a03cbb8 GHSL-2021-1026, GHSL-2021-1025
48fe71e GHSL-2021-1024
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump mistune from 0.8.4 to 2.0.3

opened on 2022-07-29 23:00:02 by dependabot[bot]

Bumps mistune from 0.8.4 to 2.0.3.

Release notes

Sourced from mistune's releases.

Version 2.0.2

Fix escape_url via lepture/mistune#295

Version 2.0.1

Fix XSS for image link syntax.

Version 2.0.0

First release of Mistune v2.

Version 2.0.0 RC1

In this release, we have a Security Fix for harmful links.

Version 2.0.0 Alpha 1

This is the first release of v2. An alpha version for users to have a preview of the new mistune.

Changelog

Sourced from mistune's changelog.

Changelog

Here is the full history of mistune v2.

Version 2.0.4
Released on Jul 15, 2022

Fix url plugin in &lt;a&gt; tag
Fix * formatting

Version 2.0.3
Released on Jun 27, 2022

Fix table plugin

Security fix for CVE-2022-34749

Version 2.0.2
Released on Jan 14, 2022
Fix escape_url
Version 2.0.1
Released on Dec 30, 2021

XSS fix for image link syntax.

Version 2.0.0
Released on Dec 5, 2021
This is the first non-alpha release of mistune v2.
Version 2.0.0rc1
Released on Feb 16, 2021

Version 2.0.0a6
</tr></table> 

... (truncated)

Commits

3f422f1 Version bump 2.0.3
a6d4321 Fix asteris emphasis regex CVE-2022-34749
5638e46 Merge pull request #307 from jieter/patch-1
0eba471 Fix typo in guide.rst
61e9337 Fix table plugin
76dec68 Add documentation for renderer heading when TOC enabled
799cd11 Version bump 2.0.2
babb0cf Merge pull request #295 from dairiki/bug.escape_url
fc2cd53 Make mistune.util.escape_url less aggressive
3e8d352 Version bump 2.0.1
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump notebook from 5.7.8 to 6.4.12

opened on 2022-06-16 23:32:36 by dependabot[bot]

Bumps notebook from 5.7.8 to 6.4.12.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump paramiko from 2.4.2 to 2.10.1

opened on 2022-03-29 22:03:41 by dependabot[bot]

Bumps paramiko from 2.4.2 to 2.10.1.

Commits

286bd9f Cut 2.10.1
4c491e2 Fix CVE re: PKey.write_private_key chmod race
aa3cc6f Cut 2.10.0
e50e19f Fix up changelog entry with real links
02ad67e Helps to actually leverage your mocked system calls
29d7bf4 Clearly our agent stuff is not fully tested yet...
5fcb8da OpenSSH docs state %C should also work in IdentityFile and Match exec
1bf3dce Changelog enhancement
f6342fc Prettify, add %C as acceptable controlpath token, mock gethostname
3f3451f Add to changelog
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump ipython from 7.5.0 to 7.16.3

opened on 2022-01-21 19:58:46 by dependabot[bot]

Bumps ipython from 7.5.0 to 7.16.3.

Commits

d43c7c7 release 7.16.3
5fa1e40 Merge pull request from GHSA-pq7m-3gw7-gq5x
8df8971 back to dev
9f477b7 release 7.16.2
138f266 bring back release helper from master branch
5aa3634 Merge pull request #13341 from meeseeksmachine/auto-backport-of-pr-13335-on-7...
bcae8e0 Backport PR #13335: What's new 7.16.2
8fcdcd3 Pin Jedi to <0.17.2.
2486838 release 7.16.1
20bdc6f fix conda build
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Philipp Schindler

PostDoc Researcher @ University of Vienna, Researcher & Consultant @ SBA Research

GitHub Repository