Core repo for election results data acquisition, transformation and output.

openelections, updated 🕥 2022-01-21 18:56:12

Open Elections core code

OpenElections core is developed and tested using Python 3. The package might not work with older Python distributions.

Getting started as a developer

You'll:

set up a virtual environment
fork/clone this repository, install dependencies
add any optional configuration details you need (e.g. Mongo or AWS)

Setting up a virtual environment

You should use pipenv to work on Open Elections inside a virtualized development environment.

The easiest way is to install these tools system-wide with pip (you may need to use sudo):

bash $ pip install --upgrade pipenv

Then, to make a virtual environment for open elections work:

bash $ pipenv install --dev

To activate the virtual environment, run:

bash $ pipenv shell

Fork and set up this project

Fork this repo by hitting the "Fork" button above, and clone your fork to your computer:

bash $ git clone https://github.com/[my_github_user]/openelections-core.git $ cd openelections-core

Turn setup and activate the virtual environment from the previous step, if you haven't already:

bash $ pipenv install --dev $ pipenv shell

Install MongoDB

To store your data in MongoDB, you need only install Mongo. The default configuration should auto-create the databases and tables you need, as you need them.

At the very least, you'll want to make sure the values in the MONGO variable work for the way you've installed and configured MongoDB on your system.

Create a settings.py file.

bash $ cp settings.py.tmplt settings.py

You can put this settings file anywhere on your filesystem. You'll need to set the OPENELEX_SETTINGS environment variable to the absolute path to the settings.py file that you created.

You can set the OPENELEX_SETTINGS environment variable by creating a .env file in your repository. For example:

OPENELEX_SETTINGS=/Users/[myusername]/Development/openelections-core/settings.py

Running pipenv shell will automatically source any variable in your .env file.

Running management commands

Test it out by running openelex --help, you should see something like:

```bash $ openelex --help Usage: openelex [OPTIONS] COMMAND [ARGS]...

Options: --help Show this message and exit.

Commands: bake.election_file Write election and candidate data with on... bake.results_status_json Output a JSON file describing available... bake.state_file Write election and candidate data along with... cache.clear Delete files in state cache diretory cache.files List files in state cache diretory datasource.elections List elections for a state. datasource.filename_url_pairs List mapping of standard filenames to source... datasource.mappings List metadata mappings for a state datasource.target_urls List source data urls for a state fetch Scrape data files and store in local file... load.run Load cached data files into the database load_metadata.run Populate metadata in database from fixture... publish Publish baked result files shell Open a Python shell, bootstrapping the... transform.list Show available data transformations transform.reverse Reverse a previously run transformation transform.run Run data transformations validate.list Show available validations for state validate.run Run data validations for state ```

Running commands looks something like this:

bash $ openelex cache.clear --state=NY 0 files deleted 0 files still in cache

You can also get help on particular commands, e.g. openelex --help cache.clear.

Configuring services (optional)

If you are going to load results, you need to configure MongoDB connections in openelex/settings.py.

If you are a core contributor who needs to publish baked results to one of the GitHub repositories, you will need to define further settings.

To set GitHub credentials, you must first create a personal access token and then uncomment and set these values in openelex/settings.py:

python GITHUB_USERNAME = '' GITHUB_ACCESS_TOKEN = ''

Load party and office metadata (optional)

You only need to do this if you plan to write data loaders or transforms.

bash $ cd openelex $ openelex load_metadata.run --collection=office $ openelex load_metadata.run --collection=party

Issues

Bump ipython from 0.13.2 to 7.16.3

opened on 2022-01-21 18:56:11 by dependabot[bot]

Bumps ipython from 0.13.2 to 7.16.3.

Release notes

Sourced from ipython's releases.

7.9.0

No release notes provided.

7.8.0

No release notes provided.

7.7.0

No release notes provided.

7.6.1

No release notes provided.

7.6.0

No release notes provided.

7.5.0

No release notes provided.

7.4.0

No release notes provided.

7.3.0

No release notes provided.

7.2.0

No release notes provided.

7.1.1

No release notes provided.

7.1.0

No release notes provided.

7.0.1

No release notes provided.

7.0.0

No release notes provided.

7.0.0-doc

No release notes provided.

7.0.0rc1

No release notes provided.

7.0.0b1

No release notes provided.

6.2.1

No release notes provided.

... (truncated)

Commits

d43c7c7 release 7.16.3
5fa1e40 Merge pull request from GHSA-pq7m-3gw7-gq5x
8df8971 back to dev
9f477b7 release 7.16.2
138f266 bring back release helper from master branch
5aa3634 Merge pull request #13341 from meeseeksmachine/auto-backport-of-pr-13335-on-7...
bcae8e0 Backport PR #13335: What's new 7.16.2
8fcdcd3 Pin Jedi to <0.17.2.
2486838 release 7.16.1
20bdc6f fix conda build
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/openelections/openelections-core/network/alerts).

Provide data in the NIST common data format

opened on 2021-11-08 17:37:41 by sfsinger19103

The National Institute of Standards and Technology has developed a common data format for election results. The more people (and election agencies) use this format, the easier it will be for all of us to share data.

Software to ease conversion to the common data format is here.

Include the territories

opened on 2021-11-08 17:34:02 by sfsinger19103

Several million Americans live and vote in the five US territories. They deserve to be included in your election results, even if they have only non-voting representation in Congress.

For the history of the exclusion of these people -- most of whom are not white -- from power in the United States, see How to Hide an Empire by Immerwahr.

Make it easy for users to see what data is currently available clean for each jurisdiction

opened on 2021-11-08 17:30:13 by sfsinger19103

Last time I checked, some of your datasets were works in progress, rather than complete. It would be great to have a resource that a visitor to the repo could find, detailing exactly what is available completely -- for each state, district, territory, and for each election:

what contests results are available?
at what geographical level?
are vote types (e.g., election day, by-mail, vote-center) broken out?
is cleaning & checking complete?

Bump urllib3 from 1.25.7 to 1.25.8

opened on 2021-04-30 20:39:15 by dependabot[bot]

Bumps urllib3 from 1.25.7 to 1.25.8.

Release notes

Sourced from urllib3's releases.

1.25.8

Release: 1.25.8

Changelog

Sourced from urllib3's changelog.

1.25.8 (2020-01-20)

Drop support for EOL Python 3.4 (Pull #1774)

Optimize _encode_invalid_chars (Pull #1787)

Commits

2a57bc5 Release 1.25.8 (#1788)
a2697e7 Optimize _encode_invalid_chars (#1787)
d2a5a59 Move IPv6 test skips in server fixtures
d44f0e5 Factorize test certificates serialization
84abc7f Generate IPV6 certificates using trustme
6a15b18 Run IPv6 Tornado server from fixture
4903840 Use trustme to generate IP_SAN cert
9971e27 Empty responses should have no lines.
62ef68e Use trustme to generate NO_SAN certs
fd2666e Use fixture to configure NO_SAN test certs
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Potential broken approach of reading csv files

opened on 2021-04-03 22:54:57 by warwickmm

The following pattern is used often to read csv files: with open(filename, 'rU') as csvfile: reader = unicodecsv.DictReader(csvfile) I think this worked in python2 since the str and bytes types were synonymous. However, this breaks in python3 since unicodecsv expects the file to be opened in binary mode, which it is not.

For example, the following fails in python3 with the error AttributeError: 'str' object has no attribute 'decode' ``` import unicodecsv

filename = 'openelex/us/md/mappings/md.csv' with open(filename, "r") as data: reader = unicodecsv.DictReader(data) for row in reader: print(row) Using `csv` instead of `unicodecsv` fixes the issue. import csv

filename = 'openelex/us/md/mappings/md.csv' with open(filename, "r") as data: reader = csv.DictReader(data) for row in reader: print(row) ```

Is there something wrong with my setup, or is this broken for other people as well?

OpenElections

The goal of OpenElections is to create the first free, comprehensive, standardized, linked set of election results data for the United States.

GitHub Repository

openelections elections