yet another foia automation service

bettergov, updated 🕥 2022-07-06 19:38:00

FOIAmail

foiamail art by lucas ian smith

A framework for mass FOIA campaigns. Distribute public records requests, manage responses and organize file attachments using GMail, Sheets and Drive.

Technically, it's a bunch of a API calls on cron.

First-time technical setup

First, set up your server, environment, credentials and FOIA template (see setting up).

Once the server, environment, repository and credentials are set up, use mgr.py to initialize the application.

Load contacts directly in Gmail via the import contacts CSV option. See docs/technical.md for detailed walkthrough of the full FOIAmail workflow for importing contacts.

When your contacts are loaded, build agency labels for your contacts:

bash python mgr.py --build-labels

Next, prepare drafts for your contacts. You can look through the drafts and find any errors before you send them out:

bash python mgr.py --build-drafts

Once everything looks good, you can send them out with this command:

bash python mgr.py --send-drafts

(The --build-drafts and --send-drafts will re-create the drafts each time, so make sure you delete them when prompted.)

Monitoring Responses

There are two ways to run the monitoring system for tagging/categorizing incoming emails, building reports and downloading responsive attachments: traditional VM using cron and Docker.

Traditional VM/Cron

Install a crontab to run the following tasks at regular intervals: - mgr.py --label, to label incoming messages by agency and status (every few minutes) - mgr.py --atts, to migrate GMail attachments to Drive (nightly) - mgr.py --report, to generate a status report of agency responses in Google Sheets (nightly, after attachments migrate)

This is a good time to double-check the server timezone is set to America/Chicago. See technical docs.

Docker

There's a Dockerfile for running the services. If you have make you can simply run:

bash make docker_start

Or you can do it yourself:

bash docker build . -t foiamail docker run --mount source=foiamail_logs,target=/home/ubuntu/foiamail/log/logs -t foiamail

Ongoing manual work

Verify draft FOIA messages before sending, including message count, contents, recipients and labels

Routine checklist for operating FOIAMail: - Monitor incoming email regularly: - respond to questions - identify issues with responses - report any problems - Apply agency labels where automation fails - Label messages '*done' after determining an agency has responded with an acceptable data file - All attachments belonging to this agency's emails will migrate to Drive overnight - Label messages '*NA' for agencies that no longer exist or have no employees - Throughout the project, particularly after the FOIA deadline has passed, use the response report to identify non-responsive agencies and go nudge them - Update contact information via the GMail interface as necessary

Technical docs

See docs/technical.md for detailed explanations of commands.

Issues

Future Roadmap

opened on 2022-09-25 00:17:54 by brandonrobertz

Hey! So I want to put some work into this as I still think FOIAmail is super compelling. Is there any interest on anyone else's end?

I think a good first move might be to kind of sketch out what work needs to happen and what the next version of the system would look like.

My understanding is this (fro https://github.com/bettergov/foiamail/issues/67): - setup push notifications to handle GMail events - use cloud functions to handle pushes - use workspace add-ons to provide UI for configuration + other screens

Some other features that I think would be super useful: - automatically tag new emails based on contacts list, previous email threads and request IDs - add project tags (so we can manage more than one project per GMail account)

Maybe there's some intermediate version of all this that could get FOIAmail back to working managing an inbox, as well?

Bump lxml from 4.5.1 to 4.9.1

opened on 2022-07-06 19:37:56 by dependabot[bot]

Bumps lxml from 4.5.1 to 4.9.1.

Changelog

Sourced from lxml's changelog.

4.9.1 (2022-07-01)

Bugs fixed

A crash was resolved when using iterwalk() (or canonicalize()) after parsing certain incorrect input. Note that iterwalk() can crash on valid input parsed with the same parser after failing to parse the incorrect input.

4.9.0 (2022-06-01)

Bugs fixed

GH#341: The mixin inheritance order in lxml.html was corrected. Patch by xmo-odoo.

Other changes

Built with Cython 0.29.30 to adapt to changes in Python 3.11 and 3.12.

Wheels include zlib 1.2.12, libxml2 2.9.14 and libxslt 1.1.35 (libxml2 2.9.12+ and libxslt 1.1.34 on Windows).

GH#343: Windows-AArch64 build support in Visual Studio. Patch by Steve Dower.

4.8.0 (2022-02-17)

Features added

GH#337: Path-like objects are now supported throughout the API instead of just strings. Patch by Henning Janssen.

The ElementMaker now supports QName values as tags, which always override the default namespace of the factory.

Bugs fixed

GH#338: In lxml.objectify, the XSI float annotation "nan" and "inf" were spelled in lower case, whereas XML Schema datatypes define them as "NaN" and "INF" respectively.

... (truncated)

Commits

d01872c Prevent parse failure in new test from leaking into later test runs.
d65e632 Prepare release of lxml 4.9.1.
86368e9 Fix a crash when incorrect parser input occurs together with usages of iterwa...
50c2764 Delete unused Travis CI config and reference in docs (GH-345)
8f0bf2d Try to speed up the musllinux AArch64 build by splitting the different CPytho...
b9f7074 Remove debug print from test.
b224e0f Try to install 'xz' in wheel builds, if available, since it's now needed to e...
897ebfa Update macOS deployment target version from 10.14 to 10.15 since 10.14 starts...
853c9e9 Prepare release of 4.9.0.
d3f77e6 Add a test for https://bugs.launchpad.net/lxml/+bug/1965070 leaving out the a...
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/bettergov/foiamail/network/alerts).

build a UI

opened on 2022-03-15 19:25:09 by mattkiefer

selecting a template, uploading a contact list and hitting send could all be managed via a gmail plugin interface, instead of command line

validate outgoing mail

opened on 2022-02-09 16:06:24 by mattkiefer

e.g. guardrails to make sure sent mail is labeled, etc.

Bump ipython from 5.3.0 to 7.16.3

opened on 2022-01-21 19:29:36 by dependabot[bot]

Bumps ipython from 5.3.0 to 7.16.3.

Commits

d43c7c7 release 7.16.3
5fa1e40 Merge pull request from GHSA-pq7m-3gw7-gq5x
8df8971 back to dev
9f477b7 release 7.16.2
138f266 bring back release helper from master branch
5aa3634 Merge pull request #13341 from meeseeksmachine/auto-backport-of-pr-13335-on-7...
bcae8e0 Backport PR #13335: What's new 7.16.2
8fcdcd3 Pin Jedi to <0.17.2.
2486838 release 7.16.1
20bdc6f fix conda build
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Add an option to exclude PDF attachment in email

opened on 2021-07-27 18:55:38 by robroc

By default, FOIAmail adds a PDF attachment of the request text as a PDF named records-request.pdf (Line 281 in compose.py) It would be good to turn off this option in the congif.yaml file, and also to be able to rename the PDF if we choose to include it.

Better Government Association

GitHub Repository