https://github.com/publici/fec-loader
Loads raw FEC filings into a database
https://github.com/publici/fec-loader
campaignfinance elections etl fec node
Last synced: 3 months ago
JSON representation
Loads raw FEC filings into a database
- Host: GitHub
- URL: https://github.com/publici/fec-loader
- Owner: PublicI
- License: mit
- Created: 2016-05-21T21:35:05.000Z (over 9 years ago)
- Default Branch: subcommands
- Last Pushed: 2023-01-06T02:02:43.000Z (almost 3 years ago)
- Last Synced: 2024-04-14T12:19:37.178Z (over 1 year ago)
- Topics: campaignfinance, elections, etl, fec, node
- Language: JavaScript
- Homepage:
- Size: 730 KB
- Stars: 19
- Watchers: 2
- Forks: 9
- Open Issues: 29
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# fec-cli
A set of flexible command line utilities designed to discover, convert and load raw FEC filings into a database in a fast, streaming manner.
`fec` is about nine times faster than similar solutions. For example, on a recent MacBook Air, a 2.3 gigabyte ActBlue filing parses in three minutes instead of 23 minutes.
It requires [Node](https://nodejs.org/). It uses [fec-parse](https://github.com/PublicI/fec-parse).
## Try
To try converting a filing to newline-separated JSON without installing fec-loader, paste the following into a terminal:
```bash
FILING_ID=1283013; curl -s "https://docquery.fec.gov/dcdev/posted/"$FILING_ID".fec" | npx -p github:PublicI/fec-loader#subcommands convert $FILING_ID > $FILING_ID".ndjson"
```
## Install
To install:
```bash
npm install -g github:PublicI/fec-loader#subcommands
```
## Setup
To set up a Postgres database for FEC filings and the environment variables needed to connect:
```bash
export PGHOST= PGDATABASE= PGUSER= PGPASSWORD=
fec init
```
## Use
To load a filing from the FEC into a Postgres database, run:
```bash
export PGHOST= PGDATABASE= PGUSER= PGPASSWORD=
FILING_ID=1283013; curl -s "https://docquery.fec.gov/dcdev/posted/"$FILING_ID".fec" | fec convert $FILING_ID --format=psql | psql
```
To list the filings available from the FEC's RSS feed run:
```bash
fec list --rss
```
To load the most recent five filings from the FEC's RSS feed, run:
```bash
for url in $(fec list --rss --headers=false --columns=fec_url --format=tsv | head -n 5); do FILING_ID=$(echo $url | tr -dc '0-9'); curl -s "https://docquery.fec.gov/dcdev/posted/"$FILING_ID".fec" | fec convert $FILING_ID --format=psql | psql -v ON_ERROR_STOP=on --single-transaction; done
```
To get just a summary line as JSON:
```bash
FILING_ID=1283013; curl -s "https://docquery.fec.gov/dcdev/posted/"$FILING_ID".fec" | head -n 10 | fec convert | sed -n 2p
```