{"id":19778399,"url":"https://github.com/evancarroll/db-texas-ethics-commission","last_synced_at":"2025-09-14T23:57:00.112Z","repository":{"id":53890610,"uuid":"134624787","full_name":"EvanCarroll/db-Texas-Ethics-Commission","owner":"EvanCarroll","description":"A schema loader for the Texas Ethics Commission","archived":false,"fork":false,"pushed_at":"2024-08-19T03:39:39.000Z","size":2946,"stargazers_count":12,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-08-27T07:51:53.842Z","etag":null,"topics":["campaign-contributions","campaign-finance","campaign-finance-data","data-import","government-data","lobbying","open-data","postgresql","public-data","texas"],"latest_commit_sha":null,"homepage":"https://www.ethics.state.tx.us/","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EvanCarroll.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-23T20:58:19.000Z","updated_at":"2025-07-22T15:32:11.000Z","dependencies_parsed_at":"2024-08-19T03:34:16.211Z","dependency_job_id":"9aac0223-a135-4ee3-a8d0-b79abbf0bfb8","html_url":"https://github.com/EvanCarroll/db-Texas-Ethics-Commission","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EvanCarroll/db-Texas-Ethics-Commission","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanCarroll%2Fdb-Texas-Ethics-Commission","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanCarroll%2Fdb-Texas-Ethics-Commission/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanCarroll%2Fdb-Texas-Ethics-Commission/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanCarroll%2Fdb-Texas-Ethics-Commission/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EvanCarroll","download_url":"https://codeload.github.com/EvanCarroll/db-Texas-Ethics-Commission/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanCarroll%2Fdb-Texas-Ethics-Commission/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275185371,"owners_count":25419919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-14T02:00:10.474Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["campaign-contributions","campaign-finance","campaign-finance-data","data-import","government-data","lobbying","open-data","postgresql","public-data","texas"],"created_at":"2024-11-12T05:29:14.817Z","updated_at":"2025-09-14T23:57:00.051Z","avatar_url":"https://github.com/EvanCarroll.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"Texas Ethics Commission - Schema Loader\n====\n\nThis is a [PostgreSQL schema loader](https://www.postgresql.org/) for the data\nprovided by [Texas Ethics Commission](https://www.ethics.state.tx.us/).\n\nWe provide a utility to\n\n* extract schema from [PDSERF/Plus format](https://dba.stackexchange.com/a/207425/2639) export by reading the ReadMe files and parse them to determine the schema, types, and keys and constraints.\n* create the tables needed to load up the 1295 certs -- these are hand written from the pdf documentation by provided by TEC\n* load the data up from csv format into the tables we create\n\nInternally, lines from PDSERF readme are either,\n\n * Table Description rows\n * Column Description-cotd rows\n * Column rows\n * Start-rows for table (Start with \"Record #:\")\n * End-rows for table   (Containing just a `-`)\n\nColumn lines are either\n\n * Indented as part of a group (array) replicated a certain amount of times\n * Derived from a \"single line\"\n\nAll data is loaded up into PostgreSQL, including the Descriptions which we pull\ndown as\n[`COMMENTS`](https://www.postgresql.org/docs/current/static/sql-comment.html).\n\nYou can find the readmes from the Texas Ethics Commission added in this project here:\n\n* [Campaign Finance](https://ethics.state.tx.us/search/cf/)\n\t* [ReadMe](https://ethics.state.tx.us/data/search/cf/CFS-ReadMe.txt) [cache in the repo](./data/TEC_CF_CSV/ReadMe.txt)\n\t* [Codes](https://ethics.state.tx.us/data/search/cf/CFS-Codes.txt)\n\t* [CSV Record Format - Old Reports](https://ethics.state.tx.us/data/search/cf/TX_ERF13_7.pdf) [cache in the repo](./data/tec_docs/TX_ERF13_7.pdf)\n\t* [CSV Record Format - New Reports](https://ethics.state.tx.us/data/search/cf/CampaignFinanceCSVFileFormat.pdf)\n* [Lobby Database](https://ethics.state.tx.us/search/lobby/)\n\t* [ReadMe](https://ethics.state.tx.us/data/search/lobby/LobbyLAR-ReadMe.txt) [cache in the repo](./data/TEC_LA_CSV/LobbyLAR-ReadMe.txt)\n* [Form 1295](https://www.ethics.state.tx.us/search/1295/)\n\t* [CSV Record Format](https://www.ethics.state.tx.us/data/search/1295/1295CertificatesCSVFormat.pdf) [cache in the repo](./data/tec_docs/1295CertificatesCSVFormat.pdf)\n\t* [1295 Online Form](https://www.ethics.state.tx.us/data/forms/1295/1295.pdf)\n\nCoverage\n====\n\nThis module loads has full coverage of the meta-data, and data of the TEC.\n\n* Lobby Reports (tables `l_*`)\n    ```text\n    tec.l_awardmementodata         tec.l_foodbeveragedata\n    tec.l_coversheetladata         tec.l_giftdata\n    tec.l_docketdata               tec.l_individualreportingdata\n    tec.l_entertainmentdata        tec.l_subjectmatterdata\n    tec.l_eventdata                tec.l_transportationdata\n    ```\n* Campaign Finance Reports (tables `c_*`)\n    ```text\n    tec.c_assetdata         tec.c_creditdata        tec.c_finaldata\n    tec.c_candidatedata     tec.c_debtdata          tec.c_loandata\n    tec.c_contributiondata  tec.c_expendcategory    tec.c_pledgedata\n    tec.c_coversheet1data   tec.c_expenddata        tec.c_spacdata\n    tec.c_coversheet2data   tec.c_expendrepayment   tec.c_traveldata\n    tec.c_coversheet3data   tec.c_filerdata\n    ```\n* 1295 Certs\n    ```text\n    tec.form1295_box123            tec.form1295_interested_party\n    ```\n\n\nLinks\n====\n\n* [TEC data](https://www.ethics.state.tx.us/dfs/search_CF.htm).\n* [PDSERF](https://dba.stackexchange.com/a/207425/2639) format\n\nInstallation\n====\n\nRequirements: PostgreSQL, git, curl\n\nRepo download and database setup (example in bash):\n\n```shell\n$ git clone https://github.com/EvanCarroll/db-Texas-Ethics-Commission.git\n$ cd ./db-Texas-Ethics-Commission\n$ make\n$ createdb mydb\n$ psql -d mydb -f ./runme.sql 2\u003e\u00261 | tee out.log\n$ make clean\n```\n\nBackground\n====\n\nCreated at [Houston Hackathon 2018](http://houstonhackathon.com/) as the sole\nwork of Evan Carroll.\n\nLicense\n====\n\nIf you use this, open source all (100%) of your stuff, or I'll litigate.\nThe GPL is not the AGPL. Please read, and be advised:\n\n[GNU Affero General Public License\nv3](https://www.gnu.org/licenses/agpl-3.0.html), see included\n[LICENSE.md](./LICENSE.md)\n\nContact\n====\n\nContact Evan Carroll 281.901.0011 for a quote on development.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevancarroll%2Fdb-texas-ethics-commission","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevancarroll%2Fdb-texas-ethics-commission","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevancarroll%2Fdb-texas-ethics-commission/lists"}