{"id":15284117,"url":"https://github.com/jehiah/socrata_to_bigquery","last_synced_at":"2026-03-05T01:02:44.590Z","repository":{"id":57581359,"uuid":"214714894","full_name":"jehiah/socrata_to_bigquery","owner":"jehiah","description":"A tool to copy public data to BigQuery","archived":false,"fork":false,"pushed_at":"2025-04-17T01:45:10.000Z","size":146,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-07T03:39:33.025Z","etag":null,"topics":["bigquery","opendata","socrata"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jehiah.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-10-12T21:03:00.000Z","updated_at":"2025-04-17T01:45:07.000Z","dependencies_parsed_at":"2024-01-16T21:30:10.696Z","dependency_job_id":"a055ea9e-045b-4fd8-a149-3061bad9b891","html_url":"https://github.com/jehiah/socrata_to_bigquery","commit_stats":{"total_commits":46,"total_committers":2,"mean_commits":23.0,"dds":"0.021739130434782594","last_synced_commit":"dba9ea4a37eb1cd233167b46213e0a460bd0ae09"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/jehiah/socrata_to_bigquery","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jehiah%2Fsocrata_to_bigquery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jehiah%2Fsocrata_to_bigquery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jehiah%2Fsocrata_to_bigquery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jehiah%2Fsocrata_to_bigquery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jehiah","download_url":"https://codeload.github.com/jehiah/socrata_to_bigquery/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jehiah%2Fsocrata_to_bigquery/sbom","scorecard":{"id":514535,"data":{"date":"2025-08-11","repo":{"name":"github.com/jehiah/socrata_to_bigquery","commit":"891834805b593d0222804c22dffeba50b41af209"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.1,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/24 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: topLevel 'contents' permission set to 'write': .github/workflows/release.yaml:6","Warn: topLevel 'packages' permission set to 'write': .github/workflows/release.yaml:7","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yaml:14: update your workflow using https://app.stepsecurity.io/secureworkflow/jehiah/socrata_to_bigquery/release.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yaml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/jehiah/socrata_to_bigquery/release.yaml/master?enable=pin","Info:   0 out of   1 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v0.0.2 not signed: https://api.github.com/repos/jehiah/socrata_to_bigquery/releases/191929696","Warn: release artifact v0.0.1 not signed: https://api.github.com/repos/jehiah/socrata_to_bigquery/releases/121055068","Warn: release artifact v0.0.2 does not have provenance: https://api.github.com/repos/jehiah/socrata_to_bigquery/releases/191929696","Warn: release artifact v0.0.1 does not have provenance: https://api.github.com/repos/jehiah/socrata_to_bigquery/releases/121055068"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 6 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-20T01:27:59.919Z","repository_id":57581359,"created_at":"2025-08-20T01:27:59.919Z","updated_at":"2025-08-20T01:27:59.919Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30104218,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T00:38:46.881Z","status":"ssl_error","status_checked_at":"2026-03-05T00:38:45.829Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","opendata","socrata"],"created_at":"2024-09-30T14:49:50.374Z","updated_at":"2026-03-05T01:02:44.525Z","avatar_url":"https://github.com/jehiah.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# socrata_to_bigquery\n\nThis tool facilitates replicating Open Data from the [Socrata Platform](https://socrata.com/) to [Google BigQuery](https://cloud.google.com/bigquery/)\n\n\u003cdiv style=\"color:red\"\u003e\u003cstrong\u003eWARNING:\u003c/strong\u003e This is Alpha Release Software. It might be useful, but it will be rough around the edges\u003c/div\u003e\n\nMany Governemnt Open-Data projects are hosted on Socrata, and searchable through the [Open Data Network](https://www.opendatanetwork.com/)\n\n* https://opendata.cityofnewyork.us/\n* https://data.ny.gov/\n* etc...\n\n## Installing\n\n```bash\ngo get github.com/jehiah/socrata_to_bigquery/...\n```\n\n## Quick Start\n\n1. `socrata_to_bigquery init`\n\n2. `socrata_to_bigquery download`\n\n3. `socrata_to_bigquery sync`\n\n\n## Documentation\n\n### `init`\n\n`init` initializes a yaml config file for synchronizing a Socrata dataset to BigQuery. \n\nUsage: `init -api-endpoint=https://path/to/api [-project-id -bq-dataset]`\n\ni.e. `socrata_to_bigquery init -api-endpoint=https://data.cityofnewyork.us/resource/nc67-uf89 -data-dir=/path/to/data`\n\nAPI endpoint is the published Socrata API endpoint for a dataset.\n\nThis config file defines all fields that will be loaded to BigQuery, and the target bigquery project and dataset. Optionally it can defines custom conversion from TEXT socrata field to richer DATE or TIME field types. It also defines the target bigquery field names.\n\nFor example, this `issue_date` is a `\"text\"` format in Socrata but it will be parsed using the Go format string `\"01/02/2006\"` and stored in a `DATE` column. `on_error = \"SKIP_ROW\"` indicates that any rows that do not meet this date format will be skipped.\n\n```\n  [schema.issue_date]\n    bigquery_type = \"DATE\"\n    description = \"Issue Date\"\n    # example_values = \"\\\"03/06/2017\\\", \\\"10/07/2017\\\", \\\"05/29/2016\\\"\"\n\n    # SKIP_VALUE | SKIP_ROW | ERROR \n    on_error = \"SKIP_ROW\"\n    required = true\n    source_field = \"issue_date\"\n    source_field_type = \"text\"\n\n    # the time.Parse format string\n    time_format = \"01/02/2006\"\n```\n\n\n```\nUsage of socrata_to_bigquery init:\n  -api-endpoint string\n    \tThe URL to the socrata dataset\n  -bq-dataset string\n    \tBigQuery Dataset\n  -data-dir string\n    \tdirectory to create config file in\n  -debug\n    \tshow debug output\n  -filename string\n    \tdefaults to ${NAME}-${ID}.toml\n  -project-id string\n    \tGoogle Cloud Project ID\n  -socrata-app-token string\n    \tSocrata App Token (also src SOCRATA_APP_TOKEN env)\n```\n\n### `download`\n\nDownload does an initial copy from Socrata to Bigquery\n\nUsage: `socrata_to_bigquery download /path/to/config.yaml`\n\ni.e. `socrata_to_bigquery download open-parking-and-camera-violations-nc67-uf89.toml`\n\n### `sync`\n\nSync does a periodic copy of new records from Socrata to BigQuery copying only new records since the most recent record in BigQuery.\n\nUsage: `socrata_to_bigquery sync /path/to/config.yaml`\n\ni.e. `socrata_to_bigquery sync open-parking-and-camera-violations-nc67-uf89.toml`\n\n## Setup\n\nSocrata API Token\n\nhttps://dev.socrata.com/docs/authentication.html\n\n```bash\nexport SOCRATA_APP_TOKEN=...\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjehiah%2Fsocrata_to_bigquery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjehiah%2Fsocrata_to_bigquery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjehiah%2Fsocrata_to_bigquery/lists"}