{"id":17283727,"url":"https://github.com/prettymuchbryce/ingestr","last_synced_at":"2025-07-25T01:06:47.773Z","repository":{"id":66163061,"uuid":"217601964","full_name":"prettymuchbryce/ingestr","owner":"prettymuchbryce","description":"Ingestr is a Go microservice that consumes Ethereum blocks for enqueing into SNS and storage in S3","archived":false,"fork":false,"pushed_at":"2019-12-30T21:34:26.000Z","size":174,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-24T11:24:00.273Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prettymuchbryce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-25T19:29:10.000Z","updated_at":"2019-12-30T21:34:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"84176d45-c901-4205-a5e9-07c9683d98d7","html_url":"https://github.com/prettymuchbryce/ingestr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/prettymuchbryce/ingestr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prettymuchbryce%2Fingestr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prettymuchbryce%2Fingestr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prettymuchbryce%2Fingestr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prettymuchbryce%2Fingestr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prettymuchbryce","download_url":"https://codeload.github.com/prettymuchbryce/ingestr/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prettymuchbryce%2Fingestr/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266936385,"owners_count":24009407,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T09:52:04.552Z","updated_at":"2025-07-25T01:06:47.736Z","avatar_url":"https://github.com/prettymuchbryce.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ingestr\n\nIngestr is a Go microservice that consumes Ethereum blocks for enqueing into SNS and storage in S3.\n\n![My image](https://prettymuchbryce.s3-us-west-1.amazonaws.com/ingestr.png)\n  \n### Description\n\nIngestr pulls new blocks out of an Ethereum node (Geth, Parity, Infura, etc), and publishes the block number to SNS, as well as caches it in a gzipped form in S3.\n\nIngestr is useful for systems that need to:\n  * Actively ingest new Ethereum blocks to perform some processing\n  * Backfill historical blocks\n  * Want to use the same code path to do both of these things\n  \nIngestr requires redis to store some information about the blocks it is working on. It is safe to run multiple ingestr instances in parallel with each other (if you're into that sort of thing), however your biggest bottleneck is likely to be throughput from your Ethereum node.\n\nFor this reason, Ingestr caches all blocks in S3 so that on subsequent runs, blocks can be fetched from there instead.\n\nAlthough Ingestr enqueues block numbers into SNS when work is finished, it's expected that downstream consumers of these events will fetch (and gunzip) the blocks directly from S3.\n  \n### Configuration\n\nPlease see the [Environment Variables](https://github.com/prettymuchbryce/ingestr/blob/master/.env).\n\n### Requirements\n* An Ethereum node (or Infura, or whatever)\n* A redis instance\n* An AWS IAM role with permission to read/write from an S3 bucket, and an SNS topic\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprettymuchbryce%2Fingestr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprettymuchbryce%2Fingestr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprettymuchbryce%2Fingestr/lists"}