{"id":13767306,"url":"https://github.com/gskril/farcaster-indexer","last_synced_at":"2025-07-20T12:32:57.119Z","repository":{"id":37488601,"uuid":"502527190","full_name":"gskril/farcaster-indexer","owner":"gskril","description":"Save all data from the Farcaster protocol to Postgres","archived":false,"fork":false,"pushed_at":"2024-05-14T20:04:48.000Z","size":689,"stargazers_count":153,"open_issues_count":6,"forks_count":66,"subscribers_count":5,"default_branch":"hubs","last_synced_at":"2025-05-20T14:06:44.273Z","etag":null,"topics":["farcaster"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gskril.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-12T05:27:47.000Z","updated_at":"2025-03-27T18:38:02.000Z","dependencies_parsed_at":"2023-01-29T18:45:43.149Z","dependency_job_id":"72cf62f7-b131-441c-b591-a68e17935434","html_url":"https://github.com/gskril/farcaster-indexer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gskril/farcaster-indexer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gskril%2Ffarcaster-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gskril%2Ffarcaster-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gskril%2Ffarcaster-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gskril%2Ffarcaster-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gskril","download_url":"https://codeload.github.com/gskril/farcaster-indexer/tar.gz/refs/heads/hubs","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gskril%2Ffarcaster-indexer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266127212,"owners_count":23880420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["farcaster"],"created_at":"2024-08-03T16:01:07.269Z","updated_at":"2025-07-20T12:32:57.102Z","avatar_url":"https://github.com/gskril.png","language":"TypeScript","funding_links":[],"categories":["Developer Resources","TypeScript","Uncategorized"],"sub_categories":["Repos","Uncategorized"],"readme":"# Farcaster Indexer\n\nThis is an indexer that listens for messages from a [Farcaster Hub](https://docs.farcaster.xyz/learn/architecture/hubs) and inserts relevant data into a postgres database.\n\nThe most performant way to run this is to co-locate everything (hub, node app, postgres, redis) on the same machine. I recommend [Latitude](https://www.latitude.sh/r/673C7DB2) (referral code for $200 of free credits).\n\n## How to run\n\nClone this repo\n\n```bash\ngit clone -b hubs https://github.com/gskril/farcaster-indexer.git\n```\n\nInstall dependencies\n\n```bash\nyarn install\n```\n\nCreate a `.env` file with your hub, database, and redis connection details\n\n```bash\ncp .env.example .env\n```\n\nRun the latest database migrations\n\n```bash\nyarn kysely:migrate\n```\n\nRun the indexer\n\n```bash\n# Recommended to get the full state. You only need to run this once.\n# Streaming will start after the backfill is complete.\nyarn run backfill\n\n# Ignores backfill and start streaming from the latest recorded event.\n# You should run this after one initial backfill.\nyarn start\n```\n\n## How it works\n\n- Backfill and streaming are separate processes.\n- Every operation is run through [BullMQ](https://bullmq.io/) for better concurrency and error handling.\n- For backfill, the indexer adds all FIDs (in batches of 100) to a queue and processes them in parallel. The `WORKER_CONCURRENCY` environment variable controls how many workers are spawned.\n- Once backfill is complete, the indexer subscribes to a hub's event stream and processes messages as they arrive. BullMQ is used as middleware to ensure that hub events are getting handled fast enough, otherwise the stream will disconnect.\n\n## Extras\n\nIf you want to add search functionality, you can manually apply the SQL migration at [src/db/search-migrations.sql](./src/db/search-migrations.sql)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgskril%2Ffarcaster-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgskril%2Ffarcaster-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgskril%2Ffarcaster-indexer/lists"}