{"id":21005213,"url":"https://github.com/matthiaszepper/nfcore_stats_backend","last_synced_at":"2026-04-08T16:01:34.139Z","repository":{"id":51028815,"uuid":"515309236","full_name":"MatthiasZepper/nfcore_stats_backend","owner":"MatthiasZepper","description":"An application to gather statistics for the nf-core community, build with FastAPI, Pydantic, Celery and SQLAlchemy.","archived":false,"fork":false,"pushed_at":"2022-09-09T19:23:05.000Z","size":144,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-05T10:37:10.270Z","etag":null,"topics":["celery","docker","docker-compose","fastapi","microservices","nf-core","postgresql","pydantic","redis","sqlmodel"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MatthiasZepper.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-18T19:06:58.000Z","updated_at":"2022-08-09T16:17:50.000Z","dependencies_parsed_at":"2023-01-18T02:30:54.265Z","dependency_job_id":null,"html_url":"https://github.com/MatthiasZepper/nfcore_stats_backend","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MatthiasZepper/nfcore_stats_backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatthiasZepper%2Fnfcore_stats_backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatthiasZepper%2Fnfcore_stats_backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatthiasZepper%2Fnfcore_stats_backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatthiasZepper%2Fnfcore_stats_backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MatthiasZepper","download_url":"https://codeload.github.com/MatthiasZepper/nfcore_stats_backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatthiasZepper%2Fnfcore_stats_backend/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31562697,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","docker","docker-compose","fastapi","microservices","nf-core","postgresql","pydantic","redis","sqlmodel"],"created_at":"2024-11-19T08:39:47.630Z","updated_at":"2026-04-08T16:01:34.116Z","avatar_url":"https://github.com/MatthiasZepper.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nf-core stats backend\n\nThe new _nf-core stats backend_ provides simple-to-use REST and GraphQL web services to query/retrieve statistics and metadata about the [nf-core community](https://nf-co.re) and their pipelines. The data is daily aggregated from Github, Twitter, Slack and Youtube. It’s designed with simplicity and performance emphasized.\n\n## Development\n\n### Roadmap\n\n- [x] Scaffold initial project and repo structure: Poetry setup and Docker compose file with containers for the services.\n- [x] Decide on the tech stack to use: FastAPI, SQLModel, Pydantic, Celery, PostgreSQL and Redis.\n- [x] Write a first simple, scheduled task as demo: Uptime checker for nf-co.re\n- [x] Write a first API to retrieve the uptime status of nf-co.re as demo.\n- [x] _Derive data models and suitable database table structure (Work in progress: 1/4 done)_\n- [x] _Write CRUD logic for the various data types and sources (Work in progress: 1/4 done)_\n- [x] Include api.routers and split endpoints to subfiles.\n- [ ] Write scheduled tasks to interact with Github API, Twitter API and Slack API to gather stats and other information.\n- [ ] Ingest output of the schedulers into the database.\n- [ ] Write REST APIs to retrieve the data.\n- [ ] Write GraphQL APIs to retrieve the data.\n- [ ] Add authentication to the endpoints.\n- [ ] Write documentation.\n- [ ] Include convenience functions, e.g. the ability to add new domains or accounts to monitor via API calls.\n- [ ] Write tests.\n- [ ] Integrate and configure alembic for database migrations?\n\n### Debugging\n\nTo enable debugging code, the container _nfcore_stats_api_ has set `stdin_open` and `tty` true, such that one can attach a terminal to the container. This is most useful in conjunction with `set_trace()`. Put\n\n```python\nimport pdb; pdb.set_trace()\n```\n\nanywhere within the body of a function. If that function is executed, you will be able to step through every command and also interactively explore the variables. To do so, you need to first attach a new terminal to the API container\n\n```bash\ndocker container attach nfcore_stats_api\n```\n\nand then send requests to the API to trigger the function execution.\n\n### Importing existing data into the database\n\nThe new backend has dedicated APIs meant to import the existing JSON files scraped by the current website. To import those\nto the database, navigate into the folder containing the existing json files and send them as request bodies to the respective endpoints:\n\n```bash\ncd /path/to/your/json/files\ncurl --data-binary \"@pipelines.json\" -H \"Content-Type: application/json\" -X PUT http://localhost:8000/import/pipelines\n```\n\nMind the `@` symbol preceding the file name. You can also specify `--data-binary \"@/path/to/your/json/files/pipelines.json\"` if you are dispatching the request from outside the folder.\n\n## Production deployment\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatthiaszepper%2Fnfcore_stats_backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatthiaszepper%2Fnfcore_stats_backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatthiaszepper%2Fnfcore_stats_backend/lists"}