{"id":20038891,"url":"https://github.com/defra/fcp-find-ai-data-ingester","last_synced_at":"2026-02-17T02:03:42.381Z","repository":{"id":238855241,"uuid":"797775783","full_name":"DEFRA/fcp-find-ai-data-ingester","owner":"DEFRA","description":"Scrapes gov.uk grant pages and stores them in Azure Blob Storage, ready for indexing in Azure AI Search","archived":false,"fork":false,"pushed_at":"2025-03-31T23:12:28.000Z","size":1018,"stargazers_count":2,"open_issues_count":5,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-09-15T03:41:27.554Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DEFRA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-05-08T13:47:50.000Z","updated_at":"2025-02-07T19:46:17.000Z","dependencies_parsed_at":"2024-06-21T03:48:15.884Z","dependency_job_id":"739f87ce-51bd-4d6a-b10b-099e5b31ea84","html_url":"https://github.com/DEFRA/fcp-find-ai-data-ingester","commit_stats":null,"previous_names":["defra/fcp-find-ai-data-ingester"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/DEFRA/fcp-find-ai-data-ingester","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Ffcp-find-ai-data-ingester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Ffcp-find-ai-data-ingester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Ffcp-find-ai-data-ingester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Ffcp-find-ai-data-ingester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DEFRA","download_url":"https://codeload.github.com/DEFRA/fcp-find-ai-data-ingester/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Ffcp-find-ai-data-ingester/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29530220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-17T00:57:22.232Z","status":"online","status_checked_at":"2026-02-17T02:00:08.105Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T10:33:50.126Z","updated_at":"2026-02-17T02:03:42.348Z","avatar_url":"https://github.com/DEFRA.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fcp-find-ai-data-ingester\n\nThe data ingester scrapes and stores govuk grant content. The grants are chunked into documents and stored in two indexes within Azure AI Search.\n\n- find-ai-vector-filterable-index-full - Contains the full documents of each grant\n- find-ai-vector-filterable-index-summaries - Stores a short summary of each grant\n\nA manifest file for each grant scheme is stored to Azure Blob Storage in order to prevent processing grants which have already been stored.\n\nThe data ingester should be ran on a timer trigger (every day), ensuring the contents of grants are up to date.\n\n## Prerequisites\n\n- Docker\n- Docker Compose\n\nOptional:\n\n- Kubernetes\n- Helm\n\n## Local Development\n\nInstall local dependencies\n```BASH\nnpm i\n```\n\nCopy and populate .env file (api keys will need to be added in manually)\n```BASH\ncp .env.example .env\n```\n\nSpin up docker container\n```BASH\ndocker-compose up \n### or to launch in the background:\ndocker-compose up -d\n```\n\nRun application on http://localhost:3000/\n\n## Running the application\n\nThe application is designed to run in containerised environments, using Docker Compose in development and Kubernetes in production.\n\n- A Helm chart is provided for production deployments to Kubernetes.\n\n### Build container image\n\nContainer images are built using Docker Compose, with the same images used to run the service with either Docker Compose or Kubernetes.\n\nWhen using the Docker Compose files in development the local `app` folder will\nbe mounted on top of the `app` folder within the Docker container, hiding the CSS files that were generated during the Docker build.  For the site to render correctly locally `npm run build` must be run on the host system.\n\n\nBy default, the start script will build (or rebuild) images so there will\nrarely be a need to build images manually. However, this can be achieved\nthrough the Docker Compose\n[build](https://docs.docker.com/compose/reference/build/) command:\n\n```\n# Build container images\ndocker-compose build\n```\n\n### Start\n\nUse Docker Compose to run service locally.\n\n* run migrations\n  * `docker-compose -f docker-compose.migrate.yaml run --rm database-up`\n* start\n  * `docker-compose up`\n* stop\n  * `docker-compose down` or CTRL-C\n\n\n## Test structure\n\nThe tests have been structured into subfolders of `./test` as per the\n[Microservice test approach and repository structure](https://eaflood.atlassian.net/wiki/spaces/FPS/pages/1845396477/Microservice+test+approach+and+repository+structure)\n\n### Running tests\n\nA convenience script is provided to run automated tests in a containerised\nenvironment. This will rebuild images before running tests via docker-compose,\nusing a combination of `docker-compose.yaml` and `docker-compose.test.yaml`.\nThe command given to `docker-compose run` may be customised by passing\narguments to the test script.\n\nExamples:\n\n```\n# Run all tests\nscripts/test\n\n# Run tests with file watch\nscripts/test -w\n```\n\n## CI \u0026 CD Pipeline\n\nThis service uses the [ADP Common Pipelines](https://github.com/DEFRA/adp-pipeline-common) for Builds and Deployments.\n\n### AppConfig - KeyVault References\nIf the application uses `keyvault references` in `appConfig.env.yaml`, please make sure the variable to be added to keyvault is created in ADO Library variable groups and the reference for the variable groups and variables are provided in `build.yaml` like below.\n\n```\nvariableGroups:\n    - fcp-find-ai-data-ingester-snd1\n    - fcp-find-ai-data-ingester-snd2\n    - fcp-find-ai-data-ingester-snd3\nvariables:\n    - fcp-find-ai-data-ingester-APPINSIGHTS-CONNECTIONSTRING\n```\n\n## Licence\n\nTHIS INFORMATION IS LICENSED UNDER THE CONDITIONS OF THE OPEN GOVERNMENT LICENCE found at:\n\n\u003chttp://www.nationalarchives.gov.uk/doc/open-government-licence/version/3\u003e\n\nThe following attribution statement MUST be cited in your products and applications when using this information.\n\n\u003e Contains public sector information licensed under the Open Government license v3\n\n### About the licence\n\nThe Open Government Licence (OGL) was developed by the Controller of Her Majesty's Stationery Office (HMSO) to enable information providers in the public sector to license the use and re-use of their information under a common open licence.\n\nIt is designed to encourage use and re-use of information freely and flexibly, with only a few conditions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Ffcp-find-ai-data-ingester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdefra%2Ffcp-find-ai-data-ingester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Ffcp-find-ai-data-ingester/lists"}