{"id":46391690,"url":"https://github.com/onsdigital/dp-search-reindex-batch","last_synced_at":"2026-03-05T09:02:58.397Z","repository":{"id":173932539,"uuid":"651132837","full_name":"ONSdigital/dp-search-reindex-batch","owner":"ONSdigital","description":"Search Reindexing Batch Job","archived":false,"fork":false,"pushed_at":"2025-11-28T15:29:33.000Z","size":152,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":7,"default_branch":"develop","last_synced_at":"2025-11-30T08:50:51.324Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ONSdigital.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-06-08T15:07:18.000Z","updated_at":"2025-11-28T15:29:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"2b5605d4-1bce-4fcf-985b-3e39d7852ae8","html_url":"https://github.com/ONSdigital/dp-search-reindex-batch","commit_stats":null,"previous_names":["onsdigital/dp-search-reindex-batch"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/ONSdigital/dp-search-reindex-batch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ONSdigital%2Fdp-search-reindex-batch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ONSdigital%2Fdp-search-reindex-batch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ONSdigital%2Fdp-search-reindex-batch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ONSdigital%2Fdp-search-reindex-batch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ONSdigital","download_url":"https://codeload.github.com/ONSdigital/dp-search-reindex-batch/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ONSdigital%2Fdp-search-reindex-batch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30117482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T08:19:04.902Z","status":"ssl_error","status_checked_at":"2026-03-05T08:17:37.148Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-03-05T09:02:57.452Z","updated_at":"2026-03-05T09:02:58.350Z","avatar_url":"https://github.com/ONSdigital.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dp-search-reindex-batch\n\nBatch nomad job for reindexing search\n\n## Getting started\n\n* Run `make debug` to run application (See [Local Prerequisites section](#local-prerequisites) first)\n* Run `make help` to see full list of make targets\n\n### Dependencies\n\n* No further dependencies other than those defined in `go.mod`\n\n### Tools\n\nTo run some of our tests you will need additional tooling:\n\n#### Audit\n\nWe use `dis-vulncheck` to do auditing, which you will [need to install](https://github.com/ONSdigital/dis-vulncheck).\n\n#### Linting\n\nWe use v2 of golangci-lint, which you will [need to install](https://golangci-lint.run/docs/welcome/install).\n\n### Configuration\n\n| Environment variable          | Default                                    | Description                                                                                                                  |\n|-------------------------------|--------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|\n| AWS_REGION                    | \"eu-west-2\"                                | AWS region                                                                                                                   |\n| AWS_SEC_SKIP_VERIFY           | false                                      | Whether to skip TLS verification for AWS requests                                                                            |\n| DATASET_API_URL               | \"\u003chttp://localhost:22000\u003e\"                 | URL of the Dataset API                                                                                                       |\n| DATASET_PAGINATION_LIMIT      | 500                                        | Number of datasets to fetch per page of requests to Dataset API                                                              |\n| ELASTIC_SEARCH_URL            | \"\u003chttp://localhost:11200\u003e\"                 | URL of elastic search server (or AWS Opensearch)                                                                             |\n| ENABLE_DATASET_API_REINDEX    | false                                      | Whether to get documents from the Dataset API for reindexing or not                                                          |\n| ENABLE_OTHER_SERVICES_REINDEX | false                                      | Whether to get documents from other upstream services or not                                                                 |\n| ENABLE_TOPIC_TAGGING          | false                                      | Whether to enable topic auto-tagging                                                                                         |\n| ENABLE_ZEBEDEE_REINDEX        | false                                      | Whether to get documents from Zebedee for reindexing or not                                                                  |\n| MAX_DATASET_EXTRACTIONS       | 20                                         | Max number of concurrent Dataset Extractions (ie. Dataset API connections)                                                   |\n| MAX_DATASET_TRANSFORMS        | 10                                         | Max number of concurrent Dataset Transformation workers                                                                      |\n| MAX_DOCUMENT_EXTRACTIONS      | 100                                        | Max number of concurrent Document Extractions (ie. Zebedee connections)                                                      |\n| MAX_DOCUMENT_TRANSFORMS       | 20                                         | Max number of concurrent Document Transformation workers                                                                     |\n| MAX_UPSTREAM_EXTRACTIONS      | 10                                         | Max number of concurrent Upstream Extractions (ie. Wagtail etc. connections)                                                 |\n| MAX_UPSTREAM_TRANSFORMS       | 10                                         | Max number of concurrent Upstream Transformation workers                                                                     |\n| OTHER_UPSTREAM_SERVICES       | [[\"http://localhost:29600\", \"/resources\"]] | List of upstream services. Each consisting a host and an endpoint. (See [other_upstream_services](#other_upstream_services)) |\n| SERVICE_AUTH_TOKEN            | \"\"                                         | Zebedee Service Auth Token for API requests                                                                                  |\n| SIGN_ELASTICSEARCH_REQUESTS   | false                                      | Whether to sign elasticsearch requests (true for AWS)                                                                        |\n| TOPIC_API_URL                 | \"\u003chttp://localhost:25300\u003e\"                 | URL of the Topic API                                                                                                         |\n| TRACKER_INTERVAL              | 5s                                         | Interval for progress tracker summary logging                                                                                |\n| UPSTREAM_PAGINATION_LIMIT     | 100                                        | Number of resources to request in each page of requests to other upstream services                                           |\n| ZEBEDEE_TIMEOUT               | 2m                                         | Timeout for Zebedee endpoints - published index can take \u003e 2 minutes                                                         |\n| ZEBEDEE_URL                   | \"\u003chttp://localhost:8082\u003e\"                  | URL of publishing zebedee                                                                                                    |\n\n#### OTHER_UPSTREAM_SERVICES\n\nThe variable for OTHER_UPSTREAM_SERVICES consist of a array of array with the following schema\n\n```yaml\nOTHER_UPSTREAM_SERVICES:\n    type: array\n    description: \"An array of other upstream services\"\n    items:\n        type: array\n        description: \u003e\n            An individual upstream service consisting of two strings,\n            a host and an endpoint\n        items:\n            type: string\n            maxItems: 2\n        example: [\"localhost:29600\", \"/resources1\"]\n```\n\nFor example:\n\n```json\n[\n    [\n        \"http://localhost29600\",\n        \"/resources\"\n    ]\n]\n```\n\nThis would then be provided as an environment variable like so:\n\n```sh\nexport OTHER_UPSTREAM_SERVICES='[[\"http://localhost:29600\",\"/resources\"]]'\n```\n\nFor the purposes of our configuration this would need to be double escaped like so:\n\n```json\n// ...json\n\"OTHER_UPSTREAM_SERVICES\":\"[[\\\\\\\"http://localhost:29600\\\\\\\", \\\\\\\"/resources/\\\\\\\"]]\"\n// ...more json\n```\n\n### Local Prerequisites\n\n* Requires ElasticSearch 7.10 running on port 11200\n* Requires Zebedee running on port 8082 (and this has a dependency on vault)\n* Requires the Dataset API running on port 22000\n* Requires the Topic API running on port 25300 if tagging by topic is desired\n\nNB. The Dataset API requires a mongo database named 'datasets', which must contain the following collections:\n\n* contacts\n* datasets\n* dimension.options\n* editions\n* instances\n* instances_locks\n\nThe Dataset API also requires this environment variable to be set to true: DISABLE_GRAPH_DB_DEPENDENCY\n\nPlease make sure your elasticsearch server is running locally on localhost:11200 and version of the server is 7.10,\nwhich is the current supported version. You may use `dp-compose/v2/stacks/search` stack for this.\n\nPlease ensure to set the 'ENABLE_TOPIC_TAGGING' flag to true, if the topic tagging feature is required.\n\nIf you want to run the reindex script locally but loading data from an environment (e.g. `sandbox`), you may\nrun `dp ssh` with port forwarding for dataset-api and zebedee (please check the services IPs and ports\nin `https://consul.dp.aws.onsdigital.uk/ui/eu/services`) For example:\n\n```shell\ndp ssh sandbox publishing 2 -p 22000:10.30.138.234:26020\ndp ssh sandbox publishing 1 -p 8082:10.30.138.93:25108\n```\n\nIf you do this the service auth token in the configuration will need to be a valid token accepted in the environment you\nare using.\n\n### Contributing\n\nSee [CONTRIBUTING](CONTRIBUTING.md) for details.\n\n### License\n\nCopyright © 2025, Office for National Statistics (\u003chttps://www.ons.gov.uk\u003e)\n\nReleased under MIT license, see [LICENSE](LICENSE.md) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonsdigital%2Fdp-search-reindex-batch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonsdigital%2Fdp-search-reindex-batch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonsdigital%2Fdp-search-reindex-batch/lists"}