{"id":19364203,"url":"https://github.com/mindrones/nestauk-dap_dv_backends","last_synced_at":"2026-04-20T19:32:39.862Z","repository":{"id":224046863,"uuid":"729196343","full_name":"mindrones/nestauk-dap_dv_backends","owner":"mindrones","description":"Copy of https://github.com/nestauk/dap_dv_backends","archived":false,"fork":false,"pushed_at":"2024-03-06T10:34:18.000Z","size":10579,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"dev","last_synced_at":"2025-11-16T19:20:45.159Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mindrones.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-08T16:15:31.000Z","updated_at":"2024-02-23T11:27:53.000Z","dependencies_parsed_at":"2024-11-10T07:36:47.742Z","dependency_job_id":null,"html_url":"https://github.com/mindrones/nestauk-dap_dv_backends","commit_stats":null,"previous_names":["mindrones/nestauk-dap_dv_backends"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mindrones/nestauk-dap_dv_backends","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindrones%2Fnestauk-dap_dv_backends","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindrones%2Fnestauk-dap_dv_backends/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindrones%2Fnestauk-dap_dv_backends/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindrones%2Fnestauk-dap_dv_backends/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mindrones","download_url":"https://codeload.github.com/mindrones/nestauk-dap_dv_backends/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindrones%2Fnestauk-dap_dv_backends/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32062410,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T07:36:34.445Z","updated_at":"2026-04-20T19:32:39.839Z","avatar_url":"https://github.com/mindrones.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dap_dv_backends Repository\n\n## Overview\n\nThis system is designed to annotate text from documents using DbPedia Spotlight from Wikipedia in such a way that annotations (topics) are Wikipedia pages.\n\nIt does so by creating an infrastructure of EC2 instances to annotate batches of documents stored in either S3 or ElasticSearch, orchestrating the flow of information automatically. You'll find a more thorough description of the architecture in [docs/architecture.md.md](docs/architecture.md.md).\n\n## Requirements\n\nThis system relies on having an ElasticSearch index. There are two scenarios:\n- The data is in this index: In this case the index acts as a source and recipient of the annotations.\n- The data is in S3: In this case the ElasticSearch index acts as a recipient of the annotations.\n\nUsing a tool (e.g. `Insomnia`) select your domain URI pointing to the ES index domain (e.g. in our case https://es.annotations.dap-tools.uk/).\n\nThen create the index:\n\n```\nPUT /my-index\n```\n\nUse that index name for data ingestion and annotation tasks.\n\n## Installations\n\nPlease refer to [docs/installation.md](docs/installation.md).\n\n## Usage\n\nTo annotate data, clients send requests to the reverse proxy, which then routes these to the appropriate service based on the request path. This setup simplifies access control, monitoring, and management of the system's components.\n\nThe process is in 3 steps:\n\n- Make sure you have AWS credentials in `~/.aws`\n- Navigate to the `dap_dv_backends` repo, then:\n\t- edit `src/services/config.mjs`\n\t- push the changes to the repo\n\t- `npm install` if needed\n\t- `npm run deployInfra`\n- To request a token, in your browser:\n\t- navigate to `https://\u003cAPI_SUBDOMAIN\u003e.\u003cBASE_DOMAIN\u003e/static/index.html` (for Nesta this would be `https://api.dap-tools.uk/auth/static/index.html`)\n\t- open the `/request` section\n\t- click on `Try it out`\n\t- insert your email, click on `Execute`: this should send you an email\n- Check your email for an email with the token then:\n\t- copy the token\n\t- click on the activation link\n- In your terminal:\n\t- `export set NESTA_EMAIL=\u003cemail\u003e`\n\t- `export set NESTA_TOKEN=\u003ctoken\u003e`\n- Navigate to the `nestauk/dsp_waifinder` repo, then:\n\t- `cd etl`\n\t- `npm install`\n\t- `npm install dap_dv_backends_utils` (this installs from a git url so make sure you have the latest version, `0.0.16`)\n\t- edit the npm script to point to the domains you have chosen, so for example\n\tin [1] check make sure the `-e` is correct\n\t- `npm run annotateData`\n- When the annotation has finished:\n\t- you should receive an email to notify you the annotation is done\n\t- navigate to your local `dap_dv_backends` repo\n\t- `npm run deleteInfra`\n\n\n[1] `\"annotateData\": \"npx annotateEsIndex -d es.annotations.dap-tools.uk -e https://api.dap-tools.uk/annotation -i ai_map -f description\"`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmindrones%2Fnestauk-dap_dv_backends","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmindrones%2Fnestauk-dap_dv_backends","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmindrones%2Fnestauk-dap_dv_backends/lists"}