{"id":29201093,"url":"https://github.com/datopian/ckan-ng-harvest","last_synced_at":"2026-06-28T19:31:17.591Z","repository":{"id":52699325,"uuid":"227401133","full_name":"datopian/ckan-ng-harvest","owner":"datopian","description":null,"archived":false,"fork":false,"pushed_at":"2021-04-20T19:03:03.000Z","size":4855,"stargazers_count":3,"open_issues_count":11,"forks_count":1,"subscribers_count":11,"default_branch":"develop","last_synced_at":"2025-11-08T03:28:35.858Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datopian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-11T15:42:25.000Z","updated_at":"2021-02-02T23:44:28.000Z","dependencies_parsed_at":"2022-08-22T07:01:03.173Z","dependency_job_id":null,"html_url":"https://github.com/datopian/ckan-ng-harvest","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/datopian/ckan-ng-harvest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datopian%2Fckan-ng-harvest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datopian%2Fckan-ng-harvest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datopian%2Fckan-ng-harvest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datopian%2Fckan-ng-harvest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datopian","download_url":"https://codeload.github.com/datopian/ckan-ng-harvest/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datopian%2Fckan-ng-harvest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34901959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-02T11:07:32.659Z","updated_at":"2026-06-28T19:31:17.576Z","avatar_url":"https://github.com/datopian.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/datopian/ckan-ng-harvest.svg?branch=develop)](https://travis-ci.org/datopian/ckan-ng-harvest)\n\n# Harvesting \n\nThis project is about harvesting from external sources to a CKAN (local or remote) instance.  \nThis project uses a custom Python library ([ckan-harvesters](https://pypi.org/project/ckan-harvesters/))in order to read and validate different types of data sources.  \n\nInstall locally\n\n```\npython3 -m venv /path-to-harvester-env\nsource /path-to-harvester-env/bin/activate\npip install -r requirements.txt\n```\n\n## Run with CKAN \n\nFirst we need to set up the local CKAN instance\n\n```\ndocker-compose up -d ckan\ndocker-compose exec ckan ckan-paster --plugin=ckan sysadmin add -c /etc/ckan/production.ini admin password=12345678 email=admin@localhost\ndocker-compose down\n```\n\nReady to start\n```\ndocker-compose up\n```\n\nThis starts a group of docker containers including CKAN (and related services) and the Harvester (with Airflow).\n\nIn the first run the local CKAN instance will be empty, there is no harvest sources to harvest from.  \nYou can add harvest source by impoting them from another CKAN instance (e.g. catalog.data.gov).\n\nNow you can see running:\n - Airflow: http://localhost:8081/ (without [DAGs](https://airflow.apache.org/docs/stable/concepts.html#dags) by now)\n - CKAN: http://localhost:5000\n\n## Create a Harvest Source\n\nYou can create a Harvest source manually from http://localhost:5000/harvest.  \nAirflow automatically get this source and create a DAG to process it.  \n\n![Auto DAG loaded](docs/imgs/auto-dag-one.png)\n\n## Import harvest sources\n\nAirflow is going to read all harvest sources and add them as DAGs.  \nWe can import all the harvest sources from a productive CKAN instance with the command `import_harvest_sources`.  \n\nFor example import all DataJSON harvest sources from _data.gov_.  \n```\ncd tools\n\n# for example import all CSW harvest sources from data.gov\npython import_harvest_sources.py \\\n    --import_from_url https://catalog.data.gov \\\n    --source_type datajson \\\n    --method GET \\\n    --destination_url http://localhost:5000 \\\n    --destination_api_key xxxxxxxxxxxxxxxx \\\n    --method GET\n\nGetting external harvest sources for https://catalog.data.gov\nSearching https://catalog.data.gov/api/3/action/package_search PAGE:1 start:0, rows:1000 with params: {'start': 0, 'rows': 1000, 'fq': '+dataset_type:harvest', 'q': '(type:harvest source_type:datajson)'}\nSearch harvest packages via GET\n149 results\n**** Importing Organization {'description': '', 'created': '2013-05-18T11:48:54.524965', 'title': 'Department of Agriculture', 'name': 'usda-gov', 'is_organization': True, 'state': 'active', ...\n\nPOST http://ckan:5000/api/3/action/organization_create headers:{'User-Agent': 'ckan-portal-filter 0.01-alpha', 'X-CKAN-API-Key': 'fd6fbdfb-de4a-43bf-97d5-8f3d8e3fa18d', 'Content-Type': 'application/json'} data:{\"description\": \"\", \"title\": \"Department of Agriculture\", \"name\": \"usda-gov\", \"is_organization\": true, ...\n\nPOST http://ckan:5000/api/3/action/package_create data:{'frequency': 'MANUAL', 'title': 'illinois json', ...\n\nCreated illinois-json\n\n...\n\n----------------------------------------------\nFinished: 149 sources\n----------------------------------------------\n\n```\nThis also import and create _organizations_.  \n\n## data.json files\n\n[More info](harvester_ng/datajson/README.md)\n\n## CSW\n\n[More info](harvester_ng/csw/README.md)\n\n\n### Some tools\n\n#### Read harvest sources\nYou can search via CKAN API the list of packages/harvest sources. \n\nGet CSW harvest sources at _data.gov_\n```\ncd tools\n\npython read_harvest_sources.py --base_url=http://nginx:8080\n\npython read_harvest_sources.py --base_url https://catalog.data.gov --source_type csw --method GET  \n# CKAN 2.3 fail with POST, current versions works fine with POST\n\nSearching https://catalog.data.gov/api/3/action/package_search PAGE:1 start:0, rows:1000 with params: {'start': 0, 'rows': 1000, 'fq': '+dataset_type:harvest', 'q': '(type:harvest source_type:csw)'}\n6 results\nHarvest source: Alaska LCC CSW Server 54778007-bac5-4f2e-8015-622176f23e02 \n\talaska-lcc-csw-server csw http://metadata.arcticlcc.org/csw\n\t{'description': 'The Arctic Landscape Conservation Cooperative (ALCC) supports conservation in the arctic by providing [applied science](http://arcticlcc.org/projects/) and [tools](http://arcticlcc.org/products/) to land managers and policy makers. It is a self-directed partnership that functions through a structure of at-large partners, a core staff, and a steering committee of cooperating agencies and organizations. Its [geographic scope](http://arcticlcc.org/projects/geospatial-data/arctic-lcc-boundaries/) ranges across North America from Alaska to Labrador. The Alaska portion encompasses three eco-regions: the Brooks Range, the Arctic Foothills, and the Arctic Coastal Plain.', 'created': '2013-08-30T12:57:14.187258', 'title': 'Arctic Landscape Conservation Cooperative', 'name': 'alcc-fws-gov', 'is_organization': True, 'state': 'active', 'image_url': 'http://arcticlcc.org/assets/images/logos/logo_small.png', 'revision_id': 'ecafc1e0-403b-488e-bca1-62b78e4abc81', 'type': 'organization', 'id': 'fe75ed93-9c93-4c56-bc02-11ab228d1abd', 'approval_status': 'approved'}\nHarvest source: NC OneMap CSW f6462f24-5acc-4eaf-937e-f8341585087c \n\tnc-onemap-csw csw http://data.nconemap.com/geoportal/csw?Request=GetCapabilities\u0026Service=CSW\u0026Version=2.0.2\n\t{'description': '', 'created': '2013-03-14T03:44:38.764842', 'title': 'State of North Carolina', 'name': 'nc-gov', 'is_organization': True, 'state': 'active', 'image_url': 'http://www.nconemap.com/portals/7/images/gpt_logo_small.png', 'revision_id': '4fb5830b-68c1-4e8b-a04e-3750c3e885c8', 'type': 'organization', 'id': '1abdfb5f-4371-4fa3-8ef3-497cebe1b3fe', 'approval_status': 'approved'}\n\n  ...\n```\n\nGet your local data.json harvest sources\n```\npython3 read_harvest_sources.py --base_url http://ckan:5000 --source_type datajson --method POST\n```\n\n#### Analyze \n\nCreates a CSV file with all the harvest sources. Analyze each one to validate it, count datastets and resources.\n\n```\npython tools/analyze_harvest_sources.py \n\nSearching https://catalog.data.gov/api/3/action/package_search PAGE:1 start:0, rows:1000 with params: {'start': 0, 'rows': 1000, 'fq': '+dataset_type:harvest', 'q': '(type:harvest)'}\n977 results\ndata/harvest_sources/datasets/harvest-source-census-5-digit-zip-code-tabulation-area-zcta5-national.json saved\n +++ [[waf-collection]] Reading source Census 5-Digit ZIP Code Tabulation Area (ZCTA5) National (1) from http://meta.geo.census.gov/data/existing/decennial/GEO/GPMB/TIGERline/TIGER2013/zcta510/\ndata/harvest_sources/datasets/harvest-source-current-american-indian-alaska-native-native-hawaiian-areas-national-aiannh.json saved\n +++ [[waf-collection]] Reading source Current American Indian/Alaska Native/Native  Hawaiian Areas National (AIANNH) (2) from http://meta.geo.census.gov/data/existing/decennial/GEO/GPMB/TIGERline/TIGER2013/aiannh/\n+++ [[datajson]] Reading source WPRDC data.json (255) from https://data.wprdc.org/data.json\n +++++++++++ ERROR\nValidation error (truncated): Error validating JsonSchema: 'bureauCode' is a required property\n\n ... \n\n```\n\nCSV results\n\n![source_analysis](docs/imgs/source_analysis.png)\n\n\n## Harvest ALL sources\n\nWe create a script](https://gitlab.com/datopian/ckan-ng-harvest/blob/develop/automate-tasks/airflow/dags/harvest_with_airflow.py) to put ALL the harvest sources in a queue using [Airflow](https://gitlab.com/datopian/ckan-ng-harvest/blob/develop/automate-tasks/airflow/README.md).  \n\n\n## Generate harvest commands list\n\nIf you want to check or analize the harvest commands you can run the script `generate_commands_list.py`.  \n\n```\ncd tools\npython generate_commands_list.py \\\n  --ckan_url http://nginx:8080 \\\n  --ckan_api_key xxxxxxxxxxxxx \\\n  --source_type datajson\n```\nThis script will list all the valid commands to harvest each source.\n\n### Tests\n\nCheck the [tests.md](tests/tests.md) file.  \n\n### Logs\n\nNotes about logging at [logging.md](docs/logging.md) file.  \n\n### Docker usage\n\nUse harvester with docker. More info [here](docker.md).  \n\n### Security scanning\n\nMerge requests are scanned using `safety` to detect known vulnerabilities in\ndependencies and with `bandit` to find security issues in the code via static\nanalysis.\n\n### Dependency management\n\nRequired dependencies are installed using pip.  Pinned production requirements are found \nin `requirements.txt`, development requirements in `dev-requirements.txt`.\n\nThese files are generated using `pip-compile` (from `pip-tools`) from `requirements.in`\nand `dev-requirements.in`, respectively.\n\nTo update the pinned requirement files, install `pip-tools` and run:\n\n```\n$ pip-compile requirements.in\n$ pip-compile dev-requirements.in\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatopian%2Fckan-ng-harvest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatopian%2Fckan-ng-harvest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatopian%2Fckan-ng-harvest/lists"}