{"id":15009195,"url":"https://github.com/ilri/rmg-dataportal-migration","last_synced_at":"2026-02-08T01:33:52.917Z","repository":{"id":50167923,"uuid":"269091393","full_name":"ilri/rmg-dataportal-migration","owner":"ilri","description":"A ckanapi-exporter workflow to help migrate ILRI's CKAN to another data repository","archived":false,"fork":false,"pushed_at":"2022-12-08T10:06:27.000Z","size":111,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-07-22T04:47:55.614Z","etag":null,"topics":["ckan","data-portals","python27"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ilri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-03T13:10:20.000Z","updated_at":"2021-08-02T12:36:45.000Z","dependencies_parsed_at":"2023-01-25T12:16:03.748Z","dependency_job_id":null,"html_url":"https://github.com/ilri/rmg-dataportal-migration","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ilri/rmg-dataportal-migration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilri%2Frmg-dataportal-migration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilri%2Frmg-dataportal-migration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilri%2Frmg-dataportal-migration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilri%2Frmg-dataportal-migration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ilri","download_url":"https://codeload.github.com/ilri/rmg-dataportal-migration/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilri%2Frmg-dataportal-migration/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29216086,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-08T00:10:47.190Z","status":"ssl_error","status_checked_at":"2026-02-08T00:10:43.589Z","response_time":63,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ckan","data-portals","python27"],"created_at":"2024-09-24T19:23:32.830Z","updated_at":"2026-02-08T01:33:52.901Z","avatar_url":"https://github.com/ilri.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# ILRI Data Portal Migration\nA workflow using [ckanapi-exporter](https://github.com/ckan/ckanapi-exporter) to export metadata from ILRI's CKAN Data Portal to CSV for migration to other data repositories. Right now it is unclear whether we'll be aligning to Dataverse's own metadata or some other schema like DDI.\n\n## Setup\nIf you have [poetry](https://python-poetry.org/) installed you can set up and activate the environment like this:\n\n```console\n$ poetry install\n$ poetry shell\n```\n\nOtherwise you will have to create the Python virtual environment and install the requirements manually:\n\n```console\n$ python2 -m virtualenv venv\n$ source venv/bin/activate\n$ pip install -r requirements.txt\n```\n\n## Running\nTo export CKAN metadata into CSV format aligning with Dataverse metadata fields:\n\n```console\n$ ckanapi-exporter --url https://data.ilri.org/portal --columns columns-ckan2dataverse.json \u003e output.csv\n```\n\n*N.B.*: The pattern specification in columns.json is very close to the structure of the JSON output from the CKAN API. See below for reference.\n\n## TODO\nSome questions we need to answer:\n\n- How to handle some overlapping and ambiguous metadata fields in CKAN:\n  - `title` vs `ILRI_prjtitle`\n  - `ILRI_prjstaff` vs `ILRI_actystaff`\n  - `ILRI_prjpartners` vs `ILRI_actypartners`\n  - `ILRI_prjregions` vs `ILRI_actyregions` (strange that `ILRI_actyregions` don't appear in PostgreSQL keys, only in REST API responses!)\n  - `ILRI_prjcountries` vs `ILRI_actycountries`\n  - `ILRI_prjpi` vs `ILRI_actypi`\n- How to handle multi-value fields?\n  - Countries seem to use `+`\n  - Project partners seem to use `;`\n  - Authors seem to use `,`\n\n## Notes\nSome notes about extracting data from CKAN using the API:\n\nGet a list of packages:\n\n```\n$ curl https://data.ilri.org/portal/api/3/action/package_list\n```\n\nDump datasets in one package using the `ckanapi` client:\n\n```\n$ ckanapi dump datasets restoration-of-degraded-land -r https://data.ilri.org/portal --datapackages=./dump_directory/\n```\n\nShow JSON representation of one package (dataset) using `ckanapi` client:\n\n```\n$ ckanapi action package_show id=restoration-of-degraded-land -r https://data.ilri.org/portal\n```\n\nSee more of CKAN's metadata fields in the `package` and `package_extra` tables in PostgreSQL:\n\n- Vanilla CKAN fields: `ckan=\u003e SELECT * FROM package LIMIT 50;`\n- Custom ILRI fields:\n\n```\nckan=# SELECT DISTINCT key FROM package_extra ORDER BY key;\n...\n ILRI_actyboundbox\n ILRI_actyboundboxcenter\n ILRI_actycitation\n ILRI_actycitationacknowledge\n ILRI_actycontactemail\n ILRI_actycontactperson\n ILRI_actycustodian\n ILRI_actycustodianemail\n ILRI_actydataowner\n ILRI_actydatavailable\n ILRI_actydatecollected\n ILRI_actydatecollectedend\n ILRI_actyfarmconsent\n ILRI_actyipownership\n ILRI_actymapextent\n ILRI_actymapzoom\n ILRI_actynatlevel\n ILRI_actyotheruse\n ILRI_actypartners\n ILRI_actypi\n ILRI_actypiemail\n ILRI_actyprojection\n ILRI_actyrelconfdata\n ILRI_actyresolution\n ILRI_actysharingagreement\n ILRI_actystaff\n ILRI_actyusageconditions\n ILRI_crpandprogram\n ILRI_prjabstract\n ILRI_prjcountries\n ILRI_prjdonor\n ILRI_prjedate\n ILRI_prjgrant\n ILRI_prjpartners\n ILRI_prjpi\n ILRI_prjpiemail\n ILRI_prjregions\n ILRI_prjsdate\n ILRI_prjspecies\n ILRI_prjstaff\n ILRI_prjtitle\n ILRI_prjwebsite\n```\n\n## License\nThis work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).\n\nThe license allows you to use and modify the work for personal and commercial purposes, but if you distribute the work you must provide users with a means to access the source code for the version you are distributing. Read more about the [GPLv3 at TL;DR Legal](https://tldrlegal.com/license/gnu-general-public-license-v3-(gpl-3)).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Filri%2Frmg-dataportal-migration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Filri%2Frmg-dataportal-migration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Filri%2Frmg-dataportal-migration/lists"}