{"id":22484266,"url":"https://github.com/CodeForAfrica/openAFRICA","last_synced_at":"2025-08-02T17:32:24.593Z","repository":{"id":83574267,"uuid":"101888981","full_name":"CodeForAfrica/openAFRICA","owner":"CodeForAfrica","description":"openAFRICA aims to be largest independent repository of open data on the African continent. This repo contains the primary deployment scripts and files. Accessible at https://openafrica.net/","archived":false,"fork":false,"pushed_at":"2024-06-11T14:28:39.000Z","size":177,"stargazers_count":27,"open_issues_count":4,"forks_count":9,"subscribers_count":21,"default_branch":"master","last_synced_at":"2024-07-09T17:03:33.649Z","etag":null,"topics":["africa","ckan","ckanext-openafrica","ckanext-socialite","data-portal","deployment","docker","docker-image","dokku","kenya","nigeria","open-africa","open-data","openafrica","rabbitmq","redis","solr","south-africa","tanzania"],"latest_commit_sha":null,"homepage":"https://openafrica.net/","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CodeForAfrica.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-30T14:06:00.000Z","updated_at":"2024-06-13T12:12:22.000Z","dependencies_parsed_at":"2023-06-13T19:15:32.434Z","dependency_job_id":"f0fb6d1b-d1f2-4384-84e8-fa3ea39eb054","html_url":"https://github.com/CodeForAfrica/openAFRICA","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CodeForAfrica%2FopenAFRICA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CodeForAfrica%2FopenAFRICA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CodeForAfrica%2FopenAFRICA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CodeForAfrica%2FopenAFRICA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CodeForAfrica","download_url":"https://codeload.github.com/CodeForAfrica/openAFRICA/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228499948,"owners_count":17929978,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["africa","ckan","ckanext-openafrica","ckanext-socialite","data-portal","deployment","docker","docker-image","dokku","kenya","nigeria","open-africa","open-data","openafrica","rabbitmq","redis","solr","south-africa","tanzania"],"created_at":"2024-12-06T17:10:31.559Z","updated_at":"2025-08-02T17:32:24.581Z","avatar_url":"https://github.com/CodeForAfrica.png","language":"Dockerfile","funding_links":[],"categories":["Dockerfile"],"sub_categories":[],"readme":"# openAFRICA\n*The continent's largest volunteer-driven open data portal.*\n\n![CKAN version](https://img.shields.io/badge/CKAN-v2.6.8-brightgreen.svg)\n\nThis repo seeks to streamline deployment of the openAFRICA platform by pulling together the different components used for [openAFRICA](https://openafrica.net/) and deploy using [dokku](http://dokku.viewdocs.io/dokku/).\n\n## CKAN\n\nCKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers datahub.io, catalog.data.gov and data.gov.uk among many other sites.\n\nWe use CKAN's own vanilla releases but because they haven't properly adopted Docker and dockerhub (yet) for deployment, we're keeping a stable version (`codeforafrica/ckan:latest`) that we can be sure plays nice with our extenstions.\n\nThe ckan extensions we are using include:\n\n- ckanext-openafrica - https://github.com/CodeForAfrica/ckanext-openafrica\n- ckanext-datarequests - https://github.com/conwetlab/ckanext-datarequests\n- ckanext-harvester - https://github.com/ckan/ckanext-harvest\n- ckanext-s3filestore - https://github.com/okfn/ckanext-s3filestore\n- ckanext-showcase - https://github.com/ckan/ckanext-showcase\n- ckanext-googleanalytics - https://github.com/ckan/ckanext-googleanalytics\n- ckanext-issues - https://github.com/ckan/ckanext-issues\n- ckanext-gdoc - https://github.com/OpenUpSA/ckanext-gdoc\n- ckanext-envvars - https://github.com/okfn/ckanext-envvars/\n\n\n---\n\n## Development\n\n\nTo set up your development environment:\n\n```sh\n$ git clone https://github.com/CodeForAfricaLabs/openAFRICA.git\n\n$ cd openAFRICA\n```\n\nRun this command (found on the docker-compose.yml):\n\n```sh\ndocker-compose build \u0026\u0026 docker-compose up\n```\n\n### Updating CKAN Docker Image\n\nTo update the `openafrica/ckan:latest` Docker image, edit `Makefile` and then run:\n\n```sh\nmake ckan\n```\n\n### Tests\n\n?\n\n---\n\n## Deployment\n\nWe use [dokku](http://dokku.viewdocs.io/dokku/) for deployment so you'd need to install and set it up first;\n\n```\n # for debian systems, installs dokku via apt-get\n $ wget https://raw.githubusercontent.com/dokku/dokku/v0.11.3/bootstrap.sh\n $ sudo DOKKU_TAG=v0.11.3 bash bootstrap.sh\n # go to your server's IP and follow the web installer\n```\n\n\n### Install + Create Dependencies\n\nOnce installed, we can do the following:\n\n1. Create the Dokku app and add a domain to it\n\n```\ndokku apps:create ckan\ndokku domains:add ckan openafrica.net\n```\n\n2. Add letsencrypt for free `https` certificate\n\nInstall the [dokku-letsencrypt](https://github.com/dokku/dokku-letsencrypt) plugin and set the config variables\n\n```\nsudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git\ndokku config:set --no-restart ckan DOKKU_LETSENCRYPT_EMAIL=support@codeforafrica.org\n```\n\n3. Create CKAN Solr Instance\n\nCKAN uses a special schema for Solr so you should deploy `openafrica/solr`\n\n```\ndokku apps:create ckan-solr\n\nsudo docker volume create --name ckan-solr\ndokku docker-options:add ckan-solr run,deploy --volume ckan-solr:/opt/solr/server/solr/ckan\n\nsudo docker pull codeforafrica/ckan-solr:2.7.6\nsudo docker tag codeforafrica/ckan-solr:2.7.6 dokku/ckan-solr:latest\n\ndokku git:from-image ckan-solr dokku/ckan-solr:latest\n\n```\n\n4. Create Redis Instance\n\nInstall the [redis](https://github.com/dokku/dokku-redis) plugin.\n\n```\nsudo dokku plugin:install https://github.com/dokku/dokku-redis.git redis\ndokku redis:create ckan-redis\n\n```\n\n5. Create CKAN DataPusher Instance\n\n[DataPusher](https://github.com/ckan/datapusher) is a standalone web service that automatically downloads any CSV or XLS (Excel) data files from a CKAN site's resources when they are added to the CKAN site, parses them to pull out the actual data, then uses the DataStore API to push the data into the CKAN site's DataStore.\n\n```\ndokku apps:create ckan-datapusher\n\nsudo docker pull openafrica/ckan-datapusher:latest\nsudo docker tag openafrica/ckan-datapusher:latest dokku/ckan-datapusher:latest\n\ndokku git:from-image ckan-datapusher dokku/ckan-datapusher:latest\n\n```\n\n6. Install Postgres (Optional)\n\nThis is an optional step if you'd like to have Postgres installed locally;\n\n```\nsudo dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres\ndokku postgres:create ckan-postgres\n\n```\n\n7. Install RabbitMQ\n\nInstall the [RabbitMQ](https://github.com/dokku/dokku-rabbitmq) plugin (The harvest extension uses this as its backend)\n\n```\nsudo dokku plugin:install https://github.com/dokku/dokku-rabbitmq.git rabbitmq\ndokku rabbitmq:create ckan-rabbitmq\n```\n\n8. Set up S3\n\nCreate a bucket and a programmatic access user, and grant the user full access to the bucket with the following policy\n\n```\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:*\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::openafrica/*\",\n                \"arn:aws:s3:::openafrica\"\n            ]\n        }\n    ]\n}\n```\n\n9. Create CKAN filestore volume\n\nCreate a named docker volume and configure ckan to use the volume just so we can configure an upload path. It should be kept clear by the s3 plugin.\n\n```\nsudo docker volume create --name ckan-filestore\ndokku docker-options:add ckan run,deploy --volume ckan-filestore:/var/lib/ckan/default\n```\n\n### Configuration\n\nNow we configure to pull the dependencies together:\n\nGet the Redis Dsn (connection details) for setting in CKAN environment in the next step with /0 appended.\n\n```\ndokku redis:info ckan-redis\n```\nGet the RabbitMQ Dsn (connection details) and extract the `username`, `password`, `hostname`, `virtualhost` and `port`. You need these details because the harvester extension in its current form does not support configuration using RabbitMQ URI scheme. The URI is in the form\n\n```\namqp://username:password@hostname:port/virtualhost\n```\n\n\nSet CKAN environment variables, replacing these examples with actual producation ones\n\n- REDIS_URL: use the Redis Dsn\n- SOLR_URL: use the alias given for the docker link below\n- BEAKER_SESSION_SECRET: this must be a secret long random string. Each time it changes it invalidates any active sessions.\n- S3FILESTORE__SIGNATURE_VERSION: use as-is - no idea why the plugin requires this.\n\n\n```\ndokku config:set ckan CKAN_SQLALCHEMY_URL=postgres://ckan_default:password@host/ckan_default \\\n                      CKAN_DATASTORE_READ_URL=postgresql://ckan_default:pass@localhost/datastore_default \\\n                      CKAN_DATASTORE_WRITE_URL=postgresql://datastore_default:pass@localhost/datastore_default \\\n                      CKAN_REDIS_URL=.../0 \\\n                      CKAN_INI=/ckan.ini \\\n                      CKAN_SOLR_URL=http://solr:8983/solr/ckan \\\n                      CKAN_SITE_URL=https://openafrica.net/ \\\n                      CKAN___BEAKER__SESSION__SECRET= \\\n                      CKAN_SMTP_SERVER= \\\n                      CKAN_SMTP_USER= \\\n                      CKAN_SMTP_PASSWORD= \\\n                      CKAN_SMTP_MAIL_FROM=hello@openafrica.net \\\n                      CKAN___CKANEXT__S3FILESTORE__AWS_BUCKET_NAME=openafrica \\\n                      CKAN___CKANEXT__S3FILESTORE__AWS_ACCESS_KEY_ID= \\\n                      CKAN___CKANEXT__S3FILESTORE__AWS_SECRET_ACCESS_KEY= \\\n                      CKAN___CKANEXT__S3FILESTORE__HOST_NAME=http://s3-eu-west-1.amazonaws.com \\\n                      CKAN___CKANEXT__S3FILESTORE__REGION_NAME=eu-west-1 \\\n                      CKAN___CKANEXT__S3FILESTORE__SIGNATURE_VERSION=s3v4 \\\n                      CKAN__HARVEST__MQ__VIRTUAL_HOST=ckan-rabbitmq \\\n                      CKAN__HARVEST__MQ__PORT=5672 \\\n                      CKAN__HARVEST__MQ__HOSTNAME=dokku-rabbitmq-ckan-rabbitmq \\\n                      CKAN__HARVEST__MQ__PASSWORD=912abee9882be7ca8718d3cab7263cfd \\\n                      CKAN__HARVEST__MQ__USER_ID=ckan-rabbitmq \\\n```\n\nLink CKAN with Redis, Solr, and CKAN DataPusher;\n```\ndokku redis:link ckan-redis ckan  #noqa\ndokku docker-options:add ckan run,deploy --link ckan-solr.web.1:solr\ndokku docker-options:add ckan run,deploy --link ckan-datapusher.web.1:ckan-datapusher\n```\n\n### Scheduled Jobs\nFor openAFRICA to work perfectly, some jobs have to run at certain times e.g. updating tracking statistics and rebuilding the search index for newly uploaded datasets. To create a scheduled job that is executed by a Dokku application, follow these steps:\n\n```sh\nsudo su dokku\ncrontab -e\n```\n\nAdd the following entries\n\n```sh\n0 * * * * echo '{}' | dokku --rm run ckan paster --plugin=ckan post -c /ckan.ini /api/action/send_email_notifications \u003e /dev/null\n\n0 * * * * dokku --rm run ckan paster --plugin=ckan tracking update -c /ckan.ini\n\n*/15 * * * * dokku --rm run ckan paster --plugin=ckanext-harvest harvester run --config=/ckan.ini\n```\n\n### Deploy CKAN\n\n\nOnce done with installing and configuring, you can push this repository to dokku:\n\n```\ngit remote add dokku dokku@openafrica.net:ckan\ngit push dokku\n```\n\n### Initialize Database\n\nBefore you can run CKAN for the first time, you need to run `db init` to initialize your database\n\n```\ndokku enter ckan\ncd src/ckan\npaster db init -c /ckan.ini\n```\n\nLastly, let's make sure we encrypt traffic:\n\n```\ndokku letsencrypt ckan\n```\n\n\n***NOTE:** Make sure to have the [appropriate permissions to push to dokku](http://dokku.viewdocs.io/dokku/deployment/user-management/).*\n\n---\n\n## Contributing\n\nThank you for considering to contribute to this project. You are awesome. :)\n\nTo get you started, here are few pointers:\n\n- We have a number of Github issues to work through here:\n  - openAFRICA deploy: https://github.com/CodeForAfricaLabs/openAFRICA/issues\n  - ckanext-openafrica: https://github.com/CodeForAfrica/ckanext-openafrica/issues\n  - ckanext-socialite: https://github.com/CodeForAfricaLabs/ckanext-socialite/issues\n  - ckanext-social: https://github.com/CodeForAfricaLabs/ckanext-social/issues\n- If you believe an issue is with CKAN core or related extenstions, post them here:\n  - CKAN core: https://github.com/ckan/ckan/issues\n  - ckanext-harvester: ?\n\nCheck out the [development docs](#development) to get started on this repo locally.\n\n\n### Security Vulnerabilities\n\nPlease report on security vulnerabilities to security@codeforafrica.org. These will be promptly acted on.\n\n---\n\n## License\n\nGNU General Public License\n\nopenAFRICA aims to be the largest independent repository of open data on the African continent.\nCopyright (C) 2017  Code for Africa\n\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see \u003chttps://www.gnu.org/licenses/\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCodeForAfrica%2FopenAFRICA","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCodeForAfrica%2FopenAFRICA","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCodeForAfrica%2FopenAFRICA/lists"}