{"id":15090837,"url":"https://github.com/mjanez/ckan-ansible","last_synced_at":"2025-10-09T15:30:39.443Z","repository":{"id":227446960,"uuid":"770427128","full_name":"mjanez/ckan-ansible","owner":"mjanez","description":"Automates CKAN deployments across different environments using Ansible","archived":false,"fork":false,"pushed_at":"2025-01-10T08:02:49.000Z","size":3285,"stargazers_count":0,"open_issues_count":1,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-10T08:33:19.642Z","etag":null,"topics":["ansible","automation","ckan","debian","deployment","metadata","open-data","posgtresql","redis","rhel","solr","vagrant"],"latest_commit_sha":null,"homepage":"","language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mjanez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-11T14:33:08.000Z","updated_at":"2025-01-10T08:02:54.000Z","dependencies_parsed_at":"2024-05-21T21:23:24.267Z","dependency_job_id":"72bf6706-16cf-4f22-9f4a-176ca9a6cfdd","html_url":"https://github.com/mjanez/ckan-ansible","commit_stats":{"total_commits":113,"total_committers":1,"mean_commits":113.0,"dds":0.0,"last_synced_commit":"f44717effccb4c434b3d8af480d3f466e2444a63"},"previous_names":["mjanez/ckan-ansible"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjanez%2Fckan-ansible","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjanez%2Fckan-ansible/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjanez%2Fckan-ansible/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjanez%2Fckan-ansible/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mjanez","download_url":"https://codeload.github.com/mjanez/ckan-ansible/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235832710,"owners_count":19052114,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","automation","ckan","debian","deployment","metadata","open-data","posgtresql","redis","rhel","solr","vagrant"],"created_at":"2024-09-25T10:34:18.237Z","updated_at":"2025-10-09T15:30:29.431Z","avatar_url":"https://github.com/mjanez.png","language":"Jinja","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CKAN Ansible Deployments\nAnsible playbook for the deployment of a custom CKAN for spatial data management in different environments.\n\nDeployments available for the following OS:\n| OS | Compatibility                                                                | Versions | Remarks |\n|--------------|-----------------------------------------------------------------------------|--|--|\n| Arch Linux           | 🚧 WIP |        -                                                   | - |\n| CentOS           | 🚧 WIP | [`CentOS 8`](https://www.centos.org/download/), [`CentOS Stream 9`](https://centos.org/stream9/)| - |\n| Debian          | 🚧 WIP |  [`12 - Bookworm`](https://www.debian.org/releases/bookworm/)                                                       | - |\n| RedHat Enterprise Linux         | ✅ Yes | [`RHEL 8`](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8), [`RHEL 9`](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9) | - |\n| Ubuntu          | 🚧 WIP |  [`20.04 - Focal Fossa`](https://releases.ubuntu.com/20.04/)| - |\n\n\nAnd CKAN versions:\n| CKAN version | Compatibility                                                                 | Versions | Remarks |\n|-------------|-----------------------------------------------------------------------------|--|--|\n| 2.8          | ❌ No         |             -                                      | Incompatible with [`Python 3`](https://docs.ckan.org/en/2.9/maintaining/upgrading/upgrade-to-python3.html#upgrading-a-ckan-install-from-python-2-to-python-3) |\n| 2.9          | ✅ Yes |  [`ckan-2.9.11`](https://docs.ckan.org/en/2.9/changelog.html#v-2-9-11-2024-03-13) | - |\n| 2.10         | ✅ Yes |  [`ckan-2.10.5`](https://docs.ckan.org/en/2.10/changelog.html#v-2-10-5-2024-08-21) | - |\n\n\n## Requirements\n- [`ansible`](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) which adds a community-curated selection of Ansible Collections.\n\n\u003e[!WARNING]\n\u003e **Do not** install the `ansible-core` minimalist version. Install `ansible` using `pip` or `pipx`:\n\u003e    ```sh\n\u003e    python3 -m ensurepip --upgrade\n\u003e    python3 -m pip install --user ansible\n\u003e    ```\n\u003e \n\u003e The full `ansible` package includes a community-curated selection of Ansible Collections, which helps avoid issues with unavailable `ansible-galaxy` modules in the `ansible-core` version.\n\nCheck the `ansible` version and collection list:\n```sh\nansible --version\n\nansible-galaxy collection list \n```\n\n### Updating Ansible collection paths\nAfter installing Ansible collections using ansible-galaxy, ensure their directories are included in the collections_paths within [`ansible.cfg`](./playbook/ansible.cfg). Follow these steps:\n\n\n1. **List Installed Collections:**\nRun the following command to display all installed Ansible collections and their respective paths:\n\n```sh\nansible-galaxy collection list\n```\n\n2. **Identify Collection Directories:**\nFrom the output, note the directories where the collections are installed. Each collection entry will display its path.\n\n3. **Update `collections_paths` in `ansible.cfg`:**\nIf the collection directories are not already included in the `collections_paths`, add them. Edit the `ansible.cfg` file to include these paths, separated by colons (`:`). \n\nReplace `/path/to/your/collection` with the actual paths obtained from the `ansible-galaxy collection list` output.\n\nFor example:\n\n```yml\n[defaults]\n\n########################################\n# Common destinations\n########################################\n\ninventory = ./inventories/development/hosts.ini\nhostfile = ./inventories/development/hosts.ini\nroles_path = ./roles/common:./roles/ckan:./roles/database:./roles/webserver:./roles/solr:./roles/redis:./roles/supervisor\nretry_files_save_path = ./config/tmp/retry/\nlog_path = ./config/ansible.log\nstdout_callback = yaml\n\n### Custom ansible-galaxy collection paths\ncollections_paths = ~/.ansible/collections:/usr/share/ansible/collections:/lib/python3.9/site-packages/ansible_collections:/path/to/your/collection\n```\n\n## CKAN Ansible Deployment\n1. Clone this repository to your local machine and edit the `ansible.cfg` to use the env what you want, by default `development`:\n\n    ```bash\n    git clone https://github.com/mjanez/ckan-ansible.git \u0026\u0026 cd ckan-ansible\n    vi playbook/ansible.cfg\n    ```\n\n2. Ensure that `inventory` and `hostfile` point to the desired hosts files, e.g., for `development` (default):\n\n    ```ini\n    [defaults]\n\n    ########################################\n    # Common destinations\n    ########################################\n\n    inventory = ./inventories/development/hosts.ini\n    hostfile = ./inventories/development/hosts.ini\n    roles_path = ./roles/common:./roles/ckan:./roles/database:./roles/webserver:./roles/solr:./roles/redis:./roles/supervisor\n    retry_files_save_path = ./config/tmp/retry/\n    log_path = ./config/ansible.log\n    stdout_callback = yaml\n    ```\n\n3. Edit the `hosts.ini` and add the target deployment servers IP addresses or `hostname` for the specific environment.\n\n    ```bash\n    vi playbook/inventories/development/hosts.ini\n    ```\n\n    ```ini\n    [ckan_servers]\n    development_01 ansible_host=192.168.68.01 ansible_user=sudouser ansible_port=222 ansible_ssh_pass=sudouserpassword ansible_connection=ssh\n    ```\n\n    Or for local use:\n\n    ```ini\n    [ckan_servers]\n    development_01 ansible_connection=local\n    ```\n\n4. Modify the host variables in `playbook/inventories/*/host_vars/*_01.yml`, for instance, [`development_01.yml`](./playbook/inventories/development/host_vars/development_01.yml). Check any necessary variables such as database credentials, CKAN versions, and other specific settings.\n\n**By default, a local PostgreSQL database is set up.** To use an external PostgreSQL database, set `use_external_postgres_database` to `true`.\n\n```yaml\n### Webserver #########################################\nproxy_server_name: localhost\nproxy_server_url: http://{{ proxy_server_name }}\nproxy_local_services_url: http://localhost\n...\n\n### Database service #########################################\npostgres_port: 5432\npostgres_dir: \"/var/lib/pgsql/data\"\n# Using an external PostgreSQL server or setup a local one\nuse_external_postgres_database: false\nckan_database: {\n  postgres_user: \"postgres\",\n  postgres_password: \"postgres\",\n  ckan_db_user: \"ckandbuser\",\n  ckan_db_password: \"ckandbpassword\",\n  ckan_db: \"ckan_db\",\n  postgres_host: \"localhost\",\n}\n\n...\n```\n\n\u003e[!CAUTION]\n\u003e The `playbook/inventories/*/host_vars/*.yml` file contain customizable configuration variables for deployment. Remember to change before running the Ansible playbook. Specifically the host users/pwds info, and CKAN configuration:  `ckan_sysadmin_name`, `ckan_sysadmin_password` and `ckan_sysadmin_email`. Also the `proxy_server_name` and `nginx_port` for correct deployment.\n\n\u003e[!IMPORTANT]\n\u003e Also if using a SSH password authentication for private repos [create a SSH key pair](.ssh/keys/README.md) and copy the keys to the `./playbook/roles/common/files/keys`. The filenames of the keypair files must begin with id_ (e.g. `id_rsa` + `id_rsa.pub`)\n\n5. Last, run the Ansible playbook to deploy CKAN on the target server. The following command will deploy CKAN on the target server using the playbook configuration. The `-vvv` flag is used for verbose output:\n\n    ```bash\n    # Location of the ansible.cfg file based on the clone directory\n    export ANSIBLE_CONFIG=$(pwd)/playbook/ansible.cfg\n\n    # Location if ckan-ansible is cloned in the home directory\n    export ANSIBLE_CONFIG=$HOME/ckan-ansible/playbook/ansible.cfg\n\n    # Run the ansible playbook, Verbose with  -vvv\n    ansible-playbook $HOME/ckan-ansible/playbook/playbook.yml\n    ```\n\n    The `ANSIBLE_CONFIG` environment variable is used to specify the location of the `ansible.cfg` file. This is useful when you have multiple Ansible configurations and you want to specify which one to use, eg. `rhel-9`, `ubuntu-20.04`, etc.\n\n6. You can manage the deployment of CKAN using two primary commands: `deploy` and `update`. More info at [deployment options](#deployment-options)\n\n\u003e[!TIP]\n\u003e**Use `deploy`** when:\n\u003e  * You are installing CKAN for the first time.\n\u003e  * You need to perform a complete reinstallation, including resetting databases.\n\u003e\n\u003e**Use `update`** when:\n\u003e  * You want to update CKAN's codebase, extensions, and associated services without affecting the existing databases.\n\u003e  * You need to apply patches, upgrades, or new features to your current CKAN deployment.\n\n\n## Test\n### Vagrant\nOnce you have [Vagrant](https://www.vagrantup.com/docs/installation), [VirtualBox](https://www.virtualbox.org/wiki/Downloads)  installed, run the following commands under your [project directory](https://learn.hashicorp.com/tutorials/vagrant/getting-started-project-setup?in=vagrant/getting-started):\n\n1. Start the VM:\n\n  ```bash\n  # Change to the project directory\n  cd ckan-ansible\n\n  # Change to a specific OS directory\n  cd vagrant/rhel/rhel-9\n\n  # Edit the host_vars file: ansible_host, ansible_port, ansible_user, ansible_ssh_pass with the Vagrantfile values and .env file as needed\n  vi playbook/inventories/development/host_vars/development_01.yml\n  vi .env\n\n  # Start the virtual machine, vagrant copy the playbook to the VM\n  vagrant up\n\n  # Launch ansible playbook\n  vagrant ssh\n  ```\n\n2. In the virtual machine, run the following commands to deploy CKAN with Ansible:\n\n  ```bash\n  # ckan-ansible has been cloned into the home directory\n  export ANSIBLE_CONFIG=$HOME/ckan-ansible/playbook/ansible.cfg\n\n  # Verbose with  -vvv\n  ansible-playbook $HOME/ckan-ansible/playbook/playbook.yml\n  ```\n\n  \u003e[!TIP]\n  \u003e `ckan-ansible/*/*` are rsynced to the VM at `/home/vagrant/ckan-ansible/*/*`, you can edit the playbook in your local machine and run the ansible-playbook command in the VM using [VSCode](#vagrant-and-visual-studio-code).\n\n\n1. Once the playbook has finished, you can access CKAN at `http://192.168.56.20` from your local machine.\n\n### Docker [WIP]\nOnce you have [Docker](https://docs.docker.com/get-docker/) installed, run the following commands under your [project directory](https://docs.docker.com/compose/gettingstarted/):\n\n```bash\n# Edit the .env file with the docker-compose values\nvi .env\n\n# Up the docker compose services\ndocker-compose up -d\n\n# Test the ssh connection\nssh -p 2222 ckan@localhost\n```\n\n### Virtual Box\nCould you  https://developers.redhat.com/rhel8/install-rhel8-vbox#overview\n\n\nRHEL info:\n- [RHEL](https://developers.redhat.com/rhel8/install-rhel8-vbox#)\n- [DVD Binary files (`.iso`)](https://developers.redhat.com/products/rhel/download)\n\n\n## Information\n### Deployment Options\nYou can manage the deployment of CKAN using two primary commands: `deploy` and `update`. Each serves a distinct purpose depending on your current setup and desired outcome.\n\n### `deploy`\nThe `deploy` command performs a **fresh deployment** of CKAN. This process involves:\n\n- **Setting up the environment from scratch**: Configuring all necessary components and dependencies required for CKAN to run.\n- **Initializing databases**: This can **suppress or reset the CKAN and Datastore databases**, effectively clearing existing data. Use this option when:\n  - Setting up CKAN for the first time.\n  - Performing a complete teardown and reinstall of your CKAN environment.\n\n**Caution**: Running `deploy` will erase existing data in the CKAN and Datastore databases. Ensure you have backups if you need to preserve existing data.\n\n### `update`\nThe `update` command is designed to **upgrade an existing CKAN installation** without affecting the databases. This process includes:\n\n- **Updating the CKAN codebase**: Pulling the latest changes and applying updates to the CKAN application.\n- **Refreshing extensions**: Updating any CKAN extensions to their latest versions.\n- **Updating `ckan-pycsw` and workers/supervisors**: Ensuring that auxiliary services are up-to-date.\n- **Restarting web services**: Applying the changes by restarting the necessary web services, ensuring minimal downtime.\n\nBy default, `update` is the preferred method when you want to apply updates or changes to your CKAN instance **without disrupting your existing data**.\n\n### Vagrant commands\n```bash\n# To obtain info of the SSH connection\nvagrant ssh-config\n\n# To stop the virtual machine\nvagrant halt\n\n# To suspend the virtual machine\nvagrant suspend\n\n# Terminate the virtual machine\nvagrant destroy --force\n```\n\n### Vagrant and Visual Studio Code\nYou can use the `vagrant ssh-config` command to get the SSH configuration for your Vagrant machine, which can be simpler. Here's how:\n\n1. Run `vagrant ssh-config` in the terminal in your Vagrant project directory. This will print the SSH configuration for your Vagrant machine.\n\n2. Copy the output of `vagrant ssh-config` into your SSH configuration file (`~/.ssh/config`).\n\n3. In VS Code, open the command palette and run the `\"Remote-SSH: Connect to Host...\"` command. Choose your Vagrant machine from the list of hosts.\n\nVS Code will connect to your Vagrant machine and you will be able to edit files directly on the Vagrant machine using VS Code.\n\n### NGINX Configuration\nThe base configuration uses an NGINX image as the front-end (ie: reverse proxy). It includes HTTPS running on port number 8443. A \"self-signed\" SSL certificate is generated. The ENV `proxy_server_name`, NGINX `server_name` directive and the `CN` field in the SSL certificate have been both set to 'localhost'. This should obviously not be used for production.\n\nCreating the SSL cert and key files as follows:\n`openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj \"/C=DE/ST=Berlin/L=Berlin/O=None/CN=localhost\" -keyout ckan-local.key -out ckan-local.crt`\nThe `ckan-local.*` files will then need to be moved into the `ckan-ansible/playbook/roles/webserver/tasks/files` directory\n\n## Structure of the Ansible playbook\n  ```bash\n    playbook/\n    ├── ansible.cfg\n    ├── playbook.yml\n    ├── inventories/\n    │   ├── group_vars/\n    │   │   ├── all.yml\n    │   │   ├── production_01.yml\n    │   ├── os_vars/\n    │   │   ├── archlinux.yml\n    │   │   ├── centos-8.yml\n    │   │   ├── centos-9.yml\n    │   │   ├── debian-12.yml\n    │   │   ├── redhat-8.yml\n    │   │   ├── redhat-9.yml\n    │   ├── production/\n    │   │   ├── hosts.ini\n    │   │   └── host_vars/\n    │   │       ├── production_01.yml\n    │   ├── development/\n    │       ├── hosts.ini\n    │       └── host_vars/\n    │           └── development_01.yml\n    └── roles/\n        ├── ckan/\n        │   ├── tasks/\n        │   │   └── main.yml\n        │   └── ...\n        ├── ckan_pycsw/\n        │   ├── tasks/\n        │   │   └── main.yml\n        │   └── ...\n        ├── common/\n        │   ├── files/\n        │   ├── tasks/\n        │   │   └── main.yml\n        │   └── ...\n        ├── database/\n        │   ├── tasks/\n        │   │   └── main.yml\n        │   └── ...\n        ├── redis/\n        │   ├── tasks/\n        │   │   └── main.yml\n        │   └── ...\n        ├── solr/\n        │   ├── tasks/\n        │   │   └── main.yml\n        │   └── ...\n        ├── supervisor/\n        │   ├── tasks/\n        │   │   └── main.yml\n        │   └── ...\n        └── webserver/\n            ├── tasks/\n            │   └── main.yml\n            └── ...\n  ```\n\nThis directory structure organizes the `ckan-ansible` project. Here's an explanation:\n\n* `ansible.cfg`: Ansible configuration file. This file sets global configurations for Ansible, such as the inventory file location, roles path, logging, and other settings. It ensures that Ansible knows where to find the necessary files and how to execute the playbooks.\n* `playbook.yml`: Ansible playbook for deploying CKAN. This file contains a series of tasks and roles that define the steps required to set up and configure CKAN and its dependencies. It orchestrates the deployment process by specifying which roles to apply and in what order.\n* `inventories/`: Inventory files for different environments (e.g., development, staging, production, etc.). These files list the servers (hosts) that Ansible will manage. Each environment has its own directory containing the relevant inventory files.\n  * `group_vars/`: Group-specific variables. These files define variables that apply to groups of hosts, allowing for centralized configuration management.\n  * `os_vars/`: OS-specific variables. These files define variables specific to different operating systems, ensuring that tasks are executed correctly on different OS types.\n  * `production/`: Production environment inventory.\n    * `hosts.ini`: Hosts file for production.\n    * `host_vars/`: Host-specific variables.\n  * `development/`: Development environment inventory.\n    * `hosts.ini`: Hosts file for development.\n    * `host_vars/`: Host-specific variables.\n* `roles/`: Ansible roles for different components.\n  * `ckan/`: CKAN installation and configuration.\n  * `ckan_pycsw/`: [ckan-pycsw](https://github.com/mjanez/ckan-pycsw) CSW/INSPIRE endpoint.\n  * `common/`: Common tasks shared across different components. Also contains the `files/keys/` directory for copying SSH keys to the target server as needed.\n  * `database/`: PostgreSQL databases.\n  * `redis/`: Redis.\n  * `solr/`: Solr.\n  * `supervisor/`: Supervisor configuration.\n  * `webserver/`: Web server (NGINX).\n\n### How They Work Together\n1. **Configuration (`ansible.cfg`)**:\n   - Sets the default inventory file location (`inventories/development/hosts.ini` or `inventories/production/hosts.ini`).\n   - Defines the roles path, so Ansible knows where to find the roles.\n   - Configures logging and other settings to control Ansible's behavior.\n\n2. **Inventory (`hosts.ini`)**:\n   - Lists the servers that Ansible will manage.\n   - Specifies connection details for each server (e.g., IP address, SSH user, port).\n   - Can be environment-specific (e.g., `development`, `staging`, `production`).\n\n3. **Playbook (`playbook.yml`)**:\n   - Defines the sequence of tasks and roles to be executed on the servers listed in the inventory.\n   - Uses the roles defined in the `roles/` directory to perform specific tasks (e.g., installing CKAN, configuring the database).\n\n4. **Variables (`group_vars/`, `host_vars/`, `os_vars/`)**:\n   - Provide configuration details that can be applied to groups of hosts, individual hosts, or specific operating systems.\n   - Ensure that tasks are executed with the correct parameters and settings.\n\n## Contributing\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\nThis project is licensed under the GNU AFFERO GENERAL PUBLIC LICENSE - see the [LICENSE](LICENSE) file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmjanez%2Fckan-ansible","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmjanez%2Fckan-ansible","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmjanez%2Fckan-ansible/lists"}