{"id":19713363,"url":"https://github.com/ctsit/redcap_deployment","last_synced_at":"2025-10-11T13:34:33.151Z","repository":{"id":46186993,"uuid":"57070252","full_name":"ctsit/redcap_deployment","owner":"ctsit","description":"Scripted REDCap deployment tools at UF CTSI","archived":false,"fork":false,"pushed_at":"2024-12-17T19:46:14.000Z","size":1746,"stargazers_count":5,"open_issues_count":15,"forks_count":13,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-05T18:52:07.055Z","etag":null,"topics":["redcap"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ctsit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-25T19:33:53.000Z","updated_at":"2024-12-17T19:46:18.000Z","dependencies_parsed_at":"2023-02-12T05:30:56.552Z","dependency_job_id":"3effbdf6-628b-49b0-9f70-3e58433cb5b3","html_url":"https://github.com/ctsit/redcap_deployment","commit_stats":null,"previous_names":[],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctsit%2Fredcap_deployment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctsit%2Fredcap_deployment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctsit%2Fredcap_deployment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctsit%2Fredcap_deployment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ctsit","download_url":"https://codeload.github.com/ctsit/redcap_deployment/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251559844,"owners_count":21609085,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["redcap"],"created_at":"2024-11-11T22:21:14.368Z","updated_at":"2025-10-11T13:34:28.107Z","avatar_url":"https://github.com/ctsit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# REDCap Packaging and Deployment Toolset\n\n[![DOI](https://zenodo.org/badge/57070252.svg)](https://zenodo.org/badge/latestdoi/57070252)\n\n## Overview\n\nThis project provides tools for scripted deployments and upgrades of REDCap instances and the extensions installed within them. The toolset achieves this through scripted building of packages of the REDCap with extensions as well as the scripted deployment of those packages to hosts. The goal of the project is to provide a tool set that can build packages rapidly and consistently across REDCap version numbers and deploy those packages to new and existing REDCap instances. This reduces the variability between development, testing, and production environments. This in turn reduces the error rates, the cost of testing, and the costs of upgrading REDCap instances.\n\nAncillary to this goal, this project provides a local REDCap instance that can be used as an educational REDCap tool and/or a software development test bed. You can use this project for any or all of these goals.\n\n\n## Requirements\n\n### REDCap\n\nA user of these tools will need to download and provide their own REDCap zip\nfile, downloaded from Vanderbilt. This REDCap .zip should be placed in the\nroot folder of this project. It should not be renamed.\n\n### Virtual Machine\n\nThis project provides a virtual machine wherein it hosts the local REDCap instance. Creating the virtual machine (VM) required the software packages Vagrant, VirtualBox, the vagrant-hostsupdater plugin and the vagrant-env plugin be installed on the host system.\n\n### Packaging and Deployment\n\nThe packaging and deployment tools are designed to deploy REDCap to Debian Linux hosts. They may or may not work with non-Debian REDCap hosts.  They cannot deploy REDCap to Windows hosts. The packaging and deployment tools are written using the [Fabric](http://www.fabfile.org/) system. The tools use Fabric 1.x\n\n\n## Installing dependencies\n\n### Install Vagrant and Virtual Box\n\nOn a Linux machine run these commands in a shell:\n\n```bash\nsudo apt-get install vagrant\nsudo apt-get install virtualbox\n```\n\nOn a Mac OSX machine:\n\n-   Download and install vagrant from \u003chttps://www.vagrantup.com/downloads.html\u003e\n-   Download and install the latest virtual box from \u003chttp://download.virtualbox.org/virtualbox/\u003e\n\nOn Mac OSX users using [Homebrew](http://brew.sh/) can install these packages\nusing the _brew_ command.  Run these commands at a shell:\n\n```bash\nbrew cask install virtualbox\nbrew cask install vagrant\n```\n\n\n### Install Vagrant plugins\n\nVagrant will need a few plugins for this VM. On any platform, run these commands in a shell:\n\n```bash\nvagrant plugin install vagrant-hostsupdater\nvagrant plugin install vagrant-env\nvagrant plugin install vagrant-vbguest\n```\n\nFor more details about Vagrant software you can go to [why-vagrant](https://www.vagrantup.com/intro/index.html#why-vagrant-) page.\n\n### Install REDCap Modules\nREDCap Deployment supports [REDCap Modules](https://github.com/vanderbilt/redcap-external-modules). In order to deploy external modules, you need to set up `settings/modules.json` file and reference it in your instance's settings.\n\nThis project provides an example file, which references a module provided by CTS-IT team. You may copy the file `settings/modules.json.example` to the name `settings/modules.json`, and customize it to your needs.\n```bash\ncp settings/modules.json.example settings/modules.json\n```\n\nHere is how `settings/modules.json` should look like:\n```json\n[\n    {\n        \"name\": \"linear_data_entry_workflow\",\n        \"version\": \"2.0.0\",\n        \"repo\": \"https://github.com/ctsit/linear_data_entry_workflow.git\",\n        \"branch\": \"develop\"\n    }\n]\n```\n\n\n### Get your REDCap zip file\n\nYou must provide a copy of the REDCap software from \u003chttps://projectredcap.org/\u003e. Save the .zip file with its default name to the root of this repository. This ensures the packaging and provisioning scripts can locate the REDCap code when needed.\n\n\n## Configure the Virtual Machine\n\nThe development environment needs to be configured before it can be started.\nCopy the file `example.env.txt` to the name `.env` and customize it for your\nuse. Minimally, you will need to set _smtp\\_smarthost_ to the dns name of a mail\nserver your development host can use to deliver mail.  This will allow you to\nbetter test features that send email.\n\n```bash\ncp example.env.txt .env\n```\n\n\n## Using the testing and development environment\n\nWith the above requirements and configuration completed, start the VM with the command\n\n```bash\nvagrant up\n```\n\nThe vagrant-hostsupdater plugin will make modifications to your hosts file as the VM starts.  If it prompts you for a password, provide the password you use to login to your computer.\n\nAfter about two minutes, the VM should be accessible at the value of the variable `URL_OF_DEPLOYED_APP` set in `.env`. By default this is [http://redcap.test/redcap/](http://redcap.test/redcap/)\n\n\n## (Re)deploying REDCap with Fabric Tools\n\nIn addition to the REDCap deployed by the Vagrant provisioning scripts, this repository includes a suite of deployment and upgrade tools that can configure a host for deployment, package REDCap with numerous extensions, deploy a new REDCap instance and upgrade an existing one.  You can use these commands on any Linux host where you have sufficient privileges or against this vagrant-deployed VM.\n\n### Fabric Prerequisites\n\nThis tool is written in Python 3 and uses Fabric 1.x methods. To use it, make sure you install the latest Fabric 1.x.  See https://www.fabfile.org/installing-1.x.html for details, the TL;DR version is\n\n```bash\npip install 'fabric\u003c2.0'\n```\n\nIf you make a mess of things before you get it right, you might have to uninstall the mess like this:\n\n```bash\nbrew uninstall fabric\npip uninstall 'fabric\u003c2.0'\npip install 'fabric\u003c2.0'\n```\n\n\n### Configure Fabric for the Virtual Machine\n\nThe Fabric tools need to be configured for the Vagrant VM before they can be used.\nCopy the file `settings/vagrant.ini.example` to the name `settings/vagrant.ini`, and customize it to your needs.\n\n```bash\ncp settings/vagrant.ini.example settings/vagrant.ini\n```\n\nCustomization is not _required_ but it is useful to add patches and language modules.\n\n\n### REDCap Deployment\n\nIf you have a REDCap zip file, say redcap7.2.2.zip, you can deploy it to the local Vagrant REDCap instance with these commands:\n\n```bash\nfab vagrant server_setup\nfab vagrant package:redcap7.2.2.zip\nfab vagrant delete_all_tables deploy:redcap-7.2.2.tgz\n```\n\n\n### REDCap upgrade\n\nAny upgrade to 7.3.0 would be as simple as\n\n```bash\nfab vagrant package:redcap7.3.0.zip\nfab vagrant upgrade:redcap-7.3.0.tgz\n```\n\nNote that you do not have to use REDCap's _upgrade_ zip files. The `upgrade` method of the Fabric tools knows to not deploy the few files that would be hazardous to an existing REDCap instance.\n\nIf the tests fail and the server is left offline, you can put it back online with\n\n```bash\nfab vagrant online\n```\n\n## Database errors\n\nSometimes the upgrade fails in the late stages with a SQL error. When this happens, the system is left offline and the database is in an inconsistent state. You will need to note the error message, review the database upgrade `.SQL` or `.PHP` file it was processing, determine the fault and correct it. Once it is corrected, you will need to re-apply the database changes the upgrade command was trying to apply when it failed and then reapply the rest of the changes. To do that, run the task `upgrade_apply_incremental_db_changes_only`. e.g.\n\n```bash\n# This command failed\nfab instance:warrior upgrade:redcap-13.11.4.tgz\n\n# ... so now run this\nfab instance:warrior upgrade.upgrade_apply_incremental_db_changes_only:redcap-13.11.4.tgz\n```\n\nIt's a long ugly task name, but it gets the job done. It will assure redcap instance is offline, apply the incremental changes, bump the redcap version, log the upgrade, and bring REDCap back online.\n\n\n## PHP Configuration\n\nWhile the deployment scripts in this repo manage the PHP file upload size for the local VM, they do not do the same for a remote host. To do that use commands much like these to increase the upload file size limits\n\n### Upgrade PHP from 7.2 to 7.4\n\n```bash\n# Upgrade PHP from 7.2 to 7.4\nsudo apt install -y libapache2-mod-php7.4 \\\n  php7.4 \\\n  php7.4-cli \\\n  php7.4-common \\\n  php7.4-curl \\\n  php7.4-gd \\\n  php7.4-imap \\\n  php7.4-json \\\n  php7.4-mbstring \\\n  php7.4-mysql \\\n  php7.4-odbc \\\n  php7.4-opcache \\\n  php7.4-readline \\\n  php7.4-soap \\\n  php7.4-xml \\\n  php7.4-zip\n\nsudo a2dismod php7.2\nsudo a2enmod php7.4\nsudo systemctl restart apache2\n\ncd /etc\nsudo -E git add .\nsudo -E git commit -m \"Commit PHP upgrades and other files\"\n\ncd /etc/php\ngrep -lr upload_max_filesize * | sudo xargs -i sed \"s/upload_max_filesize.*/upload_max_filesize = 256M/;\" -i {}\ngrep -lr post_max_size * | sudo xargs -i sed \"s/post_max_size.*/post_max_size = 256M/;\" -i {}\ngrep -lr max_input_vars * | sudo xargs -i sed \"s/.*max_input_vars.*/max_input_vars = 100000/;\" -i {}\ngrep -lr session.cookie_secure * | sudo xargs -i sed \"s/.*session.cookie_secure.*/session.cookie_secure = On/;\" -i {}\n\ncd /etc\nsudo -E git add .\nsudo -E git commit -m \"Commit PHP configuration changes\"\n\n# install composer\ncd /tmp\nphp -r \"copy('https://getcomposer.org/installer', 'composer-setup.php');\"\n# safety not guaranteed\n# php -r \"if (hash_file('sha384', 'composer-setup.php') === '906a84df04cea2aa72f40b5f787e49f22d4c2f19492ac310e8cba5b96ac8b64115ac402c8cd292b8a03482574915d1a8') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;\"\nphp composer-setup.php\nphp -r \"unlink('composer-setup.php');\"\nsudo mv composer.phar /usr/local/bin/composer\n```\n\n### Upgrade PHP from 7.4 to 8.1\n\n```bash\n# Upgrade PHP from 7.4 to 8.0\nsudo apt install -y libapache2-mod-php8.1 \\\n  php8.1 \\\n  php8.1-cli \\\n  php8.1-common \\\n  php8.1-curl \\\n  php8.1-gd \\\n  php8.1-imap \\\n  php8.1-json \\\n  php8.1-mbstring \\\n  php8.1-mysql \\\n  php8.1-odbc \\\n  php8.1-opcache \\\n  php8.1-readline \\\n  php8.1-soap \\\n  php8.1-xml \\\n  php8.1-zip\n\ncd /etc\nsudo -E git add .\nsudo -E git commit -m \"Commit PHP upgrades and other files\"\n\ncd /etc/php\ngrep -lr upload_max_filesize * | sudo xargs -i sed \"s/upload_max_filesize.*/upload_max_filesize = 256M/;\" -i {}\ngrep -lr post_max_size * | sudo xargs -i sed \"s/post_max_size.*/post_max_size = 256M/;\" -i {}\ngrep -lr max_input_vars * | sudo xargs -i sed \"s/.*max_input_vars.*/max_input_vars = 100000/;\" -i {}\ngrep -lr session.cookie_secure * | sudo xargs -i sed \"s/.*session.cookie_secure.*/session.cookie_secure = On/;\" -i {}\n\ncd /etc\nsudo -E git add .\nsudo -E git commit -m \"Commit PHP configuration changes\"\n\n# Switch to new PHP in Apache\nsudo a2dismod php7.4\nsudo a2enmod php8.1\nsudo systemctl restart apache2\n```\n\n### Upgrade PHP from 7.4 to 8.2\n\n```bash\n# Upgrade PHP from 7.4 to 8.2\nsudo apt install -y libapache2-mod-php8.2 \\\n  php8.2 \\\n  php8.2-cli \\\n  php8.2-common \\\n  php8.2-curl \\\n  php8.2-gd \\\n  php8.2-imap \\\n  php8.2-mbstring \\\n  php8.2-mysql \\\n  php8.2-odbc \\\n  php8.2-opcache \\\n  php8.2-readline \\\n  php8.2-soap \\\n  php8.2-xml \\\n  php8.2-zip\n\ncd /etc\nsudo -E git add .\nsudo -E git commit -m \"Commit PHP upgrades and other files\"\n\ncd /etc/php\ngrep -lr upload_max_filesize * | sudo xargs -i sed \"s/upload_max_filesize.*/upload_max_filesize = 256M/;\" -i {}\ngrep -lr post_max_size * | sudo xargs -i sed \"s/post_max_size.*/post_max_size = 256M/;\" -i {}\ngrep -lr max_input_vars * | sudo xargs -i sed \"s/.*max_input_vars.*/max_input_vars = 100000/;\" -i {}\ngrep -lr session.cookie_secure * | sudo xargs -i sed \"s/.*session.cookie_secure.*/session.cookie_secure = On/;\" -i {}\ngrep -lr date.timezone * | sudo xargs -i sed \"s/.*date.timezone.*=.*/date.timezone = 'America\\/New_York'/;\" -i {}\n\ncd /etc\nsudo -E git add .\nsudo -E git commit -m \"Commit PHP configuration changes\"\n\n# fix imagick\nsudo apt install -y php-imagick\nsudo sed -i 's/policy domain=\"coder\" rights=\"none\" pattern=\"PDF\"/policy domain=\"coder\" rights=\"read\" pattern=\"PDF\"/;' /etc/ImageMagick-6/policy.xml\ncd /etc\nsudo -E git add .\nsudo -E git commit -m \"Install php-imagick and adjust policy to REDCap requirements\"\n\n# Switch to new PHP in Apache\nsudo a2dismod php7.4\nsudo a2enmod php8.2\nsudo systemctl restart apache2\n```\n\n### Install specific PHP packages\n\nOn some hosts, you might need to install a specific packages. At UF, we have one host we call \"warrior\" that needs the `mpdf` package. To install a custom package, first, install `composer`\n\n```sh\n# install composer\ncd /tmp\nphp -r \"copy('https://getcomposer.org/installer', 'composer-setup.php');\"\n# safety not guaranteed\n# php -r \"if (hash_file('sha384', 'composer-setup.php') === '906a84df04cea2aa72f40b5f787e49f22d4c2f19492ac310e8cba5b96ac8b64115ac402c8cd292b8a03482574915d1a8') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;\"\nphp composer-setup.php\nphp -r \"unlink('composer-setup.php');\"\nsudo mv composer.phar /usr/local/bin/composer\n```\n\nThen run these steps in the current redcap Libraries directory as user deploy\n\n```bash\nsudo su - deploy\n#cd /var/https/stage_c/redcap_v14.3.14/Libraries/\ncd /var/www/prod/redcap_v14.6.2/Libraries/\n```\n\nThen run `composer require` with the package you need install:\n\n```sh\ncomposer require mpdf/mpdf:^8\nexit\n```\n\nRestart apache when you are done\n\n```sh\nsudo service apache2 restart\n```\n\n## Failed SQL upgrades\n\nIf an upgrade operation fails on a `.php` or a `.sql` file, the code will die like this:\n\n```\n[redcap.ctsi.ufl.edu] run: php /var/https/redcap/redcap_v14.1.3/generate_upgrade_sql_from_php.php /var/https/redcap/redcap_v14.1.3/Resources/sql/upgrade_14.01.01.php | mysql\n[redcap.ctsi.ufl.edu] out: ERROR 1553 (HY000) at line 5: Cannot drop index 'redcap_userid': needed in a foreign key constraint\n[redcap.ctsi.ufl.edu] out: \n\n\nFatal error: run() received nonzero return code 1 while executing!\n\nRequested: php /var/https/redcap/redcap_v14.1.3/generate_upgrade_sql_from_php.php /var/https/redcap/redcap_v14.1.3/Resources/sql/upgrade_14.01.01.php | mysql\nExecuted: /bin/bash -l -c \"php /var/https/redcap/redcap_v14.1.3/generate_upgrade_sql_from_php.php /var/https/redcap/redcap_v14.1.3/Resources/sql/upgrade_14.01.01.php | mysql\"\n\nAborting.\nDisconnecting from deploy@redcap.ctsi.ufl.edu... done.\n```\n\nAt this point things get messy and you will have to get creative in your problem solving. What follows is not a recipe. It's more like a pile of ingredients from which you could make an omelette. It's still your responsibility cook them right and not burn the omelette, because you're gonna have to eat it.\n\n### Debugging\n\nYou can see the output of the script/SQl file that caused the by running that PHP command or catting out that SQL file that failed. e.g.,\n\n```bash\nphp /var/https/redcap/redcap_v14.1.3/generate_upgrade_sql_from_php.php /var/https/redcap/redcap_v14.1.3/Resources/sql/upgrade_14.01.01.php\n```\n\nRunning the SQL lines in that file one by one might help you understand which one failed. It's challenging because some of those commands will fail because they _did_ run successfully the first time, but they will error the second time. This often happens when a SQL command added an object that cannot be \"re-added\". Think carefully about the responses you see when running each command. Read the REDCap community forum to see if others have experienced the errors you see and have a solution.\n\nEven if that task goes well and you can apply the file that failed there might be more files to run after that one. You can locate those by enumerating the files in the `redcap_vN.M.O/Resources/sql/` directory. Look for versioned file names that are higher than the version number on which the script failed. If this seems tedious, you might want to try the next procedure.\n\n\n### Ask REDCap to help you\n\nAnother approach is to generate all of the SQL upgrade commands between the version you are at and the version you are upgrading to by accessing the Upgrade page in the Control Center. It will make the whole blob, running the PHP scripts, concatenating their output with the SQL files and appending the lines that document the upgrade. All that's pretty useful. Do be aware that every SQL line that succeeded will still be in the blob of SQL the Control Center Upgrade page generates for you. It's more content to wade through, but it's all in one file with the last three \"upgrade event\" statements.\n\nIf you think you are done with the upgrade process, put the host back online via the Control Center _General Configuration_ page. Then check the Control Center _Configuration Check_ age for any SQL commands that still need to be applied. \n\n## Developer notes\n\nFor tips on developing in this environment, see [Development tools](docs/development_tools.md).\n\n\n## Language Configuration\n\nREDCap languages can be provided by modifying the _languages_ variable accordingly:\nas a json list, e.g., languages = [\"Chinese.6.4.3.ini\", \"German.ini\"], or\nas a file, i.e., languages = \u003clanguageFolder\u003e, in which case the language files *.ini must be located inside the specified folder\n\n\n## Contributions\n\nThis repository was created to meet the needs of the UF CTSI REDCap Team.  We\nhave shared it as an example of how scripted deployments can be done in a\nDebian Linux environment.  We welcome contributions that parameterize our work\nto make these scripts more accessible to other REDCap sites.  Please fork this\nrepository to commit and share your work.  Please make pull requests against\nthe develop branch of this repo if you would like to make a contribution.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctsit%2Fredcap_deployment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fctsit%2Fredcap_deployment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctsit%2Fredcap_deployment/lists"}