{"id":19713408,"url":"https://github.com/ctsit/redi2","last_synced_at":"2025-04-29T18:31:13.368Z","repository":{"id":79003832,"uuid":"87186201","full_name":"ctsit/redi2","owner":"ctsit","description":"REDCap Electronic Data - I (Ingester/Integrator/Importer) 2","archived":true,"fork":false,"pushed_at":"2018-10-15T21:12:06.000Z","size":14950,"stargazers_count":4,"open_issues_count":3,"forks_count":6,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-15T05:16:07.387Z","etag":null,"topics":["deprecated","redcap"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ctsit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-04T12:54:43.000Z","updated_at":"2023-10-12T18:30:28.000Z","dependencies_parsed_at":"2023-04-20T19:31:35.127Z","dependency_job_id":null,"html_url":"https://github.com/ctsit/redi2","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctsit%2Fredi2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctsit%2Fredi2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctsit%2Fredi2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctsit%2Fredi2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ctsit","download_url":"https://codeload.github.com/ctsit/redi2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251559856,"owners_count":21609089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deprecated","redcap"],"created_at":"2024-11-11T22:22:00.473Z","updated_at":"2025-04-29T18:31:13.355Z","avatar_url":"https://github.com/ctsit.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# REDI2 v1.4.1\nREDCap Electronic Data - I (Ingester/Integrator/Importer) 2\n\nREDI2 is a suite of tools designed to transform the CSV data and load it into the fields and events of a REDCap project. REDI2 excels at transforming large, longitudinal data sets and loading them into REDCap in a scripted, repeatable fashion. REDI2 is fast and has minimal effects on REDCap server load.\n\n## Meet the parts of redi2 ##\n![data_flow](https://docs.google.com/drawings/d/1bVDUGXkr1n2RrGORnIeeY2nuyVz8BhUtTBgcgNcQeKw/pub?w=843\u0026h=713)\n\nredi2 is really just a collection of tools and a few scripts which help you get set up.\n\n### Claw ###\n\nClaw grabs files over sftp\n\n### Auditor ###\n\nAuditor redacts, maps, cleans and basically does whatever to the output of claw which should be a CSV.\n\n### Optimus ###\n\nOptimus takes your particular CSV and your project specific rules and transforms the data into a form which should\nbe easy to make right with redcap. Here is when branching logic should be applied and any derived data which\ncan be inferred, but is not explicitly present in your raw CSV should be made.\n\n\n### Lineman ###\n\nLineman is the first tool to talk to redcap. It grabs information related to the data you are passing in and\nvalidates that it can be imported. It will change the fields that it must in order to make sure that it will go in.\nFor example, it can take dates and map them to the correct event in the redcap.\n\n### Pigeon ###\n\nPigeon carries the data to redcap. This is the last step and the only one where data will change on the\nredcap server. This tool takes a greedy approach, so it will attempt to push in all that it can at once.\nIt can recover from errors and will keep trying to push the data in till it gives up and pushes in one record\nat a time.\n\n## Vagrant testing ##\n\nRED-I2 can be tested on a vagrant. In order to do so the following steps need to be completed in order.\n\n- clone the repo\n- go into the redi2/vagrant/vagrant directory\n- get a copy of redcap version 6.16.8 zip file and place in the redi2/vagrant/vagrant directory with the name redcap6.16.8.zip\n- rename the file `rename_to_dot_env` to `.env` in the same directory `$ cp rename_to_dot_env .env`\n- run vagrant up. (Make sure to install the plugins in the plugins.txt file)\n- open your browser to http://redi2.dev/redcap/\n-- go to Projects \u003e HCV Target \u003e API, take note of your token\n- run vagrant ssh from the vagrant/vagrant directory and go to ~/redi2/synthetic_data\n- edit the synth.pigeon.conf.yaml and insert the token from before\n- run `bash prep_synthetic_data.sh` if it fails then your install went wrong or more likely you dont have the right token\n\nNow if you check your redcap you should see that there are four subjects entered. These are used in the full synthetic\ndata run that we are about to do. Go back the terminal which is logged into the vagrant and do the following:\n\n- cd ~\n- cd redi2/synthetic_data\n- In the synth.optimus.conf.yaml, synth.lineman.conf.yaml change the token to the one in the synth.pigeon.conf.yaml\n- run `bash synth_run_prepare.sh`\n- cd ~/redi2/redi2/NEW_SITE\n- run `bash synth.run.sh`\n- go look at your redcap and see the data that has been added. Additionally check the NEW_SITE/data folder for the files generated during the run\n\n## Deploying RED-I 2 ##\n\nThere are two strategies for deployment depending on the what the deploy target looks like.\nIf you are able to talk to both github and PyPi on the hosting system then follow the first install.\nIf your system is more locked down and you cannot talk to either github or PyPi then follow install 2.\n\n### Install 1 ###\n\nWhat follows are the requirements and steps to install redi2 with access to both github and PyPi\n\n#### Requirements ####\n\n  * git\n  * python3\n  * virtualenv\n  * access to github\n  * access to PyPi\n\n#### Steps ####\n\nClone this repo and enter it.\n\nRun `bash install_normal.sh`.\n\nEdit the `NEW_SITE` directory in the `redi2` directory to have the right name.\n\nBuild your configs in the `configs` directory\n\nSet the `run.sh` script to run when you want it to.\n\n### Install 2 ###\n\nThis install is for those who have a target system that is not able to call out to github or PyPi for\nwhatever reason. The general strategy is to do as much work in an environment like the one to which you\nwill be deploying and bringing all that stuff over.\n\n#### Requirements ####\n\n##### System 1: talks to outside world system #####\n  * git\n  * python3\n  * virtualenv\n  * tar\n  * access to github\n  * access to PyPi\n\n##### System 2: target system #####\n  * ssh access\n  * python3\n  * virtualenv\n\n#### Vagrant Steps ####\n\nBy using vagrant up, one gets a copy of the redi2.tar.gz in `repo/vagrant/redi2_deploy_tar` that is used in the\nmanual steps. It will be build on the debian jessie vagrant and bundled from there. If you want to skip the\nsteps that take place on System 1 then this is what you should do.\n\n##### Host machine #####\n\n`cd redi2/vagrant/vagrant`\n\n`vagrant up`\n\nNow go to the \"Host machine\" section of the \"Manual Steps\" and continue from there using the tar in\n`redi2/vagrant/redi2_deploy_tar`\n\n#### Manual Steps ####\n\n##### System 1 #####\n\nClone this repo and enter it on System 1.\n\n`cd redi2/build_scripts`\n\nRun `bash package.sh`.\n\nReturn to the directory where redi2 was cloned.\n\n##### Host machine #####\n\nUse `scp` to get the `redi2.tar.gz` directory to System 2. It will be in the directory above.\n\nUse `ssh` to gain access to System 2.\n\n##### System 2 #####\n\nRun `tar -xzf redi2.tar.gz` to extract the directory in the location that you want to install redi2\n\n`cd redi2/deploy_scripts`\n\nRun `bash install_two_remote.sh`\n\nEdit the `NEW_SITE` directory in the `redi2` directory to have the right name.\n\nBuild your configs in the `configs` directory\n\nAdd `source ../correct_path.env` to the line right below the virtualenv source line in the run.sh\n\nSet the `run.sh` script to run when you want it to.\n\n\n## Putting it all together ##\n\nOnce one has built their configs all that remains is to run the tools in the right order. Pretty easy!\nHappy redcapping!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctsit%2Fredi2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fctsit%2Fredi2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctsit%2Fredi2/lists"}