{"id":17322308,"url":"https://github.com/marlycormar/icare","last_synced_at":"2026-04-29T21:01:44.129Z","repository":{"id":152036195,"uuid":"122672042","full_name":"marlycormar/iCARE","owner":"marlycormar","description":"Convert sqlite db and tabular data into a mysql db for the iCARE project","archived":false,"fork":false,"pushed_at":"2020-07-28T23:10:27.000Z","size":56,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"develop","last_synced_at":"2025-02-01T09:12:51.817Z","etag":null,"topics":["icare","mysql","python","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marlycormar.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-23T20:59:13.000Z","updated_at":"2020-07-28T23:10:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"25d39736-271c-4dbe-ba6c-54599658b7a5","html_url":"https://github.com/marlycormar/iCARE","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marlycormar%2FiCARE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marlycormar%2FiCARE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marlycormar%2FiCARE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marlycormar%2FiCARE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marlycormar","download_url":"https://codeload.github.com/marlycormar/iCARE/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245778161,"owners_count":20670682,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["icare","mysql","python","sqlite"],"created_at":"2024-10-15T13:41:40.149Z","updated_at":"2025-10-14T04:04:39.561Z","avatar_url":"https://github.com/marlycormar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"##  Install aws console\n\n- Install aws:\n\n        pip install aws-shell\n\n- Run the shell:\n\n        aws-shell\n\n- If it fails with error `command not found`, add `~/.local/bin` to the current `$PATH` variable:\n\n        export PATH=~/.local/bin:$PATH\n\nand run `aws-shell` again.\n\n- To exit the shell press `fn + F10`.\n\n\n## Configure aws\n\n- Once inside the `aws-shell` run:\n\n        configure\n\n- You will be prompted to provide the following information:\n    - AWS Access Key ID:\n    - AWS Secret Access Key:\n    - Default region name: us-east-1\n    - Default output format:\n\n\n- Test that you can enumerate the Farsight files in the bucket:\n\n        s3 ls s3://fgs-ufh-farsight-results\n\n- Exit the shell.\n\n\n## Download all files from the amazon bucket\n\n    cd $directory_with_farsight_files\n    aws s3 sync s3://fgs-ufh-farsight-results .\n\n\n## Convert amazon bucket tabular data into a mysql db\n\nUse a python script.\n\n- If needed, install the mysql modules for python\n\n        pip install mysql-python\n\n- Configure Python script\n\nThe python script reads its configuration from the environment.  These variables are required:\n\n    export mysql_db=icare\n    export directory_with_farsight_files=/Users/pbc/tmp\n    export sql_dump=../honeyguide/db/farsight.sql\n\n- Run the script by doing:\n\n        python2.7 create_and_populate_table.py\n\n## Get data from REDCap\n\nWe will use [QUAIL](git@github.com:ctsit/QUAIL.git) to get data from REDCap projects.\n\nNote: Quail requires Python3.\n\n- Install QUAIL's prerequisites\n\n        git clone git@github.com:ctsit/cappy.git\n        pip install -e cappy\n\n- Install QUAIL as a python module by first cloning it in a local directory:\n\n        git clone git@github.com:marlycormar/QUAIL.git\n\n    and then installing it:\n\n        pip install -e QUAIL\n\n- General usage:\n\n        quail install \u003croot\u003e\n        quail redcap generate ( \u003cquail.conf.yaml\u003e \u003cproject_name\u003e \u003ctoken\u003e \u003curl\u003e ) [-i --initialize]\n        quail redcap get_meta (\u003cproject_name\u003e) [ -q \u003cquail.conf.yaml\u003e ]\n        quail redcap get_data (\u003cproject_name\u003e) [ -q \u003cquail.conf.yaml\u003e ]\n        quail redcap gen_meta (\u003cproject_name\u003e) [ -q \u003cquail.conf.yaml\u003e ]\n        quail redcap gen_data (\u003cproject_name\u003e) [ -q \u003cquail.conf.yaml\u003e ]\n        quail redcap make_import_files (\u003cproject_name\u003e) [ -q \u003cquail.conf.yaml\u003e ]\n\n- To run QUAIL:\n\n        export directory_for_redcap_data=`mktemp -d`\n        quail install $directory_for_redcap_data\n        export OLDDIR=`pwd`\n        cd $directory_for_redcap_data\n        quail redcap generate quail.conf.yaml \"Malignant\" 78JNMKDR346hhKJbj85DE4 http://redcap.test/redcap/api/\n        quail redcap get_meta \"Malignant\"\n        quail redcap get_data \"Malignant\"\n        quail redcap gen_meta \"Malignant\"\n        quail redcap gen_data \"Malignant\"\n        quail redcap make_import_files \"Malignant\"\n        export TODAYS_DATE=`date +%Y-%m-%d`\n        cd $OLDDIR\n\n- This will save the information for the REDCap project into a sqlite database in the directory `$directory_for_redcap_data/batches/Malignant/2018-03-19/data.db` were `2018-03-19` represents the current date.\n\n\n## Convert sqlite db into mysql db\n\n- The python script reads its configuration from the environment.  These variables are required:\n\n        export mysql_db=icare\n        export path_to_sqlitedb=$directory_for_redcap_data/batches/Malignant/$TODAYS_DATE/data.db\n        export path_to_mysql_dump=~/git/honeyguide/db/malignant.sql\n\n- Run script:\n\n        python sqlite_to_mysql.py\n\n- Erase the SQLite inputs\n    \n        rm -rf $directory_for_redcap_data\n\n\n## Using honeyguide\n\n- Go to https://github.com/marlycormar/honeyguide/tree/mysql, clone the repo locally, and checkout the mysql branch.\n\n        git clone git@github.com:marlycormar/honeyguide.git\n        cd honeyguide\n        git checkout mysql\n\n- Configure Honeyguide to load our data into a MySQL DB. Use fake.env as a template.  Copy it to \".env\" and customize as needed.\n\n        cp fake.env .env\n        # edit .env\n\n- Build and start honeyguide\n\n        docker-compose up --build\n\n- To access the web interface of redash go to [http://localhost:5000/](http://localhost:5000/).\n\n- To add the datasource, you will need the container ip address. To get that, inspect the container and look for the `Gateway` ip address:\n\n        docker inspect container_id\n\n- To add a mysql datasource you will need the following information:\n\n        Host: container_ip\n        Port: 3306\n        Password: password\n        Database Name: redash\n\nNotice that both the `malignant` and the `farsight` data sources will be loaded into the `icare` database in docker mysql instance.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarlycormar%2Ficare","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarlycormar%2Ficare","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarlycormar%2Ficare/lists"}