{"id":36653707,"url":"https://github.com/dimagi/commcare-export","last_synced_at":"2026-01-12T10:15:05.327Z","repository":{"id":6891233,"uuid":"8140822","full_name":"dimagi/commcare-export","owner":"dimagi","description":"A command-line tool and Python library to generate customized exports from CommCareHQ.","archived":false,"fork":false,"pushed_at":"2025-11-08T13:02:06.000Z","size":1456,"stargazers_count":8,"open_issues_count":16,"forks_count":19,"subscribers_count":27,"default_branch":"master","last_synced_at":"2025-11-08T14:17:09.639Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dimagi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2013-02-11T15:48:46.000Z","updated_at":"2025-11-08T12:04:05.000Z","dependencies_parsed_at":"2024-03-08T08:25:13.446Z","dependency_job_id":"713bcefc-cb98-471d-9682-c284427fc039","html_url":"https://github.com/dimagi/commcare-export","commit_stats":{"total_commits":769,"total_committers":28,"mean_commits":"27.464285714285715","dds":0.5318595578673602,"last_synced_commit":"f41d4bba992c4827466f626fa86767c94ecb7b9c"},"previous_names":[],"tags_count":93,"template":false,"template_full_name":null,"purl":"pkg:github/dimagi/commcare-export","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-export","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-export/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-export/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-export/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dimagi","download_url":"https://codeload.github.com/dimagi/commcare-export/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-export/sbom","scorecard":{"id":343134,"data":{"date":"2025-08-11","repo":{"name":"github.com/dimagi/commcare-export","commit":"3f744344b162e9bbcc52afb78d7982a9c90ea87b"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/release_actions.yml:1","Warn: no topLevel permission defined: .github/workflows/test.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":9,"reason":"11 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 9","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Info: Possibly incomplete results: error parsing shell code: \u003e must be followed by a word: examples/scheduled_run_linux.sh:0","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release_actions.yml:12: update your workflow using https://app.stepsecurity.io/secureworkflow/dimagi/commcare-export/release_actions.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release_actions.yml:26: update your workflow using https://app.stepsecurity.io/secureworkflow/dimagi/commcare-export/release_actions.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release_actions.yml:36: update your workflow using https://app.stepsecurity.io/secureworkflow/dimagi/commcare-export/release_actions.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release_actions.yml:50: update your workflow using https://app.stepsecurity.io/secureworkflow/dimagi/commcare-export/release_actions.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:36: update your workflow using https://app.stepsecurity.io/secureworkflow/dimagi/commcare-export/test.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:46: update your workflow using https://app.stepsecurity.io/secureworkflow/dimagi/commcare-export/test.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/test.yml:70: update your workflow using https://app.stepsecurity.io/secureworkflow/dimagi/commcare-export/test.yml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/release_actions.yml:17","Warn: pipCommand not pinned by hash: .github/workflows/release_actions.yml:21","Warn: pipCommand not pinned by hash: .github/workflows/release_actions.yml:22","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:52","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:53","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:55","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:56","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:57","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:58","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:59","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:60","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   3 third-party GitHubAction dependencies pinned","Info:   0 out of  11 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact 1.13.0 not signed: https://api.github.com/repos/dimagi/commcare-export/releases/150904202","Warn: release artifact 1.12.0 not signed: https://api.github.com/repos/dimagi/commcare-export/releases/150282104","Warn: release artifact 1.11.0 not signed: https://api.github.com/repos/dimagi/commcare-export/releases/148653914","Warn: release artifact 1.13.0 does not have provenance: https://api.github.com/repos/dimagi/commcare-export/releases/150904202","Warn: release artifact 1.12.0 does not have provenance: https://api.github.com/repos/dimagi/commcare-export/releases/150282104","Warn: release artifact 1.11.0 does not have provenance: https://api.github.com/repos/dimagi/commcare-export/releases/148653914"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":4,"reason":"6 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2017-48 / GHSA-chqf-hx79-gxc6","Warn: Project is vulnerable to: GHSA-v9hf-5j83-6xpp","Warn: Project is vulnerable to: GHSA-34jh-p97f-mpxf","Warn: Project is vulnerable to: PYSEC-2023-212 / GHSA-g4mx-q9vg-27p4","Warn: Project is vulnerable to: GHSA-pq67-6m6q-mj2v","Warn: Project is vulnerable to: PYSEC-2023-192 / GHSA-v845-jxx5-vc9f"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-18T06:25:26.773Z","repository_id":6891233,"created_at":"2025-08-18T06:25:26.773Z","updated_at":"2025-08-18T06:25:26.773Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28338064,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T06:09:07.588Z","status":"ssl_error","status_checked_at":"2026-01-12T06:05:18.301Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-12T10:15:03.887Z","updated_at":"2026-01-12T10:15:05.311Z","avatar_url":"https://github.com/dimagi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"CommCare Export\n===============\n\nhttps://github.com/dimagi/commcare-export \n\n[![Build Status](https://app.travis-ci.com/dimagi/commcare-export.svg?branch=master)](https://app.travis-ci.com/dimagi/commcare-export)\n[![Test coverage](https://coveralls.io/repos/dimagi/commcare-export/badge.png?branch=master)](https://coveralls.io/r/dimagi/commcare-export)\n[![PyPI version](https://badge.fury.io/py/commcare-export.svg)](https://badge.fury.io/py/commcare-export)\n\nA command-line tool (and Python library) to generate customized exports from the [CommCare HQ](https://www.commcarehq.org) [REST API](https://wiki.commcarehq.org/display/commcarepublic/Data+APIs).\n\n* [User documentation](https://wiki.commcarehq.org/display/commcarepublic/CommCare+Data+Export+Tool)\n* [Changelog](https://github.com/dimagi/commcare-export/releases)\n\nInstallation \u0026 Quick Start\n--------------------------\n\nFollowing commands are to be run on a terminal or a command line.\n\nOnce on a terminal window or command line, for simplicity, run commands from the home directory.\n\n### Python\n\nCheck which Python version is installed.\n\nThis tool is tested with Python versions from 3.9 to 3.13.\n\n```shell\n$ python3 --version\n```\nIf Python is installed, its version will be shown.\n\nIf Python isn't installed, [download and install](https://www.python.org/downloads/)\na version of Python from 3.9 to 3.13.\n\n## Virtualenv (Optional)\n\nIt is recommended to set up a virtual environment for CommCare Export\nto avoid conflicts with other Python applications.\n\nMore about virtualenvs on https://docs.python.org/3/tutorial/venv.html\n\nSetup a virtual environment using:\n\n```shell\n$ python3 -m venv venv\n```\n\nActivate virtual environment by running:\n\n```shell\n$ source venv/bin/activate\n```\n\n**Note**: virtualenv needs to be activated each time you start a new terminal session or command line prompt.\n\nFor convenience, to avoid doing that, you can create an alias to activate virtual environments in\n\"venv\" directory by adding the following to your\n`.bashrc` or `.zshrc` file:\n\n```shell\n$ alias venv='if [[ -d venv ]] ; then source venv/bin/activate ; fi'\n```\n\nThen you can activate virtual environments with simply typing\n```shell\n$ venv\n```\n\n## Install CommCare Export\n\n[uv](https://docs.astral.sh/uv/) is a fast Python package installer and resolver.\n\n```shell\n$ uv pip install commcare-export\n```\n\n## CommCare HQ\n\n1. Sign up for [CommCare HQ](https://www.commcarehq.org/) if you have not already.\n\n2. Create a project space and application.\n\n3. Visit the Release Manager, make a build, click the star to release it.\n\n4. Use Web Apps and fill out some forms.\n\n5. Modify one of example queries in the `examples/` directory, modifying the \"Filter Value\" column\n    to match your form XMLNS / case type. \n    See [this page](https://confluence.dimagi.com/display/commcarepublic/Finding+a+Form%27s+XMLNS) to \n    determine the XMLNS for your form.\n\nNow you can run the following examples:\n\n```shell\n$ commcare-export \\\n     --query examples/demo-registration.xlsx \\\n     --project YOUR_PROJECT \\\n     --output-format markdown\n\n$ commcare-export \\\n     --query examples/demo-registration.json \\\n     --project YOUR_PROJECT \\\n     --output-format markdown\n\n$ commcare-export \\\n     --query examples/demo-deliveries.xlsx \\\n     --project YOUR_PROJECT \\\n     --output-format markdown\n\n$ commcare-export \\\n     --query examples/demo-deliveries.json \\\n     --project YOUR_PROJECT \\\n     --output-format markdown\n```\n\nYou'll see the tables printed out. Change to `--output-format sql --output URL_TO_YOUR_DB --since DATE` to\nsync all forms submitted since that date.\n\nExample query files are provided in both Excel and JSON format.  It is recommended\nto use the Excel format as the JSON format may change upon future library releases.\n\nCommand-line Usage\n------------------\n\nThe basic usage of the command-line tool is with a saved Excel or JSON query (see how to write these, below)\n\n```shell\n$ commcare-export --commcare-hq \u003cURL or alias like \"local\" or \"prod\"\u003e \\\n                  --username \u003cusername\u003e \\\n                  --project \u003cproject\u003e \\\n                  --api-version \u003capi version, defaults to latest known\u003e \\\n                  --version \u003cprint current version\u003e \\\n                  --query \u003cexcel file, json file, or raw json\u003e \\\n                  --output-format \u003ccsv, xls, xlsx, json, markdown, sql\u003e \\\n                  --output \u003cfile name or SQL database URL\u003e \\\n                  --users \u003cexport data about project's mobile workers\u003e \\\n                  --locations \u003cexport data about project's location hierarchy\u003e \\\n                  --with-organization \u003cexport users, locations and joinable form or case tables\u003e\n```\n\nSee `commcare-export --help` for the full list of options.\n\n### Logging\n\nBy default, commcare-export writes logs to a file named\n`commcare_export.log` in the current working directory. Log entries are\nappended to this file across multiple runs to preserve history.\n\nYou can customize the log directory:\n\n```shell\n$ commcare-export --log-dir /path/to/logs \\\n     --query my-query.xlsx \\\n     --project myproject\n```\n\nTo disable file logging and show all output in the console only:\n\n```shell\n$ commcare-export --no-logfile \\\n     --query my-query.xlsx \\\n     --project myproject\n```\n\n\u003e [!NOTE]\n\u003e The log directory will be created automatically if it doesn't exist.\n\u003e If the specified directory cannot be created or written to,\n\u003e commcare-export will fall back to console-only logging with a warning\n\u003e message.\n\nThere are example query files for the CommCare Demo App (available on the CommCare HQ Exchange) in the `examples/`\ndirectory.\n\n`--output`\n\nCommCare Export uses SQLAlachemy's [create_engine](http://docs.sqlalchemy.org/en/latest/core/engines.html) to establish a database connection. This is based off of the [RFC-1738](https://www.ietf.org/rfc/rfc1738.txt) protocol. Some common examples:\n\n```\n# Postgres\npostgresql+psycopg2://scott:tiger@localhost/mydatabase\n\n# MySQL\nmysql+pymysql://scott:tiger@localhost/mydatabase\n\n# MSSQL\nmssql+pyodbc://scott:tiger@localhost/mydatabases?driver=ODBC+Driver+17+for+SQL+Server\n```\n\n\nExcel Queries\n-------------\n\nAn Excel query is any `.xlsx` workbook. Each sheet in the workbook represents one table you wish\nto create. There are two grouping of columns to configure the table:\n\n - **Data Source**: Set this to `form` to export form data, or `case` for case data.\n - **Filter Name** / *Filter Value*: These columns are paired up to filter the input cases or forms.\n - **Field**: The destination in your SQL database for the value.\n - **Source Field**: The particular field from the form you wish to extract. This can be any JSON path.\n\n\nJSON Queries\n------------\n\nJSON queries are a described in the table below. You build a JSON object that represents the query you have in mind.\nA good way to get started is to work from the examples, or you could make an Excel query and run the tool\nwith `--dump-query` to see the resulting JSON query.\n\n\nUser and Location Data\n----------------------\n\nThe --users and --locations options export data from a CommCare project that\ncan be joined with form and case data. The --with-organization option does all\nof that and adds a field to Excel query specifications to be joined on.\n\nSpecifying the --users option or --with-organization option will export an\nadditional table named 'commcare_users' containing the following columns:\n\n| Column                           | Type | Note                                |\n|----------------------------------|------|-------------------------------------|\n| id                               | Text | Primary key                         |\n| default_phone_number             | Text |                                     |\n| email                            | Text |                                     |\n| first_name                       | Text |                                     |\n| groups                           | Text |                                     |\n| last_name                        | Text |                                     |\n| phone_numbers                    | Text |                                     |\n| resource_uri                     | Text |                                     |\n| commcare_location_id             | Text | Foreign key to `commcare_locations` |\n| commcare_location_ids            | Text |                                     |\n| commcare_primary_case_sharing_id | Text |                                     |\n| commcare_project                 | Text |                                     |\n| username                         | Text |                                     |\n\nThe data in the 'commcare_users' table comes from the [List Mobile Workers\nAPI endpoint](https://confluence.dimagi.com/display/commcarepublic/List+Mobile+Workers).\n\nSpecifying the --locations option or --with-organization options will export\nan additional table named 'commcare_locations' containing the following columns:\n\n| Column                       | Type | Note                                          |\n|------------------------------|------|-----------------------------------------------|\n| id                           | Text |                                               |\n| created_at                   | Date |                                               |\n| domain                       | Text |                                               |\n| external_id                  | Text |                                               |\n| last_modified                | Date |                                               |\n| latitude                     | Text |                                               |\n| location_data                | Text |                                               |\n| location_id                  | Text | Primary key                                   |\n| location_type                | Text |                                               |\n| longitude                    | Text |                                               |\n| name                         | Text |                                               |\n| parent                       | Text | Resource URI of parent location               |\n| resource_uri                 | Text |                                               |\n| site_code                    | Text |                                               |\n| location_type_administrative | Text |                                               |\n| location_type_code           | Text |                                               |\n| location_type_name           | Text |                                               |\n| location_type_parent         | Text |                                               |\n| *location level code*        | Text | Column name depends on project's organization |\n| *location level code*        | Text | Column name depends on project's organization |\n\nThe data in the 'commcare_locations' table comes from the Location API\nendpoint along with some additional columns from the Location Type API\nendpoint. The last columns in the table exist if you have set up\norganization levels for your projects. One column is created for each\norganization level. The column name is derived from the Location Type\nthat you specified. The column value is the location_id of the containing\nlocation at that level of your organization. Consider the example organization\nfrom the [CommCare help page](https://confluence.dimagi.com/display/commcarepublic/Setting+up+Organization+Levels+and+Structure).\nA piece of the 'commcare_locations' table could look like this:\n\n| location_id | location_type_name | chw    | supervisor | clinic | district |\n|-------------|--------------------|--------|------------|--------|----------|\n| 939fa8      | District           | NULL   | NULL       | NULL   | 939fa8   |\n| c4cbef      | Clinic             | NULL   | NULL       | c4cbef | 939fa8   |\n| a9ca40      | Supervisor         | NULL   | a9ca40     | c4cbef | 939fa8   |\n| 4545b9      | CHW                | 4545b9 | a9ca40     | c4cbef | 939fa8   |\n\nIn order to join form or case data to 'commcare_users' and 'commcare_locations'\nthe exported forms and cases need to contain a field identifying which user\nsubmitted them. The --with-organization option automatically adds a field\ncalled 'commcare_userid' to each query in an Excel specification for this\npurpose. Using that field, you can use a SQL query with a join to report\ndata about any level of you organization. For example, to count the number\nof forms submitted by all workers in each clinic:\n\n```sql\nSELECT l.clinic,\n       COUNT(*)\nFROM form_table t\nLEFT JOIN (commcare_users u\n           LEFT JOIN commcare_locations l\n           ON u.commcare_location_id = l.location_id)\nON t.commcare_userid = u.id\nGROUP BY l.clinic;\n```\n\nNote that the table names 'commcare_users' and 'commcare_locations' are\ntreated as reserved names and the export tool will produce an error if\ngiven a query specification that writes to either of them.\n\nThe export tool will write all users to 'commcare_users' and all locations to\n'commcare_locations', overwriting existing rows with current data and adding\nrows for new users and locations. If you want to remove obsolete users or\nlocations from your tables, drop them and the next export will leave only\nthe current ones. If you modify your organization to add or delete levels,\nyou will change the columns of the 'commcare_locations' table and it is\nvery likely you will want to drop the table before exporting with the new\norganization.\n\nScheduling the DET\n------------------\nScheduling the DET to run at regular intervals is a useful tactic to keep your \ndatabase up to date with CommCare HQ.\n\nA common approach to scheduling DET runs is making use of the operating systems' scheduling\nlibraries to invoke a script to execute the `commcare-export` command. Sample scripts can be\nfound in the `examples/` directory for both Windows and Linux.\n\n### Windows\nOn Windows systems you can make use of the [task scheduler](https://sqlbackupandftp.com/blog/how-to-schedule-a-script-via-windows-task-scheduler/)\nto run scheduled scripts for you.\n\nThe `examples/` directory contains a sample script file, `scheduled_run_windows.bat`, which can be used by the\ntask scheduler to invoke the `commcare-export` command. \n\nTo set up the scheduled task you can follow the steps below.\n1. Copy the file `scheduled_run_windows.bat` to any desired location on your system (e.g. `Documents`)\n2. Edit the copied `.bat` file and populate your own details\n3. Follow the steps outlined [here](https://sqlbackupandftp.com/blog/how-to-schedule-a-script-via-windows-task-scheduler/),\nusing the .bat file when prompted for the `Program/script`.\n\n\n### Linux\nOn a Linux system you can make use of the [crontab](https://www.techtarget.com/searchdatacenter/definition/crontab)\ncommand to create scheduled actions (cron jobs) in the system.\n\nThe `examples/` directory contains a sample script file, `scheduled_run_linux.sh`, which can be used by the cron job.\nTo set up the cron job you can follow the steps below.\n1. Copy the example file to the home directory\n\u003e cp ./examples/scheduled_run_linux.sh ~/scheduled_run_linux.sh\n2. Edit the file to populate your own details\n\u003e nano ~/scheduled_run_linux.sh\n3. Create a cron job by appending to the crontab file\n\u003e crontab -e\n\nMake an entry below any existing cron jobs. The example below executes the script file at the top of\nevery 12th hour of every day\n\u003e 0 12 * * * bash ~/scheduled_run_linux.sh\n\nYou can consult the [crontab.guru](https://crontab.guru/) tool which is very useful to generate and interpret\nany custom cron schedules. \n\nPython Library Usage\n--------------------\n\nAs a library, the various `commcare_export` modules make it easy to\n\n - Interact with the CommCare HQ REST API\n - Execute \"Minilinq\" queries against the API (a very simple query language, described below)\n - Load and save JSON representations of Minilinq queries\n - Compile Excel configurations to Minilinq queries\n\nTo directly access the CommCare HQ REST API:\n\n```python\nfrom commcare_export.checkpoint import CheckpointManagerWithDetails\nfrom commcare_export.commcare_hq_client import CommCareHqClient, AUTH_MODE_APIKEY\nfrom commcare_export.commcare_minilinq import get_paginator, PaginationMode\n\nusername = 'some@username.com'\ndomain = 'your-awesome-domain'\nhq_host = 'https://commcarehq.org'\nAPI_KEY= 'your_secret_api_key'\n\napi_client = CommCareHqClient(hq_host, domain, username, API_KEY, AUTH_MODE_APIKEY)\ncase_paginator=get_paginator(resource='case', pagination_mode=PaginationMode.date_modified)\ncase_paginator.init()\ncheckpoint_manager=CheckpointManagerWithDetails(None, None, PaginationMode.date_modified)\n\ncases = api_client.iterate('case', case_paginator, checkpoint_manager=checkpoint_manager)\n\nfor case in cases:\n\tprint(case['case_id'])\n\n```\n\nTo issue a `minilinq` query against it, and then print out that query in a JSON serialization:\n\n```python\nimport json\nimport sys\nfrom commcare_export.minilinq import *\nfrom commcare_export.commcare_hq_client import CommCareHqClient\nfrom commcare_export.commcare_minilinq import CommCareHqEnv\nfrom commcare_export.env import BuiltInEnv, JsonPathEnv\nfrom commcare_export.writers import StreamingMarkdownTableWriter\n\napi_client = CommCareHqClient(\n    url=\"http://www.commcarehq.org\",\n    project='your_project',\n    username='your_username',\n    password='password',\n    version='0.5'\n)\n\nsource = Map(\n   source=Apply(\n       Reference(\"api_data\"),\n       Literal(\"form\"),\n       Literal({\"filter\": {\"term\": {\"app_id\": \"whatever\"}}})\n   ),\n   body=List([\n       Reference(\"received_on\"),\n       Reference(\"form.gender\"),\n   ])\n)\n\nquery = Emit(\n   'demo-table',\n   [\n       Literal('Received On'),\n       Literal('Gender')\n   ],\n   source\n)\n\nprint(json.dumps(query.to_jvalue(), indent=2))\n\nresults = query.eval(BuiltInEnv() | CommCareHqEnv(api_client) | JsonPathEnv())\n\nif len(list(env.emitted_tables())) \u003e 0:\n    with StreamingMarkdownTableWriter(sys.stdout) as writer:\n        for table in env.emitted_tables():\n            writer.write_table(table)\n```\n\nWhich will output JSON equivalent to this:\n\n```json\n{\n    \"Emit\": {\n        \"headings\": [\n            {\n                \"Lit\": \"Received On\"\n            },\n            {\n                \"Lit\": \"Gender\"\n            }\n        ],\n        \"source\": {\n            \"Map\": {\n                \"body\": {\n                    \"List\": [\n                        {\n                            \"Ref\": \"received_on\"\n                        },\n                        {\n                            \"Ref\": \"form.gender\"\n                        }\n                    ]\n                },\n                \"name\": null,\n                \"source\": {\n                    \"Apply\": {\n                        \"args\": [\n                            {\n                                \"Lit\": \"form\"\n                            },\n                            {\n                                \"Lit\": {\n                                    \"filter\": {\n                                        \"term\": {\n                                            \"app_id\": \"whatever\"\n                                        }\n                                    }\n                                }\n                            }\n                        ],\n                        \"fn\": {\n                            \"Ref\": \"api_data\"\n                        }\n                    }\n                }\n            }\n        },\n        \"table\": \"demo-table\"\n    }\n}\n```\n\n\nMiniLinq Reference\n------------------\n\nThe abstract syntax can be directly inspected in the `commcare_export.minilinq` module. Note that the choice between functions and primitives is deliberately chosen\nto expose the structure of the MiniLinq for possible optimization, and to restrict the overall language.\n\nHere is a description of the astract syntax and semantics\n\n| Python                        | JSON                                                | Which is evaluates to            |\n|-------------------------------|-----------------------------------------------------|----------------------------------|\n| `Literal(v)`                  | `{\"Lit\": v}`                                        | Just `v`                         |\n| `Reference(x)`                | `{\"Ref\": x}`                                        | Whatever `x` resolves to in the environment |\n| `List([a, b, c, ...])`        | `{\"List\": [a, b, c, ...}`                           | The list of what `a`, `b`, `c` evaluate to |\n| `Map(source, name, body)`     | `{\"Map\": {\"source\": ..., \"name\": ..., \"body\": ...}` | Evals `body` for each elem in `source`. If `name` is provided, the elem will be bound to it, otherwise it will replace the whole env. |\n| `FlatMap(source, name, body)` | `{\"FlatMap\": {\"source\" ... etc}}` | Flattens after mapping, like nested list comprehensions |\n| `Filter(source, name, body)`  | etc | |\n| `Bind(value, name, body)`     | etc | Binds the result of `value` to `name` when evaluating `body` |\n| `Emit(table, headings, rows)` | etc | Emits `table` with `headings` and `rows`. Note that `table` is a string, `headings` is a list of expressions, and `rows` is a list of lists of expressions. See explanation below for emitted output. |\n| `Apply(fn, args)` | etc | Evaluates `fn` to a function, and all of `args`, then applies the function to the args. |\n\nBuilt in functions like `api_data` and basic arithmetic and comparison are provided via the environment,\nreferred to be name using `Ref`, and utilized via `Apply`.\n\nList of builtin functions:\n\n| Function                       | Description                                                                    | Example Usage                    |\n|--------------------------------|--------------------------------------------------------------------------------|----------------------------------|\n| `+, -, *, //, /, \u003e, \u003c, \u003e=, \u003c=` | Standard Math                                                                  |                                  |\n| len                            | Length                                                                         |                                  |\n| bool                           | Bool                                                                           |                                  |\n| str2bool                       | Convert string to boolean. True values are 'true', 't', '1' (case insensitive) |                                  |\n| str2date                       | Convert string to date                                                         |                                  |\n| bool2int                       | Convert boolean to integer (0, 1)                                              |                                  |\n| str2num                        | Parse string as a number                                                       |                                  |\n| format-uuid                    | Parse a hex UUID, and format it into hyphen-separated groups                   |                                  |\n| substr                         | Returns substring indexed by [first arg, second arg), zero-indexed.            | substr(2, 5) of 'abcdef' = 'cde' |\n| selected-at                    | Returns the Nth word in a string. N is zero-indexed.                           | selected-at(3) - return 4th word |\n| selected                       | Returns True if the given word is in the value.                                | selected(fever)                  |\n| count-selected                 | Count the number of words                                                      |                                  |\n| json2str                       | Convert a JSON object to a string                                              |                                  |\n| template                       | Render a string template (not robust)                                          | template({} on {}, state, date)  |\n| attachment_url                 | Convert an attachment name into it's download URL                              |                                  |\n| form_url                       | Output the URL to the form view on CommCare HQ                                 |                                  |\n| case_url                       | Output the URL to the case view on CommCare HQ                                 |                                  |\n| unique                         | Ouptut only unique values in a list                                            |                                  |\n\nOutput Formats\n--------------\n\nYour MiniLinq may define multiple tables with headings in addition to their body rows by using `Emit`\nexpressions, or may simply return the results of a single query.\n\nIf your MiniLinq does not contain any `Emit` expressions, then the results of the expression will be\nprinted to standard output as pretty-printed JSON.\n\nIf your MiniLinq _does_ contain `Emit` expressions, then there are many formats available, selected\nvia the `--output-format \u003cformat\u003e` option, and it can be directed to a file with the `--output \u003cfile\u003e` command-line option.\n\n - `csv`: Each table will be a CSV file within a Zip archive.\n - `xls`: Each table will be a sheet in an old-format Excel spreadsheet.\n - `xlsx`: Each table will be a sheet in a new-format Excel spreadsheet.\n - `json`: The tables will each be a member of a JSON dictionary, printed to standard output\n - `markdown`: The tables will be streamed to standard output in Markdown format (very handy for debugging your queries)\n - `sql`: All data will be idempotently \"upserted\" into the SQL database you specify, including creating the needed tables and columns.\n\n\nDependencies\n------------\n\nRequired dependencies will be automatically installed. Optional dependencies\nfor specific export formats can be installed as extras:\n\n```shell\n# To export \"xlsx\"\n$ uv pip install \"commcare-export[xlsx]\"\n\n# To export \"xls\"\n$ uv pip install \"commcare-export[xls]\"\n\n# To sync with a Postgres database\n$ uv pip install \"commcare-export[postgres]\"\n\n# To sync with a mysql database\n$ uv pip install \"commcare-export[mysql]\"\n\n# To sync with a database which uses odbc (e.g. mssql)\n$ uv pip install \"commcare-export[odbc]\"\n\n# To sync with another SQL database supported by SQLAlchemy\n$ uv pip install \"commcare-export[base_sql]\"\n# Then install the Python package for your database\n```\n\nContributing\n------------\n\n0\\. Sign up for GitHub, if you have not already, at https://github.com.\n\n1\\. Fork the repository at https://github.com/dimagi/commcare-export.\n\n2\\. Clone your fork, install into a virtualenv, and start a feature branch\n\n```shell\n$ git clone git@github.com:your-username/commcare-export.git\n$ cd commcare-export\n$ uv venv\n$ source .venv/bin/activate  # On Windows: .venv\\Scripts\\activate\n$ uv pip install -e \".[test]\"\n$ git checkout -b my-super-duper-feature\n```\n\n3\\. Make your edits.\n\n4\\. Make sure the tests pass. The best way to test for all versions is to sign up for https://travis-ci.org and turn on automatic continuous testing for your fork.\n\n```shell\n$ py.test\n=============== test session starts ===============\nplatform darwin -- Python 2.7.3 -- pytest-2.3.4\ncollected 17 items\n\ntests/test_commcare_minilinq.py .\ntests/test_excel_query.py ....\ntests/test_minilinq.py ........\ntests/test_repeatable_iterator.py .\ntests/test_writers.py ...\n\n============ 17 passed in 2.09 seconds ============\n```\n\n5\\. Type hints are used in the `env` and `minilinq` modules. Check that any changes in those modules adhere to those types:\n\n```shell\n$ mypy --install-types @mypy_typed_modules.txt\n```\n\n6\\. Push the feature branch up\n\n```shell\n$ git push -u origin my-super-duper-feature\n```\n\n7\\. Visit https://github.com/dimagi/commcare-export and submit a pull request.\n\n8\\. Accept our gratitude for contributing: Thanks!\n\nRelease process\n---------------\n\n1\\. Create a tag for the release\n\n```shell\n$ git tag -a \"X.YY.0\" -m \"Release X.YY.0\"\n$ git push --tags\n```\n\n2\\. Create the distribution\n\n```shell\n$ uv build\n```\n\nEnsure that the archives in `dist/` have the correct version number (matching the tag name).\n\n3\\. Upload to pypi\n\n```shell\n$ uv publish\n```\n\n4\\. Verify upload\n\nhttps://pypi.python.org/pypi/commcare-export\n\n5\\. Create a release on github\n\nhttps://github.com/dimagi/commcare-export/releases\n\nOnce the release is published a GitHub workflow is kicked off that compiles executables of the DET compatible with\nLinux and Windows machines, adding it to the release as assets.\n\n[For Linux-based users] If you decide to download and use the executable file, please make sure the file has the executable permission enabled,\nafter which it can be invoked like any other executable though the command line.\n\n\nTesting and Test Databases\n--------------------------\n\nThe following command will run the entire test suite (requires DB environment variables to be set as per below):\n\n```shell\n$ py.test\n```\n\nTo run an individual test class or method you can run, e.g.:\n\n```shell\n$ py.test -k \"TestExcelQuery\"\n$ py.test -k \"test_get_queries_from_excel\"\n```\n\nTo exclude the database tests you can run:\n\n```shell\n$ py.test -m \"not dbtest\"\n```\n\nWhen running database tests, supported databases are PostgreSQL, MySQL, MSSQL.\n\nTo run tests against selected databases can be done using test marks as follows:\n```shell\n$ py.test -m [postgres,mysql,mssql]\n```\n\nUse Docker and docker-compose to start database services for tests:\n\n1. Start the services:\n   ```shell\n   docker-compose up -d\n   ```\n\n2. Wait for services to be healthy:\n   ```shell\n   docker-compose ps\n   ```\n\n3. Run your tests. The default environment variables in\n   `tests/conftest.py` work automatically:\n   - PostgreSQL: `postgresql://postgres@localhost/`\n   - MySQL: `mysql+pymysql://travis@/`\n   - MS SQL Server: `mssql+pyodbc://SA:Password-123@localhost/`\n\n   If needed, you can override with environment variables:\n   ```shell\n   export POSTGRES_URL='postgresql://postgres@localhost/'\n   export MYSQL_URL='mysql+pymysql://root@localhost/'\n   export MSSQL_URL='mssql+pyodbc://SA:Password-123@localhost/'\n   ```\n4. Stop the services when done:\n   ```shell\n   docker-compose down\n   ```\n   To also remove the data volumes:\n   ```shell\n   docker-compose down -v\n   ```\n\n\u003e [!NOTE]\n\u003e For MS SQL Server tests, you'll need the ODBC Driver for SQL Server\n\u003e installed on your host system for the `pyodbc` connection to work.\n\nFrom [learn.microsoft.com](https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server)\n([source](https://github.com/MicrosoftDocs/sql-docs/blob/live/docs/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server.md))\n\n#### Debian/Ubuntu\n\n```shell\n# Download the package to configure the Microsoft repo\ncurl -sSL -O https://packages.microsoft.com/config/debian/$(grep VERSION_ID /etc/os-release | cut -d '\"' -f 2 | cut -d '.' -f 1)/packages-microsoft-prod.deb\n# Install the package\nsudo dpkg -i packages-microsoft-prod.deb\n# Delete the file\nrm packages-microsoft-prod.deb\n\nsudo apt-get update\nsudo ACCEPT_EULA=Y apt-get install -y msodbcsql18\n\nodbcinst -q -d\n```\n\n#### Mac OS\n\n```shell\n/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)\"\nbrew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release\nbrew update\nHOMEBREW_ACCEPT_EULA=Y brew install msodbcsql18\n```\n\n\nIntegration Tests\n-----------------\nRunning the integration tests requires API credentials from CommCare HQ\nthat have access to the `corpora` domain. This user should only have\naccess to the corpora domain.\n\nThese need to be set as environment variables as follows:\n\n```shell\n$ export HQ_USERNAME=\u003cusername\u003e\n$ export HQ_API_KEY=\u003capikey\u003e\n```\n\nFor Travis builds these are included as encrypted vars in the travis\nconfig.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimagi%2Fcommcare-export","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdimagi%2Fcommcare-export","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimagi%2Fcommcare-export/lists"}