{"id":43352199,"url":"https://github.com/dimagi/commcare-analytics","last_synced_at":"2026-02-02T02:32:42.780Z","repository":{"id":39826316,"uuid":"472639201","full_name":"dimagi/commcare-analytics","owner":"dimagi","description":"CommCareHQ Superset Integration","archived":false,"fork":false,"pushed_at":"2025-05-15T22:22:43.000Z","size":20594,"stargazers_count":2,"open_issues_count":7,"forks_count":0,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-05-15T23:25:13.008Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dimagi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-03-22T06:22:55.000Z","updated_at":"2025-03-27T08:43:00.000Z","dependencies_parsed_at":"2024-05-31T12:53:10.026Z","dependency_job_id":"c6aff113-04f9-4758-bdfe-501e201e1f83","html_url":"https://github.com/dimagi/commcare-analytics","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dimagi/commcare-analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dimagi","download_url":"https://codeload.github.com/dimagi/commcare-analytics/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fcommcare-analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29001654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-02T01:32:03.847Z","status":"online","status_checked_at":"2026-02-02T02:00:07.448Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-02T02:32:42.180Z","updated_at":"2026-02-02T02:32:42.770Z","avatar_url":"https://github.com/dimagi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"CommCare HQ Superset Integration\n================================\n\nThis is a Python package that integrates Superset and CommCare HQ.\n\nLocal Development\n-----------------\n\n### Preparing CommCare HQ\n\nThe 'User configurable reports UI' feature flag must be enabled for the\ndomain in CommCare HQ, even if the data sources to be imported were\ncreated by Report Builder, not a UCR.\n\n\n### Setting up a dev environment\n\nWhile doing development on top of this integration, it's useful to\ninstall this via `pip -e` option so that any changes made get reflected\ndirectly without another `pip install`.\n\n- Set up a virtual environment.\n- Clone this repo and change into the directory of this repo.\n- Run `cp superset_config.example.py superset_config.py` and override\n  the config appropriately.\n- Run `pip install -e '.[dev]'`\n\n### CommCare HQ OAuth Integration\n\n- Create an OAuth application on CommCare HQ using Django Admin at the URL\n  `\u003chq_host\u003e/admin/oauth2_provider/application/` with the following settings:\n\n  - Redirect URIs: `\u003csuperset_host\u003e/oauth-authorized/commcare`\n\n  - Leave \"Post logout redirect URIs\" empty.\n\n  - Client type: Confidential\n\n  - Authorization grant type: Authorization code\n\n  - Give your OAuth application a name that users would recognize,\n    like \"CommCare Analytics\" or \"HQ Superset\". This name will appear\n    in CommCare HQ's dialog box requesting authorization from the\n    user.\n\n  - Leave \"Skip authorization\" unchecked\n\n  - Algorithm: No OIDC support\n\n- Update `OAUTH_PROVIDERS` setting in `superset_config.py` with OAuth\n  client credentials obtained from HQ.\n\n\n### Initialize Superset\n\nRead through the initialization instructions at\nhttps://superset.apache.org/docs/installation/installing-superset-from-scratch/#installing-and-initializing-superset.\n\nCreate a database for Superset, and a database for storing data from\nCommCare HQ. Adapt the username and database names to suit your\nenvironment.\n```bash\n$ createdb -h localhost -p 5432 -U postgres superset\n$ createdb -h localhost -p 5432 -U postgres superset_hq_data\n```\n\nSet the following environment variables:\n```bash\n$ export FLASK_APP=superset\n$ export SUPERSET_CONFIG_PATH=/path/to/superset_config.py\n```\n\nSet this environment variable to allow OAuth 2.0 authentication with\nCommCare HQ over insecure HTTP. (DO NOT USE THIS IN PRODUCTION.)\n```bash\n$ export AUTHLIB_INSECURE_TRANSPORT=1\n```\n\nInitialize the databases. Create an administrator. Create default roles\nand permissions:\n```bash\n$ superset db upgrade\n$ superset db upgrade --directory hq_superset/migrations/\n$ superset fab create-admin\n$ superset load_examples  # (Optional)\n$ superset init\n```\nYou may skip `superset load_examples`, although they could be useful.\n\nYou should now be able to run superset using the `superset run` command:\n```bash\n$ superset run -p 8088 --with-threads --reload --debugger\n```\n\nYou can now log in as a CommCare HQ web user.\n\nIn order for CommCare HQ to sync data source changes, you will need to\nallow OAuth 2.0 authentication over insecure HTTP. (DO NOT USE THIS IN\nPRODUCTION.) Set this environment variable in your CommCare HQ Django\nserver. (Yes, it's \"OAUTHLIB\" this time, not \"AUTHLIB\" as before.)\n```bash\n$ export OAUTHLIB_INSECURE_TRANSPORT=1\n```\n\n\n### Logging in as a local admin user\n\nThere might be situations where you need to log into Superset as a local\nadmin user, for example, to add a database connection. To enable local\nuser authentication, in `superset_config.py`, set\n`AUTH_TYPE = AUTH_DB`.\n\nDoing this will prevent CommCare HQ users from logging in, so it should\nonly be done in production environments when CommCare Analytics is not\nin use.\n\nTo return to allowing CommCare HQ users to log in, set it back to\n`AUTH_TYPE = AUTH_OAUTH`.\n\n\n### Importing UCRs using Redis and Celery\n\nCelery is used to import UCRs that are larger than\n`hq_superset.views.ASYNC_DATASOURCE_IMPORT_LIMIT_IN_BYTES`. If you need\nto import UCRs larger than this, you need to run Celery to import them.\nHere is how celery can be run locally.\n\n- Install and run Redis\n- Add Redis and Celery config sections from\n  `superset_config.example.py` to your local `superset_config.py`.\n- Run\n  `celery --app=superset.tasks.celery_app:app worker --pool=prefork -O fair -c 4`\n  in the Superset virtualenv.\n\n\n### Overwriting templates\nSuperset provides a way to update HTML templates by adding a file called\n`tail_js_custom_extra.html`.\nThis file can be used to insert HTML or script for all pages.\nThis isn't documented in superset but can be seen in the superset's \n[basic template](https://github.com/apache/superset/blob/f453d5d7e75cfd403b5552d6719b8ebc1f121d9e/superset/templates/superset/basic.html#L131).\n\n\n### Testing\n\nInstall requirements for running tests:\n\n    $ pip install -e '.[test]'\n    $ pytest\n\nThe test runner can only run tests that do not import from Superset. The\ncode you want to test will need to be in a module whose dependencies\ndon't include Superset.\n\n### Testing on staging\nIn order to test your feature branch on staging you need to\n1. Check out to `master` branch (make sure it's up to date)\n2. Add your feature branch to the `branches` section in `scripts/staging.yml` file\n3. Make a PR to `master`\n4. Once merged, run `scripts/rebuildstaging` (this will rebuild the staging branch on your machine)\n5. Push the new `staging` branch to remote\n6. Deploy `staging` (see Deploying Staging below)\n\n#### Deploying staging\n1. SSH to `commcare-analytics-staging.dimagi.com`\n2. Virtual environment should already be activated\n3. Uninstall current superset version\n\n```shell\n$ pip uninstall commcare-analytics\n```\n\n4. Install new superset version\n\n```shell\n$ pip install git+https://github.com/dimagi/commcare-analytics.git@staging\n```\n\n5. Restart superset \u0026 celery as needed\n\n```shell\n$ sudo supervisorctl restart superset\n$ sudo supervisorctl restart commcare_analytics_celery_default\n```\n\n\n\n### Creating a migration\n\nYou will need to create an Alembic migration for any new SQLAlchemy\nmodels that you add. The Superset CLI should allow you to do this:\n\n```shell\n$ superset db revision --autogenerate -m \"Add table for Foo model\"\n```\n\nHowever, problems with this approach have occurred in the past. You\nmight have more success by using Alembic directly. You will need to\nmodify the configuration a little to do this:\n\n1. Copy the \"HQ_DATA\" database URI from `superset_config.py`.\n\n2. Paste it as the value of `sqlalchemy.url` in\n   `hq_superset/migrations/alembic.ini`.\n\n3. Edit `env.py` and comment out the following lines:\n   ```\n   hq_data_uri = current_app.config['SQLALCHEMY_BINDS'][HQ_DATA]\n   decoded_uri = urllib.parse.unquote(hq_data_uri)\n   config.set_main_option('sqlalchemy.url', decoded_uri)\n   ```\n\nThose changes will allow Alembic to connect to the \"HD Data\" database\nwithout the need to instantiate Superset's Flask app. You can now\nautogenerate your new table with:\n\n```shell\n$ cd hq_superset/migrations/\n$ alembic revision --autogenerate -m \"Add table for Foo model\"\n```\n\n\nUpgrading Superset\n------------------\n\n`dimagi-superset` is a requirement of this `commcare-analytics` package.\nIt is a fork of `apache-superset`, and adds important features to it,\nnecessary for `commcare-analytics`. For more information about how to\nupgrade `dimagi-superset`, see\n[Dimagi Superset Fork](apache-superset.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimagi%2Fcommcare-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdimagi%2Fcommcare-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimagi%2Fcommcare-analytics/lists"}