{"id":35060882,"url":"https://github.com/openspending/os-conductor","last_synced_at":"2025-12-27T10:31:53.585Z","repository":{"id":52160589,"uuid":"44969473","full_name":"openspending/os-conductor","owner":"openspending","description":"OS Conductor is a set of integration web services of OpenSpending, responsible for identity, notification, and access control.","archived":false,"fork":false,"pushed_at":"2021-05-06T19:03:33.000Z","size":242,"stargazers_count":6,"open_issues_count":5,"forks_count":10,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-04-08T16:29:32.106Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openspending.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-10-26T13:17:33.000Z","updated_at":"2019-11-07T05:29:51.000Z","dependencies_parsed_at":"2022-08-28T15:41:13.767Z","dependency_job_id":null,"html_url":"https://github.com/openspending/os-conductor","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/openspending/os-conductor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fos-conductor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fos-conductor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fos-conductor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fos-conductor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openspending","download_url":"https://codeload.github.com/openspending/os-conductor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fos-conductor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28077501,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-27T02:00:05.897Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-27T10:31:52.393Z","updated_at":"2025-12-27T10:31:53.570Z","avatar_url":"https://github.com/openspending.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OS Conductor\n\n[![Gitter](https://img.shields.io/gitter/room/openspending/chat.svg)](https://gitter.im/openspending/chat)\n[![Travis](https://img.shields.io/travis/openspending/os-conductor.svg)](https://travis-ci.org/openspending/os-conductor)\n[![Coveralls](http://img.shields.io/coveralls/openspending/os-conductor/master.svg)](https://coveralls.io/r/openspending/os-conductor)\n[![Issues](https://img.shields.io/badge/issue-tracker-orange.svg)](https://github.com/openspending/openspending/issues)\n[![Docs](https://img.shields.io/badge/docs-latest-blue.svg)](http://docs.openspending.org/)\n\nWeb services for OpenSpending, responsible for:\n\n- user authentication, identity and access control\n- package upload, management, and status\n- package search of os-package-registry\n- upload to the S3 datastore\n\nos-conductor uses [Flask](http://flask.pocoo.org/) web framework.\n\n## Quick start for development\n\nClone the repo, install dependencies from pypi, and run the server. See the [docs](http://docs.openspending.org/en/latest/developers/conductor/) for more information.\n\nThe `os-types` node utility is used to perform fiscal modelling for the processed datapackage. To install, use npm:\n\n`$ npm install -g os-types`\n\n### Tests\n\nWith a running ElasticSearch server available on localhost:9200:\n\n```\n$ pip install tox  # install tox\n$ tox\n```\n\n### Environmental Variables\n\nOS Conductor requires environmental variables to be set, either in the local environment or in a `.env` file in the root directory.\n\n```ini\n# Required settings\n\n# Base URL for the application, e.g. 'http://localhost' or 'https://openspending.org'\nOS_BASE_URL=\n# Address for the postgres instance, e.g. postgresql://postgres@db/postgres\nOS_CONDUCTOR_ENGINE=\n# Address for ElasticSearch instance\nOS_ELASTICSEARCH_ADDRESS=\n\n# OAuth credentials. See the OAuth Credentials section below for details.\nOS_CONDUCTOR_SECRETS_0=\nOS_CONDUCTOR_SECRETS_1=\nOS_CONDUCTOR_SECRETS_2=\nOS_CONDUCTOR_SECRETS_3=\n\n# AWS S3 credentials\nOS_ACCESS_KEY_ID=\nOS_SECRET_ACCESS_KEY=\nOS_S3_HOSTNAME=\nOS_STORAGE_BUCKET_NAME=\n\n# Optional settings\n\n# Address for memcached server, e.g. http://cache:11211\nOS_CONDUCTOR_CACHE=\n\n# Address for the redis os-api-cache server, e.g. redis\nOS_API_CACHE=\n\n# If this env var exists, the entrypoint script will check whether ElasticSearch is healthy before allowing os-conductor to start.\nOS_CHECK_ES_HEALTHY=\n\n# If using the fake-s3 docker container for development, openspending/fakes3, add these settings:\nUSE_FAKE_S3=True\nOS_S3_PORT=4567\n```\n\n### OAuth Credentials\n\nOS Conductor needs credentials for authentication and authorization tasks. Credential values is set on `OS_CONDUCTOR_SECRETS_\u003cn\u003e` env vars. We provide a python script to help generate these values in `docker/secrets/generate-secrets/to_env_vars.py`.\n\n1. Create a [Google OAuth Credentials](https://console.developers.google.com/apis/credentials) and retain the Client ID and Secret Key\n2. Paste the Client ID and Secret Key values in to `google.key` and `google.secret.key` files respectively within the `generate-secrets` directory.\n3. Run the python script:\n\n```bash\n$ cd docker/secrets/generate-secrets\n$ python ./to_env_vars.py\n```\n\n4. Copy the generated env var key/values into your local environment or `.env` file in the root directory.\n\n\n### Admin tools\n\nVarious admin tools are available in the `/tools` directory. Some tools require dependencies to be installed from `/tools/requirements.txt`.\n\n#### `remove_package.py`\n\nRemove a named package (or packages) from the ElasticSearch index, and hence from searches and discovery within OpenSpending. Removing a package from the index won't remove it from the AWS datastore.\n\n\n## API Endpoints\n\nConductor current ships with the following blueprints, and their API endpoints.\n\n### Get authorized upload URL(s)\n`/datastore/authorize`\n\n**Method:** `POST`\n\n**Query Parameters:**\n\n - `jwt` - permission token (received from `/user/authorize`)\n\n**Headers:**\n\n - `Auth-Token` - permission token (can be used instead of the `jwt` query parameter)\n\n**Body:**\n\nJSON content with the following structure:\n```js\n{\n    \"metadata\": {\n        \"owner\": \"\u003cuser-id-of-uploader\u003e\",\n        \"name\": \"\u003cdata-set-unique-id\u003e\"\n    },\n    \"filedata\": {\n        \"\u003crelative-path-to-file-in-package-1\u003e\": {\n            \"length\": 1234, // length in bytes of data\n            \"md5\": \"\u003cmd5-hash-of-the-data\u003e\",\n            \"type\": \"\u003ccontent-type-of-the-data\u003e\",\n            \"name\": \"\u003cfile-name\u003e\"\n        },\n        \"\u003crelative-path-to-file-in-package-2\u003e\": {\n            \"length\": 4321,\n            \"md5\": \"\u003cmd5-hash-of-the-data\u003e\",\n            \"type\": \"\u003ccontent-type-of-the-data\u003e\",\n            \"name\": \"\u003cfile-name\u003e\"\n        }\n        ...\n    }\n}\n```\n\n`owner` must match the `userid` that is in the authentication token.\n\n### Get information regarding the datastore\n`/datastore/info`\n\n**Method:** `GET`\n\n**Query Parameters:**\n\n - `jwt` - permission token (received from `/user/authorize`)\n\n**Headers:**\n\n - `Auth-Token` - permission token (can be used instead of the `jwt` query parameter)\n\n**Returns:**\n\nJSON content with the following structure:\n```js\n{\n    \"prefixes\": [\n        \"https://datastore.openspending.org/123456789\",\n        ...\n    ]\n}\n```\n\n`prefixes` is the list of possible prefixes for an uploaded file for this user.\n\n\n### Load a datastore package into the OpenSpending DB\n`/package/upload`\n\n**Method:** `POST`\n\n**Query Parameters:**\n\n - `jwt` - permission token (received from `/user/authorize`)\n - `datapackage` - URL of the Fiscal DataPackage to load\n\n### Check on the status of uploading a package to the OpenSpending DB\n`/package/status`\n\n**Method:** `GET`\n\n**Query Parameters:**\n\n - `datapackage` - URL of the Fiscal DataPackage being loaded\n\n**Returns:**\n\n```json\n{\n    \"status\": \"\u003cstatus-code\u003e\",\n    \"progress\": 123,\n    \"error\": \"\u003cerror-message-if-applicable\u003e\"\n}\n```\n\n - `status-code`: one of the following:\n    - `queued`: Waiting in queue for an available processor\n    - `initializing`: Getting ready to load the package\n    - `loading-datapackage`: Reading the Fiscal Data Package\n    - `validating-datapackage`: Validating Data Package correctness\n    - `loading-resource`: Loading Resource data\n    - `deleting-table`: Clearing previous rows for this dataset from the database\n    - `creating-table`: Preparing space for rows in the database\n    - `loading-data-ready`: Starting to load rows to database\n    - `loading-data`: Loading data into the database\n    - `creating-babbage-model`: Converting the Data Package into an API model\n    - `saving-metadata`: Saving package metadata\n    - `done`: Done\n    - `fail`: Failed\n - `progress`: # of records loaded so far\n\nWil return an `HTTP 404` if the package is not being loaded right now.\n\n### Toggle or set a package's privacy setting\n`/package/publish`\n\n**Method:** `POST`\n\n**Query Parameters:**\n\n - `jwt` - permission token (received from `/user/authorize`)\n - `id` - Unique identifier of the datapackage to modify\n - `publish` - Publishing status, either:\n    - `true`: force publish,\n    - `false`: force private,\n    - `toggle`: toggle the state\n\n**Returns:**\n\n```js\n{\n    \"success\": true,\n    \"published\": true  // or false\n}\n```\n\n### Search for specific packages\n`/search/package`\n\n**Method:** `GET`\n\n**Query Parameters:**\n\n - `jwt` - authentication token (received from `/user/check`)\n - `q` - match-all query string\n - `package.title` - filter by package title\n - `package.author` - filter by package author\n - `package.description` - filter by package description\n - `package.regionCode` - filter by package region code\n - `package.countryCode` - filter by package region code\n - `package.packageCode` - filter by package region code\n - `size` - number of results to return\n\nAll values for all parameters (except `jwt`) should be passed as JSON values.\n\n**Returns:**\n\nAll packages that match the filter.\n\nIf authentication-token was provided, then private packages from the authenticated user will also be included.\nOtherwise, only public packages will be returned.\n\n```js\n[\n    {\n        \"id\": \"\u003cpackage-unique-id\u003e\",\n        \"model\": { ... }, // Babbage model\n        \"package\": { .... }, // Original FDP\n        \"origin_url\": \"\u003curl-to-the-datapackage.json\u003e\"\n    }\n]\n```\n\n### Check an authentication token's validity\n`/user/check`\n\n**Method:** `GET`\n\n**Query Parameters:**\n\n - `jwt` - authentication token\n - `next` - URL to redirect to when finished authentication\n\n**Returns:**\n\nIf authenticated:\n\n```js\n{\n    \"authenticated\": true,\n    \"profile\": {\n        \"id\": \"\u003cuser-id\u003e\",\n        \"name\": \"\u003cuser-name\u003e\",\n        \"email\": \"\u003cuser-email\u003e\",\n        \"avatar_url\": \"\u003curl-for-user's-profile-photo\u003e\",\n        \"idhash\": \"\u003cunique-id-of-the-user\u003e\",\n        \"username\": \"\u003cuser-selected-id\u003e\" // If user has a username\n    }\n}\n```\n\nIf not:\n\n```js\n{\n    \"authenticated\": false,\n    \"providers\": {\n        \"google\": {\n            \"url\": \"\u003curl-for-logging-in-with-the-Google-provider\u003e\"\n        }\n    }\n}\n```\n\nWhen the authentication flow is finished, the caller will be redirected to the `next` URL with an extra query parameter\n`jwt` which contains the authentication token. The caller should cache this token for further interactions with the API.\n\n### Get permission for a service\n`/user/authorize`\n\n**Method:** `GET`\n\n**Query Parameters:**\n\n - `jwt` - user token (received from `/user/check`)\n - `service` - the relevant service (e.g. `os.datastore`)\n\n**Returns:**\n\n```js\n{\n    \"token\": \"\u003ctoken-for-the-relevant-service\u003e\"\n    \"userid\": \"\u003cunique-id-of-the-user\u003e\",\n    \"permissions\": {\n        \"permission-x\": true,\n        \"permission-y\": false\n    },\n    \"service\": \"\u003crelevant-service\u003e\"\n}\n```\n\n__Note__: as of yet: the `permissions` property is still returned empty. Real permissions will be implemented soon.\n\n### Change the username\n`/user/update`\n\n**Method:** `POST`\n\n**Query Parameters:**\n\n - `jwt` - authentication token (received from `/user/check`)\n - `username` - A new username for the user profile (this action is only allowed once)\n\n**Returns:**\n\n```js\n{\n    \"success\": true,\n    \"error\": \"\u003cerror-message-if-applicable\u003e\"\n}\n```\n\n__Note__: trying to update other user profile fields like `email` will fail silently and return\n\n```js\n{\n    \"success\": true\n}\n```\n\n### Receive authorization public key\n`/user/public-key`\n\n**Method:** `GET`\n\n**Returns:**\n\nThe conductor's public key in PEM format.\n\nCan be used by services to validate that the permission token is authentic.\n\n### Read authentication JS library\n`/user/lib`\n\n**Method:** `GET`\n\n**Returns:**\n\nAuthentication Javascript library with Angular 1.x binding.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenspending%2Fos-conductor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenspending%2Fos-conductor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenspending%2Fos-conductor/lists"}