{"id":35060970,"url":"https://github.com/openspending/dpkg-uk25k","last_synced_at":"2025-12-27T10:32:07.617Z","repository":{"id":2401398,"uuid":"3368518","full_name":"openspending/dpkg-uk25k","owner":"openspending","description":"UK departmental spending above 25000 GBP","archived":false,"fork":false,"pushed_at":"2019-11-01T16:10:18.000Z","size":177,"stargazers_count":4,"open_issues_count":1,"forks_count":7,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-08T16:29:30.540Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://data.gov.uk/openspending","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openspending.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-02-06T16:23:20.000Z","updated_at":"2018-02-09T12:56:50.000Z","dependencies_parsed_at":"2022-09-09T19:12:37.518Z","dependency_job_id":null,"html_url":"https://github.com/openspending/dpkg-uk25k","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/openspending/dpkg-uk25k","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fdpkg-uk25k","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fdpkg-uk25k/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fdpkg-uk25k/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fdpkg-uk25k/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openspending","download_url":"https://codeload.github.com/openspending/dpkg-uk25k/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openspending%2Fdpkg-uk25k/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28077505,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-27T02:00:05.897Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-27T10:32:06.719Z","updated_at":"2025-12-27T10:32:07.611Z","avatar_url":"https://github.com/openspending.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"UK departmental spending over GBP 25000\n=======================================\n\nThis repository contains scripts to acquire, clean and process the \nspending information released by the UK central government. \n\n\nETL stages\n----------\n\nThe scripts have several stages that need to be run in order:\n\n* ``build_index`` - will find all related metadata (tagged: \n  spend-transactions) on data.gov.uk\n* ``retrieve`` will then try to fetch all the files\n* ``extract`` will attempt to parse CSV/XLS/... and load it into a DB\n* ``combine`` column names are mapped and values are stored in one central table\n* ``cleanup`` parses dates and amounts. Reconciles entity names with Nomenklatura.\n* ``validate`` discards any transactions that don't have a parsed date and amount\n* ``report`` creates the report HTML\n* ``dump`` dumps all the database to spending.csv\n* ``transfer`` transfers spending.csv to an OpenSpending website\n\n\nSetup\n-----\n\nFirst clone this repo::\n\n  git clone https://github.com/okfn/dpkg-uk25k.git\n\nOn Ubuntu you need to install a number of packages::\n\n  sudo apt-get install python-dev postgresql libpq-dev libxml2-dev libxslt1-dev\n\nYou need to install the dependencies (best in a python virtual environment)::\n\n  virtualenv pyenv-dpkg-uk25k\n  pyenv-dpkg-uk25k/bin/pip install -r dpkg-uk25k/requirements.txt\n\nNote: If you encounter ``Error: pg_config executable not found.`` when installing psycopg2 or compile errors for lxml ``xslt-config: not found``, it means that some of the packages were not successfully installed on the previous step.\n\nThe default configuration is in ``default.ini``. If you want to change the configuration, copy it ``config.ini`` and edit it there. To save specifying the report directory when you run report.py, add it to the config. To do the transfer stage, then an openspending.apikey needs to be specified.\n\nNow create a postgres user for your unix user name::\n\n  sudo -u postgres createuser -D -R -S $USER\n\nNow check your postgres cluster is configured as UTF8 - it needs to be.\n\n  psql -l\n\nYou should see UTF8 in the Encoding column. If it is something else then you should either create a new template with the correct encoding, or recreate your whole cluster. Note this will delete any other postgres databases! To do this::\n\n  sudo -u postgres pg_dropcluster --stop 9.1 main\n  sudo -u postgres pg_createcluster --start 9.1 main --locale=en_US.UTF-8\n\nIf you have any issues with making the database UTF8, see: http://stackoverflow.com/questions/8394789/postgresql-9-1-installation-and-database-encoding#answer-8405325\n\nBefore you can run the scripts you need to prepare a database::\n\n  sudo -u postgres createdb uk25k\n\nAnd allow access to the database by editing /etc/postgresql/9.1/main/pg_hba.conf and adding this line::\n\n  local uk25k all trust\n\nNow restart postgres::\n\n  sudo service postgresql restart\n\n\nRunning the ETL scripts\n-----------------------\n\nRun the ETL scripts like this::\n\n  . pyenv-dpkg-uk25k/bin/activate\n  cd dpkg-uk25k\n  python build_index.py\n  python retrieve.py\n  python extract.py\n  python combine.py\n  python cleanup.py\n  python validate.py\n  python report.py\n\nOr do the whole lot together::\n\n  python build_index.py \u0026\u0026 python retrieve.py \u0026\u0026 python extract.py \u0026\u0026 python combine.py \u0026\u0026 python cleanup.py \u0026\u0026 python validate.py \u0026\u0026 python report.py\n\nWhen running the scripts multiple times, previously successful resources will not be processed again. Use --force to ensure they are. If you want to start completely from fresh, you can delete and recreate all tables like this::\n\n  sudo -u postgres dropdb uk25k\n  sudo -u postgres createdb uk25k\n\nTo limit the analysis to one publisher, specify the name as a parameter to build_index::\n\n  python build_index.py wales-office\n\nAll the later steps can be confined to a particular publisher, dataset or resource using options. Use '--help' on each command for more details.\n\nAnd finally, if you want to dump the resulting spend database to spending.csv and load it into OpenSpending then you can do::\n\n  python dump.py\n  python transfer.py\n\n\nOptimising\n----------\n\nWhen the spending table gets big (10 millions rows seen 2/2014) then queries in cleanup.py and validate.py get very slow unless you create an index:\n\nCREATE INDEX spending_resource_id_index ON spending (resource_id);\n\n\nPunted\n------\n\n* PDFs\n* Zip files containing a bunch of CSVs (potentially for a number of publishers)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenspending%2Fdpkg-uk25k","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenspending%2Fdpkg-uk25k","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenspending%2Fdpkg-uk25k/lists"}