{"id":20246633,"url":"https://github.com/civicdatalab/up-fiscal-data-backend","last_synced_at":"2026-06-04T02:31:23.572Z","repository":{"id":52799841,"uuid":"295728331","full_name":"CivicDataLab/up-fiscal-data-backend","owner":"CivicDataLab","description":null,"archived":false,"fork":false,"pushed_at":"2023-05-04T05:10:28.000Z","size":18,"stargazers_count":0,"open_issues_count":7,"forks_count":1,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-01-14T02:11:52.719Z","etag":null,"topics":["budget","data-mining","data-pipeline","open-data","selenium","spending"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CivicDataLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-15T12:56:29.000Z","updated_at":"2021-01-19T03:06:58.000Z","dependencies_parsed_at":"2024-11-14T09:31:57.886Z","dependency_job_id":"ac7464e2-4d81-4c01-8b8f-6a3998a96e23","html_url":"https://github.com/CivicDataLab/up-fiscal-data-backend","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CivicDataLab%2Fup-fiscal-data-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CivicDataLab%2Fup-fiscal-data-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CivicDataLab%2Fup-fiscal-data-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CivicDataLab%2Fup-fiscal-data-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CivicDataLab","download_url":"https://codeload.github.com/CivicDataLab/up-fiscal-data-backend/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241692621,"owners_count":20004262,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["budget","data-mining","data-pipeline","open-data","selenium","spending"],"created_at":"2024-11-14T09:31:32.290Z","updated_at":"2026-06-04T02:31:23.546Z","avatar_url":"https://github.com/CivicDataLab.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Uttar Pradesh Fiscal Data Backend\n\nA data scraping pipeline was setup to mine relevant data and sources from the Uttar Pradesh fiscal data portal, [Koshvani](http://koshvani.up.nic.in/).\n\n## Table of Contents\n\n[Platform](https://github.com/CivicDataLab/up-fiscal-data-backend#platform)\n\n[Tools](https://github.com/CivicDataLab/up-fiscal-data-backend#tools)\n\n[Setup](https://github.com/CivicDataLab/up-fiscal-data-backend#setup)\n\n[Challenges](https://github.com/CivicDataLab/up-fiscal-data-backend#challenges)\n\n[Contributions](https://github.com/CivicDataLab/up-fiscal-data-backend#contributions)\n\n[Repo Structure](https://github.com/CivicDataLab/up-fiscal-data-backend#repo-structure)\n\n## Platform\n\n**Platfrom Name** : Koshvani web -- A Gateway to Finance Activities in the State of Uttar Pradesh\n**Platform URL** : http://koshvani.up.nic.in/\n\nA more detailed analysis of the platform and in-scope data can be found [here](https://github.com/CivicDataLab/up-fiscal-data/blob/master/01-data-scoping/budget-portal.md).\n\n## Tools\n\nThough the data on the Koshvani platform is available in structured format to us and analyse, scraping it through traditional methods was turning out to be a challenge.\n\nKeeping in mind the platform structure and behaviour, a [decision](https://github.com/CivicDataLab/up-fiscal-data/blob/master/00-docs/decisions/003-selnium.md) was undertaken to select [Selenium](https://www.selenium.dev/) as the mode of data mining and storing. The Selenium framework allows to automate browser actions to extract in-scope datasets.\n\n## Setup\n\nInstructions for setting up the data pipeline.\n\n`\u003c\u003cTBD\u003e\u003e`\n\n## Challenges\n\nDuring the data scraping exercise, the following challenges were faced during mining of the data. The respective resolutions for those challeges are also documented here.\n\n| Challenge | Resolution |\n|---|---|\n|   |   |\n|   |   |\n|   |   |\n\n## Contributions\n\nYou can refer to the [contributing guidelines](https://github.com/CivicDataLab/up-fiscal-data-backend/blob/master/contribute/CONTRIBUTING.md) and understand how to contribute.\n\n## Repo Structure\n\n```\nroot\n└── contribute/\n    └── CODE-OF-CONDUCT.md\n    └── CONTRIBUTING.md\n└── LICENSE.md\n└── README.md\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcivicdatalab%2Fup-fiscal-data-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcivicdatalab%2Fup-fiscal-data-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcivicdatalab%2Fup-fiscal-data-backend/lists"}