{"id":20696635,"url":"https://github.com/maxvalue/kem-contacts","last_synced_at":"2025-07-28T05:34:17.511Z","repository":{"id":37061519,"uuid":"205864314","full_name":"MaxValue/KEM-Contacts","owner":"MaxValue","description":"Get all metadata about the Klima- und Energieregionen from https://www.klimaundenergiemodellregionen.at/modellregionen/liste-der-regionen/","archived":false,"fork":false,"pushed_at":"2024-05-14T22:16:06.000Z","size":92,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-03-11T02:47:58.143Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MaxValue.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-02T13:34:19.000Z","updated_at":"2023-03-04T18:32:06.000Z","dependencies_parsed_at":"2024-11-17T00:31:05.375Z","dependency_job_id":null,"html_url":"https://github.com/MaxValue/KEM-Contacts","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/MaxValue/KEM-Contacts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxValue%2FKEM-Contacts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxValue%2FKEM-Contacts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxValue%2FKEM-Contacts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxValue%2FKEM-Contacts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MaxValue","download_url":"https://codeload.github.com/MaxValue/KEM-Contacts/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxValue%2FKEM-Contacts/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267468540,"owners_count":24092334,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T00:14:40.258Z","updated_at":"2025-07-28T05:34:17.490Z","avatar_url":"https://github.com/MaxValue.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Crawler - KEM Contacts\n\nGet all metadata about the Klima- und Energieregionen from https://www.klimaundenergiemodellregionen.at/modellregionen/liste-der-regionen/\n\n## Contents\n* [Getting Started](#getting-started)\n*    [Prerequisites](#prerequisites)\n* [Deployment](#deployment)\n* [Built With](#built-with)\n* [Contributing](#contributing)\n*    [Roadmap](#roadmap)\n* [Versioning](#versioning)\n* [Authors](#authors)\n* [License](#license)\n* [Acknowledgments](#acknowledgments)\n*    [Project History](#project-history)\n\n## Getting Started\n\nThese instructions will get you a copy of the project up and running on your local machine for development. See deployment for notes on how to deploy the project on a live system.\n\n### Prerequisites\n\nWhat things you need to install the software and how to install them\n\nI recommend you to use the `setup_development.sh` script by running\n\n```\n./setup_development.sh\n```\n\nbut if you don't want to do that, here is the complete list of dependencies:\n\n* [Python 3.6.8](https://www.python.org/downloads/)\n* [Python 3 PIP 9.0.1](https://pip.pypa.io/)\n* [Python Venv 3.6.7-1](https://docs.python.org/3/library/venv.html)\n* [Scrapy 1.7.3](https://scrapy.org/)\n* [Sqlalchemy 1.3.7](https://www.sqlalchemy.org)\n* [Pyexcel 0.5.15](https://github.com/pyexcel/pyexcel/)\n* [Pyexcel-ods 0.5.6](https://github.com/pyexcel/pyexcel-ods)\n* [Pyexcel-xls 0.5.8](https://github.com/pyexcel/pyexcel-xls)\n* [Pyexcel-xlsxw 0.4.2](https://github.com/pyexcel/pyexcel-xlsxw)\n\n## Deployment\n\nActivate the environment\n\n```\nsource venv/bin/activate\n```\n\nChange to the scrapy project\n\n```\ncd kem\n```\n\nStart the crawler\n\n```\nscrapy crawl getcontacts\n```\n\nAfter the crawler finishes, you'll want to export the data:\n\n```\n./export.py results.db kem getcontacts 1 KEM-Contacts_YYYY-MM-DD\n```\n\nwhere 1 is the job id and YYYY-MM-DD should be replaced by the date on which you crawled the website.\n\nYou will see the job id at the beginning of the log `log.txt`: `Job ID is: XX`.\n\n## Built With\n\n* [Ubuntu 18.04.3 LTS](https://ubuntu.com/) - The operating system I use\n* [Sublime Text 3](https://www.sublimetext.com/) - The code editor I use\n* [Python 3.6.8](https://www.python.org/downloads/) - The programming language\n* [Python 3 PIP 9.0.1](https://pip.pypa.io/) - The package manager of the programming language\n* [Python Venv 3.6.7-1](https://docs.python.org/3/library/venv.html) - The project bundler of the programming language\n* [Scrapy 1.7.3](https://scrapy.org/) - The crawling framework\n* [Sqlalchemy 1.3.7](https://www.sqlalchemy.org) - The database interface library\n* [Pyexcel 0.5.15](https://github.com/pyexcel/pyexcel/) - For exporting to spreadsheet formats\n* [Pyexcel-ods 0.5.6](https://github.com/pyexcel/pyexcel-ods) - For exporting as ODS spreadsheet\n* [Pyexcel-xls 0.5.8](https://github.com/pyexcel/pyexcel-xls) - For exporting as XLS spreadsheet\n* [Pyexcel-xlsxw 0.4.2](https://github.com/pyexcel/pyexcel-xlsxw) - For exporting as XLSX spreadsheet\n\n## Contributing\n\nPlease open an issue if you want to help or have questions.\n\n### Roadmap\nThings I already plan to implement, but didn't have yet:\n- [ ] Change database scheme to be individual to crawler, make exporter therefore export specific table.\n\n## Versioning\n\nWe use [SemVer](http://semver.org/) for versioning. For the versions available, see the tags on this repository.\n\n## Authors\n\n* **Max Fuxjäger** - *Initial work* - [MaxValue](https://gitlab.com/MaxValue)\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE.txt](LICENSE.txt) file for details.\n\n### Project History\nThis project was created because I (Max) was asked to crawl this website.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxvalue%2Fkem-contacts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxvalue%2Fkem-contacts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxvalue%2Fkem-contacts/lists"}