Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sabujxi/python-scraper-and-data-analysts-admin-panel-in-django
A data scraper from texas govt site and a helping web app for managing, reviewing and editing the data
https://github.com/sabujxi/python-scraper-and-data-analysts-admin-panel-in-django
analyst data data-analysis data-entry data-scraper django django-application python python-scraper real-estate regex scraper texas
Last synced: about 1 month ago
JSON representation
A data scraper from texas govt site and a helping web app for managing, reviewing and editing the data
- Host: GitHub
- URL: https://github.com/sabujxi/python-scraper-and-data-analysts-admin-panel-in-django
- Owner: SabujXi
- License: gpl-3.0
- Created: 2017-06-22T12:21:08.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2022-12-07T23:59:17.000Z (almost 2 years ago)
- Last Synced: 2023-03-03T05:46:31.134Z (over 1 year ago)
- Topics: analyst, data, data-analysis, data-entry, data-scraper, django, django-application, python, python-scraper, real-estate, regex, scraper, texas
- Language: Python
- Size: 1.18 MB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Python Scraper and Data Analysts Admin Panel in Django
A data scraper from texas govt site and a helping web app for managing, reviewing and editing the data
A friend of mine taken data entry contract for for their farm. It was a lot of data.
You need to go the govt site of Texas, serch city one by one, go to the company/customer lists and copy those data to excel steet. It could take one mongth (two weeks if two person did this) for a single data entry operator or analyst to add those data and keep those data organized in excel steet.
In the middle of their work I told them to tell me all the problems they are facing and all the fetures/issues this site have.
I analyzed the source code of the site and also saw that there are some issue with sessions so it is not an easy task to make a scraper fot the site. But still after testing and tricking the site I made a very simple but perfect scraper. Mined all the data and then made an admin panel for the data entry operators and/or analysts.
It's no facncy work but it's a two pieces software that made their work very easy and some great amount of money with perfect data output and client satisfaction. I am uploading this project for those who may need such apps and may use my work as reference or helper to build theri project or may directly use this (with some variable value change) if their work is similar.
Note: I made this two pieces software for having the work done perfectly and deveoped it very quickly. So, I did not follow all the best practices of software development and good practices of python. Even regular expressions used here are perfect but not clean because it was made to have the work done perfectly not to make it a viral project.
You can contact me through http://sabuj.me or go to my linkedin account where my user name is SabujXi or also same user name on twitter or facebook.
Below are some screenshots - I haven't taken any screenshot during scraping, so the following screenshots are from the panel of data entry/analyst operator.
