{"id":18597671,"url":"https://github.com/lealre/crawler-to-dash","last_synced_at":"2026-04-19T04:34:31.291Z","repository":{"id":257647934,"uuid":"833297953","full_name":"lealre/crawler-to-dash","owner":"lealre","description":"A dashboard featuring data scraped from a real estate website.","archived":false,"fork":false,"pushed_at":"2024-11-12T18:28:39.000Z","size":127258,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-16T14:10:02.215Z","etag":null,"topics":["boto3","bs4","dash-plotly","httpx","mongodb","pandas","pymongo"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lealre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-24T18:46:27.000Z","updated_at":"2024-11-12T18:28:43.000Z","dependencies_parsed_at":"2025-05-16T14:20:07.669Z","dependency_job_id":null,"html_url":"https://github.com/lealre/crawler-to-dash","commit_stats":null,"previous_names":["lealre/crawler-to-dash"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lealre/crawler-to-dash","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fcrawler-to-dash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fcrawler-to-dash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fcrawler-to-dash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fcrawler-to-dash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lealre","download_url":"https://codeload.github.com/lealre/crawler-to-dash/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fcrawler-to-dash/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31995131,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boto3","bs4","dash-plotly","httpx","mongodb","pandas","pymongo"],"created_at":"2024-11-07T01:29:04.035Z","updated_at":"2026-04-19T04:34:31.267Z","avatar_url":"https://github.com/lealre.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Property Sales Dashboard Using Real Data Crawled from Real Estate Website\n\nThis repository consists of an implementation of a dashboard containing data scraped from a real estate website: [Imovirtual](https://www.imovirtual.com/), a Portuguese real estate website offering homes, apartments, and other properties for sale and rent. Using MongoDB as the database, it crawls raw data, cleans it, and makes it ready to be used in the dashboard.\n\nBoth the dashboard and the scripts to crawl the data were implemented using Python. The dashboard uses the [Dash](https://dash.plotly.com/) and [Dash Bootstrap Components](https://dash-bootstrap-components.opensource.faculty.ai/) frameworks. To scrape the data, it uses [Requests](https://requests.readthedocs.io/en/latest/), asynchronous requests with [HTTPX](https://www.python-httpx.org/), and [BeautifulSoup](https://beautiful-soup-4.readthedocs.io/en/latest/).\n\nBy setting the environmental variables, the script can store the scraped raw data in three different sources: MongoDB using [pymongo](https://pymongo.readthedocs.io/en/stable/#), an AWS S3 bucket as a JSON file (using [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)), and local storage as a JSON file.\n\nBelow is a quick demonstration of the dashboard.\n\n\u003cimg src=\"media/demo.gif\" style=\"width: 100%;\"\u003e\n\nIt's possible to run the dashboard using Docker Compose.\n\n## Table of Contents\n\n- [How it works](#how-it-works)\n  - [Data Ingestion](#data-ingestion)\n  - [Dashboard](#dashboard)\n- [How to run this project](#how-to-run-this-project)\n  - [Dash with Docker](#dash-with-docker)\n  - [Local Setup](#local-setup)\n  - [MongoDB local backup](#mongodb-local-backup)\n- [Further Improvements](#further-improvements)\n\n## How it works\n\nThe project is divided into two blocks that can work separately, each inside the `src` folder:\n\n- **Data Ingestion**: Responsible for crawling the data, consolidating it in the database while avoiding duplicates, and preparing the data for use in the dashboard.\n- **Dashboard**: Uses the cleaned and filtered data from the database.\n\n\u003cimg src=\"media/diag.png\" style=\"width: 100%;\"\u003e\n\n\n### Data Ingestion\n\nAll the code related to data ingestion is inside the `src/ingestion` folder. It creates three collections in MongoDB:\n\n- A raw collection for storing the crawled data.\n- A consolidated collection that maintains an historical record of the data.\n- A dashboard collection used by the dashboard.\n\nThe raw collection, which comes from the crawler, can store data in MongoDB, an AWS S3 bucket as a JSON file, or locally as a JSON file, depending on the settings in the `.env` file.\n\nThe consolidation process creates a new collection in MongoDB and removes duplicate values from the raw data. It compares the raw data with the consolidated collection, allowing only new advertisements to be inserted and updating advertisements that are no longer available on the website. It filters unique entries using the same ID retrieved from the website for each advertisement. The goal of consolidation is to maintain a historical record of advertisements, even if they are no longer available on the website.\n\nFor the data used in the dashboard, a new collection is created. This pipeline extracts data from one of the previous collections (raw or consolidated), filters and transforms it so that it is ready for use in the dashboard.\n\nFor this specific website, it was possible to use asynchronous requests. In the first request, pagination information is retrieved for our search. This allows us to make an initial request to obtain this information, construct a block of URLs for requests, and perform asynchronous requests. After the requests, the data is extracted.\n\n### Dashboard\n\nAll the data used in the dashboard is loaded directly from the MongoDB collection designed for it. The folder and file structures inside `src/dash` were designed to load the data from MongoDB only once, and the components import the data from the same source file.\n\nTo build the graphs and manipulate the data, it uses [Plotly](https://plotly.com/python/) and [Pandas](https://pandas.pydata.org/docs/index.html).\n\n\u003cimg src=\"media/dash1.png\" style=\"width: 100%;\"\u003e\n\u003cimg src=\"media/dash2.png\" style=\"width: 100%;\"\u003e\n\n## How to run this project\n\nThis section explains how to run the project.\n\nAll the steps here are intended for a `bash` terminal.\n\nThe project setup uses [`pyenv`](https://github.com/pyenv/pyenv) and [`poetry`](https://python-poetry.org/).\n\nAs mentioned before, this project operates in two blocks, and it is possible to run both of them independently. Using Docker Compose, you can run the dashboard locally connected to the database. The Docker Compose setup includes an entrypoint that populates the database with the JSON file located at [scripts/data/data.json](scripts/data/data.json) if the collection does not already exist in MongoDB.\n\n1 - Clone the repo locally:\n```bash\ngit https://github.com/lealre/crawler-to-dash.git\n```\n\n2 - Access the project directory:\n```bash\ncd crawler-to-dash\n```\n\nTo run this properly, it's necessary to create the `.env` variable file in the root of the project. An example can be found in [.env-example](.env-example). The default configuration to connect with MongoDB is:\n```\nMONGO_HOST='mongodb'\nMONGO_PORT=27017\nMONGO_DATABASE='db'\n```\n\n### Dash with Docker\n\n- [Docker](https://www.docker.com/)\n- [Install Docker Compose](https://docs.docker.com/compose/install/)\n\nAfter completing steps 1 and 2, and with the `.env` variable file configured:\n\nBuild the image:\n```bash\ndocker compose buid\n```\n\nBuild the container:\n```bash\ndokcer compose up\n```\n\nAccess the local host:\n```\nhttp://localhost:8051/\n```\n\n**NOTE:** It may be necessary to make the script `./entrypoint.sh` executable before building the container:\n```bash\nchmod +x entrypoint.sh\n```\n\n### Local Setup\n\nAfter completing steps 1 and 2, and with the `.env` variable file configured:\n\n3 - Set the Python version with `pyenv`:\n```bash\npyenv local 3.12.2\n```\n\n4 - Create the virtual environment:\n```bash\npoetry env use 3.12.2\n```\n\n5 - Activate the virtual environment:\n```bash\npoetry shell\n```\n\n6 - Install dependencies:\n```bash\npoetry install\n```\n\n7 - Run the data pipeline, from crawling to dashboard data:\n```bash\ntask crawl_to_dash\n```\n\nIt is also possible to run each step separately:\n\nTo run just the crawler:\n```bash\ntask crawl\n```\n\nBased on the `.env` variables, it will store the data in different possible sources.\n```\nUSE_STORAGE_LOCAL=False\nUSE_STORAGE_MONGO=False\nUSE_STORAGE_AWS_S3=False\n```\n\nTo run just the data consolidation:\n```bash\ntask consolidate\n```\n\nTo generate the dash data:\n```bash\ntask dash_data\n```\n\n### MongoDB local Backup\n\nThe script [mongo_backup.sh](mongo_backup.sh) dumps the database to local storage in a file format. It uses the paths and container name specified in `.env` file.\n\nThe script first dumps the database content to a file inside the container, and it then copies the dump file from the container to the local storage.\n\nMake the script executable:\n```bash\nchmod +x mongo_backup.sh\n```\n\nExecute the backup script:\n```bash\n./mongo_backup.sh\n```\n\nThe `.env` file should contain the following variables:\n```\nCONTAINER_NAME=\"container-name\"\nBACKUP_PATH=\"/path/in/container\"\nLOCAL_BACKUP_PATH=\"/local/path/to/export\"\n```\n\n## Further Improvements\n\n- Improve the dashboard's style, including font size, colors, and callback interactions.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flealre%2Fcrawler-to-dash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flealre%2Fcrawler-to-dash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flealre%2Fcrawler-to-dash/lists"}