{"id":20846932,"url":"https://github.com/amiraflak/digipulse","last_synced_at":"2026-04-06T08:31:29.196Z","repository":{"id":183795049,"uuid":"669315135","full_name":"AmirAflak/DigiPulse","owner":"AmirAflak","description":"Highly powerfull and reliable Digikala products monitoring data pipeline.","archived":false,"fork":false,"pushed_at":"2023-09-26T20:21:51.000Z","size":905,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-25T19:55:06.860Z","etag":null,"topics":["cassandra-database","celery-redis","digikala","fastapi","jupyter-notebook","nosql-database","pydantic","python","selenium-webdriver"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AmirAflak.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-21T22:41:51.000Z","updated_at":"2023-12-23T19:23:07.000Z","dependencies_parsed_at":"2025-03-12T11:42:47.604Z","dependency_job_id":"617ddc06-29d6-469d-985f-8cc60819f12a","html_url":"https://github.com/AmirAflak/DigiPulse","commit_stats":null,"previous_names":["amiraflak/digipulse"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AmirAflak/DigiPulse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmirAflak%2FDigiPulse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmirAflak%2FDigiPulse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmirAflak%2FDigiPulse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmirAflak%2FDigiPulse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AmirAflak","download_url":"https://codeload.github.com/AmirAflak/DigiPulse/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmirAflak%2FDigiPulse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31464604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T21:22:52.476Z","status":"online","status_checked_at":"2026-04-06T02:00:07.287Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra-database","celery-redis","digikala","fastapi","jupyter-notebook","nosql-database","pydantic","python","selenium-webdriver"],"created_at":"2024-11-18T02:18:30.565Z","updated_at":"2026-04-06T08:31:29.182Z","avatar_url":"https://github.com/AmirAflak.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DigiPulse\n[![MIT-License](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/AmirAflak/DigiPulse/blob/main/LICENSE)\n[![Python](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/downloads/)\n[![cassandra_driver](https://img.shields.io/badge/cassandra__driver-3.28.0-blue.svg)](https://pypi.org/project/cassandra-driver/)\n[![celery](https://img.shields.io/badge/celery-5.3.4-blue.svg)](https://pypi.org/project/celery/)\n[![fastapi](https://img.shields.io/badge/fastapi-0.103.1-blue.svg)](https://pypi.org/project/fastapi/)\n[![pydantic](https://img.shields.io/badge/pydantic-1.10.12-blue.svg)](https://pypi.org/project/pydantic/)\n[![selenium](https://img.shields.io/badge/selenium-4.13.0-blue.svg)](https://pypi.org/project/selenium/)\n\n## Description\nDigipulse is a Python project designed to crawl products from the [Digikala](https://www.digikala.com) and track their prices and other characteristics. It utilizes Selenium for crawling, Redis as a queuing system for scraping tasks, CassandraDB (DataStax Cloud) for storing JSON data, and FastAPI to create a REST API for CRUD operations with the database.\n\nThe project enables you to efficiently collect and manage product data from Digikala, automating the process of tracking prices and other properties over time.\n\n## Features\n* Crawl products from the Digikala website.\n* Track and monitor price changes and other characteristics of the products.\n* Queue scraping tasks on Redis for efficient distribution to Celery workers.\n* Store and manage JSON product data in CassandraDB on [DataStax Cloud](https://astra.datastax.com/).\n* Expose a REST API built with FastAPI for CRUD operations on the database.\n* Use Pydantic models for request and response validation in the FastAPI endpoints.\n\n## Installation \n1. Clone the repo\n   ```sh\n   git clone https://github.com/AmirAflak/DigiPulse.git\n   ```\n2. Navigate into the project directory:\n   ```sh\n   cd DigiPulse\n   ```\n3. Set up a Python virtual environment:\n   ```sh\n   python3 -m venv .venv\n   source .venv/bin/activate\n   ```\n4. Install the project dependencies:\n   ```sh\n   pip install -r requirements.txt\n   ```\n5. Configure the project:\n* Edit config.py and adjust the necessary settings, such as database connection details and API configuration.\n6. Run the application:\n   ```sh\n   uvicorn app.main:app --reload\n   ```\n   The API should now be accessible at http://localhost:8000.\n\n  \n## Usage\nOnce the application is running, you can use the exposed endpoints of the REST API to interact with the database and perform CRUD operations on the product data.\n\n### Screenshots\nThese screenshots from the DigiPulse portray the successful implementation of real-time price tracking through API integration and the utilization of Cassandra's robust database capabilities.\n\n* \u003cb\u003eREST API:\u003c/b\u003e Implementation of the API for real-time price tracking in the DigiPulse. Through this functionality, users can list various products and keep track of their prices dynamically. The API integration enables seamless communication between the DigiPulse platform and external systems or databases to gather up-to-date pricing information. This real-time price tracking feature enriches the user experience by providing accurate and timely data on product prices, allowing users to make informed decisions.\n\n\u003cbr /\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"[https://github.com/AmirAflak/DigiPulse\"\u003e\n    \u003cimg src=\"images/api.jpg\" alt=\"Logo\" width=\"600\" height=\"300\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003cbr /\u003e \n\n* \u003cb\u003eCQL Console:\u003c/b\u003e Captures a table in the Cassandra database within the DigiPulse project, as viewed through the DataStax CQL console. Specifically, the table shown reflects the structured data stored in the Cassandra database, which supports the efficient management and retrieval of product-related information. Cassandra's distributed architecture and scalability make it a suitable choice for the DigiPulse project, ensuring reliable and high-performance data storage and retrieval. By utilizing the power of Cassandra, the project gains the ability to handle large volumes of data while maintaining system responsiveness and resilience.\u003cbr /\u003e\n\u003cbr /\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"[https://github.com/AmirAflak/DigiPulse\"\u003e\n    \u003cimg src=\"images/db.jpg\" alt=\"Logo\" width=\"400\" height=\"400\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003c!-- CONTRIBUTING --\u003e\n## Contributing\n\nContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.\n\nIf you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag \"enhancement\".\nDon't forget to give the project a star! Thanks again!\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n## License\nThis project is licensed under the MIT License. See the [LICENSE](https://github.com/AmirAflak/DigiPulse/blob/main/LICENSE) file for details.\n\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famiraflak%2Fdigipulse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famiraflak%2Fdigipulse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famiraflak%2Fdigipulse/lists"}