{"id":15199447,"url":"https://github.com/miirzamiir/technews","last_synced_at":"2026-03-05T08:05:08.168Z","repository":{"id":252678477,"uuid":"840053655","full_name":"miirzamiir/TechNews","owner":"miirzamiir","description":"A technology NEWS API.","archived":false,"fork":false,"pushed_at":"2024-12-09T12:29:23.000Z","size":75,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-14T00:31:14.084Z","etag":null,"topics":["celery-flower","celery-redis","celerybeat","django","django-rest-framework","docker","python","selenium"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miirzamiir.png","metadata":{"files":{"readme":"README.md","changelog":"news/__init__.py","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-08T21:57:14.000Z","updated_at":"2024-09-03T12:21:13.000Z","dependencies_parsed_at":"2024-08-28T21:04:20.249Z","dependency_job_id":"d99171bf-58dd-43f8-86c5-33e95390d49c","html_url":"https://github.com/miirzamiir/TechNews","commit_stats":{"total_commits":30,"total_committers":1,"mean_commits":30.0,"dds":0.0,"last_synced_commit":"884637a75a89f6ba979f7765a96855f34e679a78"},"previous_names":["miirzamiir/technews"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miirzamiir%2FTechNews","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miirzamiir%2FTechNews/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miirzamiir%2FTechNews/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miirzamiir%2FTechNews/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miirzamiir","download_url":"https://codeload.github.com/miirzamiir/TechNews/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241678157,"owners_count":20001682,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery-flower","celery-redis","celerybeat","django","django-rest-framework","docker","python","selenium"],"created_at":"2024-09-28T02:01:08.012Z","updated_at":"2025-11-28T11:03:05.065Z","avatar_url":"https://github.com/miirzamiir.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TechNews\n\n**TechNews** is a RESTful API designed for the backend of a technology news website. This project was developed as part of a backend internship at **Roshan** and is divided into three main challenges.\n\n## Challenge 1: News Retrieval API\n\nThe first challenge focuses on creating a RESTful API for retrieving news articles. Each `news` record includes attributes such as `title`, `text`, `tags`, and `resource`. The following tasks were undertaken to complete this challenge:\n\n1. **Database Model Design \u0026 Implementation:**  \n   Designed and implemented the database structure to efficiently store and manage news articles.\n\n2. **API Implementation:**  \n   Developed API endpoints to create, read, update, and delete news records.\n\n3. **Filter by Tag Feature:**  \n   Added functionality to filter news articles based on specific tags.\n\n4. **Unit Testing:**  \n   Wrote unit tests to ensure the API's reliability and correctness.\n\n### Project Setup and Initialization\n\nThe project was initiated with the following steps:\n\n1. **Repository Creation:**  \n   A new Git repository was created, and initial files such as `.gitignore` and `README.md` were added.\n\n2. **Starting the Project:**  \n   The Django project was initialized with the following command:  \n   ```bash\n   django-admin startproject TechNews .\n\n3. **Adding Dependencies:**\\\n   A `requirements.txt` file was created to list all necessary packages for the project.\n\n4. **Database Configuration and App Creation:**\\\n   After configuring the database in `settings.py`, the `news` app was created using the command:\n   ```bash\n   python3 manage.py startapp news\n\n- These steps were needed in order to initiate the project. From this point, as outlined in the project document, the project will have a new branch called `challenge1` and all changes related to the first challenge will be done there.\n   \n### Updates in Branch `challenge1`\n1. **Design and Implementation of models**:\\\nThe `News` and `Tag` models were created to represent the news articles and their associated tags. The `Tag` model contains a single attribute, `tag_label`, representing the name of the `tag`. The `News` model includes the following attributes:\n   - `title`: The `title` of the `news`.\n   - `text`: The content of the `news`.\n   - `resource`: A URL field storing the original source of the `news`.\n   - `tags`: A `ManyToManyField` linking `news` to multiple `Tag` instances.\n\n2. **Implementation of `News` and `Tags` APIs**:\\\nTo implement these endpoints, serializers were created for each model. The `NewsSerializer` and `TagSerializer` were developed based on the defined models. These serializers were then utilized in the corresponding `viewsets`, `NewsViewSet` and `TagViewSet`, which extend `ReadOnlyModelViewSet`. This allows for efficient retrieval of news articles and tags through the API.\n\n3. **Filtering by `tag`** :\nThe `NewsViewSet` now supports filtering by `tag`, implemented using the `DjangoFilterBackend` from the `django-filter` package.\n\n4. **Writing unit tests**:\nUnit tests have been created for the `models`, `serializers`, and `views` within the `news` app. These tests are located in the `tests` directory. To run the tests, use the following command:\n   ```bash\n   python3 manage.py test\n   ```\n   And also to generate a coverage report for these tests, run:\n   ```bash\n   coverage run manage.py test \u0026\u0026 coverage html\n   ```\n   The coverage report will be available at `project_root/htmlcov/index.html` .\n\n*  Also **search functionality** and **pagination** were implemented in both the `News` and `Tag` views.\n\n- Since **Challenge 1** has been done properly, at this point branch `challenge1` will be merged into branch `master` .\n\n## Challenge 2: Collecting News Data\n\nThe second challenge focuses on gathering news data, requiring the development of a crawler to extract information from [Zoomit](https://zoomit.ir). For this task, a new branch named `challenge2` was created.\n\n### Updates in Branch `challenge2`\n\nTo create the crawler, it was necessary to design an appropriate architecture. **The first challenge** was deciding where to place the crawler within the project, and **the second challenge** was determining the best implementation approach.  \nGiven that the webpage is dynamic, I used **Selenium** for the crawling process. The crawler is located at `news/utils/zoomit_crawler.py`.\n\nTo simplify the execution of the crawler, a command was created for running the crawler. This command is implemented in `news/management/commands/crawl.py` using Django's `BaseCommand`. You can now easily run the crawler with the following command:\n\n```bash\npython3 manage.py crawl \u003cfrom_page\u003e \u003cto_page\u003e\n```\nThis command will crawl the archive of [Zoomit](https://zoomit.ir). The first argument( `from_page` ) specifies the starting page, and the second argument( `to_page` ) defines the ending page.\n\n## Pre-Challenge Modifications\n\nTo enhance the efficiency of Challenge 3, I implemented several modifications to the `News` model and the `ZoomitCrawler`. Specifically, I added a `date` field to the `News` model, which necessitated updates across various components, including:\n\n- **NewsSerializer**: Adjusted to accommodate the new `date` field.\n- **NewsModelViewSet**: Updated to ensure proper handling of the `date` attribute in API responses.\n- **NewsModelTest**: Revised to include tests for the new `date` functionality, ensuring data integrity.\n- **ZoomitCrawler**: Modified to utilize the `date` field when crawling news articles.\n\nFollowing these changes, I introduced the `crawl_unseen_news` method within the `ZoomitCrawler`. This method iterates over the Zoomit archive, crawling news articles until it encounters one that is already stored in the database. It includes a `stop` parameter, which specifies the page number at which the crawler will cease collecting news links if no new articles are detected. This method can be executed using the following command:\n\n```bash\npython3 manage.py crawl\n```\n\n## Challenge 3: Crawler Automation and Dockerizing the Project\n\nThe third challenge centers on automating the news crawler with **Celery** and **Celery Beat**, monitoring the automated process using **Celery Flower**, and Dockerizing the entire project.\n\n### Automation with Celery\n\nTo automate the crawler, a **Message Broker** was required. I opted for **Redis** due to its simplicity and robust performance. The steps taken include:\n\n- **Defining Celery Tasks**: I created tasks that encapsulate the crawling logic, allowing for asynchronous execution.\n- **Scheduling with Celery Beat**: I configured Celery Beat to schedule the crawling tasks at specified intervals, ensuring continuous operation.\n- **Monitoring with Celery Flower**: I integrated Celery Flower to provide a real-time dashboard for monitoring task execution and performance metrics.\n\n### Dockerizing the Project\n\nTo facilitate deployment and ensure consistency across environments, I Dockerized the project. The following steps were undertaken:\n\n- **Creating the Dockerfile**: I wrote a `Dockerfile` to define the application environment, including dependencies and configurations.\n- **Setting Up docker-compose.yaml**: This file was created to manage multi-container Docker applications, allowing for easy orchestration of services.\n- **Handling Database Preparation**: To address potential latency issues during database preparation, I utilized **wait-for-it**. This script ensures that the application waits for the database to be ready before proceeding.\n- **Custom Database Image**: Since the project requires a backup of the data, I created a custom `Dockerfile` for the database service rather than using the standard `postgres` image.\n- **Creating docker-entrypoint.sh**: This script was developed to automate the migration process before launching the Django application, ensuring that the database schema is up-to-date.\n\n---\n\nThank you for taking the time to read this document. Your feedback and insights are always welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiirzamiir%2Ftechnews","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiirzamiir%2Ftechnews","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiirzamiir%2Ftechnews/lists"}