{"id":21253339,"url":"https://github.com/francisjgarcia/autodeskargas_scraper_api","last_synced_at":"2025-07-11T01:33:23.351Z","repository":{"id":181640686,"uuid":"637534979","full_name":"francisjgarcia/autodeskargas_scraper_api","owner":"francisjgarcia","description":"API scraper for forums which are posted movies and series in Spanish. The scraper is written in Python and uses Scrapy to parse the HTML and extract the data. The API is written in Django and uses Django REST Framework to expose the data","archived":false,"fork":false,"pushed_at":"2023-12-13T11:03:02.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-12-13T12:27:48.185Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/francisjgarcia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-05-07T21:04:32.000Z","updated_at":"2023-12-13T12:27:56.571Z","dependencies_parsed_at":"2023-12-13T12:27:52.084Z","dependency_job_id":"8bcd8d3c-e952-402b-a5d4-277cb27c300a","html_url":"https://github.com/francisjgarcia/autodeskargas_scraper_api","commit_stats":null,"previous_names":["francisjgarcia/autodeskargas_scraper_api"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francisjgarcia%2Fautodeskargas_scraper_api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francisjgarcia%2Fautodeskargas_scraper_api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francisjgarcia%2Fautodeskargas_scraper_api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francisjgarcia%2Fautodeskargas_scraper_api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/francisjgarcia","download_url":"https://codeload.github.com/francisjgarcia/autodeskargas_scraper_api/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225669616,"owners_count":17505386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-21T03:51:18.194Z","updated_at":"2024-11-21T03:51:18.805Z","avatar_url":"https://github.com/francisjgarcia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AutoDesKargaS (Scraper API)\n\n## Table of contents\n\n- [AutoDesKargaS (Scraper API)](#autodeskargas-scraper-api)\n  - [Table of contents](#table-of-contents)\n  - [Description](#description)\n    - [Websites scraped](#websites-scraped)\n  - [Usage](#usage)\n    - [Set environment variables](#set-environment-variables)\n      - [Docker compose](#docker-compose)\n      - [Application](#application)\n    - [Build and run container](#build-and-run-container)\n  - [Using API](#using-api)\n    - [Movies](#movies)\n      - [Get movies](#get-movies)\n      - [Get specific movie quality](#get-specific-movie-quality)\n      - [Push thanks button](#push-thanks-button)\n      - [Get download urls](#get-download-urls)\n      - [Add new comment to post in forum](#add-new-comment-to-post-in-forum)\n    - [Series](#series)\n\n## Description\n\nAPI scraper for forums which are posted movies and series in Spanish. The scraper is written in Python and uses Scrapy to parse the HTML and extract the data. The API is written in Django and uses Django REST Framework to expose the data.\n\n### Websites scraped\n\n- [DescargasDD](https://descargasdd.org/)\n\n## Usage\n\n### Set environment variables\n\n#### Docker compose\n\nThe docker compose file uses the environment variables to build the container. The environment variables are defined in the file `.env` in the folder `deployments`.\n\n```bash\nCOMPOSE_PROJECT_NAME=autodeskargas # Name of the project\n# API movie environment variables\nSCRAPER_MOVIES_EXPOSE_PORT=autodeskargas_scraper_movies # Name of the image to build of movies API scraper\nSCRAPER_MOVIES_EXPOSE_PORT=8000 # Port exposed by the container\n# API series environment variables\nSCRAPER_SERIES_EXPOSE_PORT=autodeskargas_scraper_series # Name of the image to build of series API scraper\nSCRAPER_SERIES_EXPOSE_PORT=8001 # Port exposed by the container\n```\n\n#### Application\n\nThe application uses the environment variables. The environment variables are defined in the file `.env` in the folder `app`.\n\n```bash\n# Django settings\nSECRET_KEY=s3cr3t_k3y # Django secret key\nDEBUG=True # Django debug mode (True or False)\nALLOWED_HOSTS=* # Django allowed hosts (comma separated)\n\n# Scraping type\nSCRAPY_TYPE=movies # Set scraping type (movies or series)\n\n# Scraping DescargasDD\nBASE_URL=https://descargasdd.org # Base url of the forum\nWEB_USER=username # Username of the forum \nWEB_PASSWORD=p4ssw0rd # Password of the forum\n# Scraping always or only news\nSCRAPY_ALWAYS=True # Scrapy always all posts of forum (True or False)\n```\n\n### Build and run container\n\nThere are two ways to run the container.\nThe first one is to build the container and run it.\n\nTo build the container of movies API scraper, you can execute the following commands:\n```bash\ncd autodeskargas_scraper_api\n\ndocker build -f deployments/Dockerfile -t autodeskargas_scraper_movies .\n\ndocker run -d --name autodeskargas_scraper_movies --env-file app/.env -p 8000:8000 autodeskargas_scraper_movies \n```\n\nTo build the container of series API scraper, you can execute the following commands:\n```bash\ncd autodeskargas_scraper_api\n\ndocker build -f deployments/Dockerfile -t autodeskargas_scraper_series .\n\ndocker run -d --name autodeskargas_scraper_series --env-file app/.env -p 8001:8001 autodeskargas_scraper_series \n```\n\n\nThe second one is execute the docker-compose file.\n\nIf you want to run both containers for movies and series, you can execute the following commands:\n```bash\ncd autodeskargas_scraper_movies/deployments\n\ndocker-compose up -d\n```\n\nOr if you want to run only one container, you can execute the following commands:\n\nTo run only the container of movies API scraper:\n```bash\ncd autodeskargas_scraper_movies/deployments\n\ndocker-compose up -d autodeskargas_scraper_movies\n```\n\nTo run only the container of series API scraper:\n```bash\ncd autodeskargas_scraper_series/deployments\n\ndocker-compose up -d autodeskargas_scraper_series\n```\n\n## Using API\n\n### Movies\n\n#### Get movies\n\nTo get all movies, you can consume the endpoint `/api/v1/movies` with the method `GET`.\n\n```bash\ncurl -X GET http://localhost:8000/api/v1/movies\n```\n\n#### Get specific movie quality\n\nTo get a specific movie quality, you can consume the endpoint `/api/v1/movies` with the method `POST` and the body:\n\n```json\n{\n    \"quality\": \"\u003cquality\u003e\"\n}\n```\n\nWhere `\u003cquality\u003e` is the quality of the movie. The possible values are:\n\n- `micro1080`\n- `bdrip1080`\n- `h265`\n- `4k`\n- `animation_micro1080`\n- `animation_bdrip1080`\n- `animation_h265`\n- `animation_4k`\n- `anime_micro1080`\n- `anime_micro1080_vose`\n- `anime_bdrip1080`\n- `anime_bdrip1080_vose`\n- `anime_h265`\n- `anime_h265_vose`\n\n```bash\ncurl -X POST http://localhost:8000/api/v1/movies \\\n    -H 'Content-Type: application/json' \\\n    -d '{\"quality\": \"\u003cquality\u003e\"}'\n```\n\n#### Push thanks button\n\nBefore get the download urls of the movies, you need to push the thanks button. To do this, you can consume the endpoint `/api/v1/thanks` with the method `POST` and the body:\n\nFor example:\n```json\n{\n    \"link\": 12345\n}\n```\n\nWhere `\u003clink\u003e` is the url ID of the movie in the forum.\n\n```bash\ncurl -X POST http://localhost:8000/api/v1/thanks \\\n    -H 'Content-Type: application/json' \\\n    -d '{\"link\": \u003clink\u003e}'\n```\n\n#### Get download urls\n\nTo get the download urls of the movies, you can consume the endpoint `/api/v1/download_urls` with the method `POST` and the body:\n\nFor example:\n```json\n{\n    \"link\": 12345\n}\n```\n\nWhere `\u003clink\u003e` is the url ID of the movie in the forum.\n\n```bash\ncurl -X POST http://localhost:8000/api/v1/download_urls \\\n    -H 'Content-Type: application/json' \\\n    -d '{\"link\": \u003clink\u003e}'\n```\n\n#### Add new comment to post in forum\n\nTo add a new comment to a post in the forum, you can consume the endpoint `/api/v1/comments` with the method `POST` and the body:\n\nFor example:\n```json\n{\n    \"link\": 12345,\n    \"comment\": \"This is a comment\"\n}\n```\n\nWhere `\u003clink\u003e` is the url ID of the movie in the forum and `\u003ccomment\u003e` is the message to add to the post.\n\n```bash\ncurl -X POST http://localhost:8000/api/v1/comments \\\n    -H 'Content-Type: application/json' \\\n    -d '{\"link\": \u003clink\u003e, \"comment\": \"\u003ccomment\u003e\"}'\n```\n\n### Series\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrancisjgarcia%2Fautodeskargas_scraper_api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrancisjgarcia%2Fautodeskargas_scraper_api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrancisjgarcia%2Fautodeskargas_scraper_api/lists"}