{"id":24680576,"url":"https://github.com/fantaso/scrapy-tradefest","last_synced_at":"2026-05-07T00:34:41.949Z","repository":{"id":37060382,"uuid":"275686924","full_name":"Fantaso/scrapy-tradefest","owner":"Fantaso","description":"Project consists to allow a user to scrape the \"furniture\" section of tradefest.io (platform to find conventions \u0026 expos events) using Scrapy as a framework to extract, transform and store data.","archived":false,"fork":false,"pushed_at":"2022-11-22T06:15:40.000Z","size":66,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-21T19:45:22.551Z","etag":null,"topics":["docker","python3","scrapy","selenium"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Fantaso.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-28T23:35:37.000Z","updated_at":"2020-07-20T19:22:47.000Z","dependencies_parsed_at":"2023-01-20T12:33:33.281Z","dependency_job_id":null,"html_url":"https://github.com/Fantaso/scrapy-tradefest","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Fantaso/scrapy-tradefest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fantaso%2Fscrapy-tradefest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fantaso%2Fscrapy-tradefest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fantaso%2Fscrapy-tradefest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fantaso%2Fscrapy-tradefest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Fantaso","download_url":"https://codeload.github.com/Fantaso/scrapy-tradefest/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fantaso%2Fscrapy-tradefest/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267873923,"owners_count":24158733,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","python3","scrapy","selenium"],"created_at":"2025-01-26T14:13:54.618Z","updated_at":"2026-05-07T00:34:41.852Z","avatar_url":"https://github.com/Fantaso.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- logo --\u003e\n\u003ca href=\"https://www.fantaso.de\"\u003e\n\u003cimg src=\"/readme/fantaso.png\" align=\"right\" /\u003e\n\u003c/a\u003e\n\n\u003c!-- header --\u003e\n\u003ch1 style=\"text-align: left; margin-top:0px;\"\u003e\n  Convention \u0026 Expos Tradefest Scraper\n\u003c/h1\u003e\n\n\u003e Events scraper app with Scrapy and Selenium.\n\n\n\u003c!-- build --\u003e\n\u003c!-- [![Build Status][travis-image]][travis-link] --\u003e\n\n\nProject consists to allow a user to scrape the \"furniture\" section of\ntradefest.io (platform to find conventions \u0026 expos events) using Scrapy as a\nframework to extract, transform and store data.\n\n\u003cbr\u003e\u003cbr\u003e\n\n---\n## Index:\n- #### Usage: with Docker\n    1. Getting and Running the docker image\n    2. What data are we scraping and where is stored\n\n- #### Information:\n- #### Maintainer\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n---\n## Usage: with Docker ![container][docker]\n#### 1. Getting and Running the docker image\n\nimage is hosted in docker hub registry freely available [fantaso/scrapy_tradefest](https://hub.docker.com/repository/docker/fantaso/scrapy_tradefest)\n\nPulling image from Dockerhub\n    \n```sh\ndocker pull fantaso/scrapy_tradefest\n```\n\nor directly run it\n\n```sh\nmkdir -p output/logs \\\n\u0026\u0026 docker run --rm -v \"$(pwd)\"/output:/home/app/output -t fantaso/scrapy_tradefest\n```\n\n- Here `-v \"$(pwd)\"/output:/home/app/output` we are just binding a volume to synchronize \na folder `output/` in our computer and mapping it to a folder inside the docker\ncontainer `/home/app/output` where the data scraped will be stored. \n- Because of permissions and docker problems binding to sync the folder in the docker \nwith our local machine. **we need to create first the folder with `mkdir -p output/logs`\nto avoid problems running the docker**.\n\n**NOTE:** This is what we get when we want to run a container binding a volume\nto our local machine using a non root user to run the container.\n\n---\n#### 2.  What data are we scraping and where is stored\n\nWe want to store the data scraped in our machine. So, we are binding a folder inside\nthe docker container (`output/`) to a folder inside our local machine (PC were docker container runs).\n    \nOutput folder contains:\n    \n- `feeds` contains all the scraped data in different formats (csv, xml, json)\n- `logs`  contains the scraping log files\n- `media` contains the images we wanted to scraped as well as automatically generated\n          thumbnails from the images scraped in different sizes (small, medium).\n          \n**NOTE:** all data files generated for logs and feeds are named or formatted\nwith the current time when the docker image is run. `e.g: \"2020-06-28 23:47:18.csv\"```\n\nFields to be scraped from each event or expo:\n- `url`: url of the detailed event\n- `listed_name`: name of the event in the paginated list\n- `detailed_name`: name of the event in the detailed event page\n- `date`: date of the event\n- `city`: city where the event is taken place\n- `country`: country where the event is taken place\n- `venue`: location or place where the event is taken place\n- `duration`: time duration of the event\n- `final_grade`: rating of the event (client naming requirement!)\n- `total_reviews`: quantity of reviewers (related to the \"final_grade\")\n- `attendees`: quantity of people that attended the event\n- `exhibitors`: quantity of exhibitors that were part of the event\n- `hashtags`: tags of the event\n- `website`: official website of the event. \n- `description`: descriptions of the event\n- `image_urls`: url for the logo (image) of the of the event\n\n\n\u003cbr\u003e\n\n## Information:\n| Technology Stack |  |  |\n| :- | :-: | :- |\n| Python                    | ![back-end][python]                   | Back-End |\n| Scrapy                    | ![scraper framework][scrapy]          | Scraper Framework |\n| Selenium                  | ![browser automation][selenium]       | Browser Automation |\n| Docker                    | ![container][docker]                  | Container |\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n## Maintainer\nGet in touch -–\u003e [fantaso][fantaso]\n\n\n\n\u003c!-- Links --\u003e\n\u003c!-- Profiles --\u003e\n[github-profile]: https://github.com/fantaso/\n[linkedin-profile]: https://www.linkedin.com/\n[fantaso]: https://github.com/fantaso/\n\u003c!-- Extra --\u003e\n\n\u003c!-- Repos --\u003e\n[github-repo]: https://github.com/Fantaso/tradefest_scraper\n\n\u003c!-- Builds --\u003e\n[travis-link]: https://travis-ci.org/\n[travis-image]: https://travis-ci.org/\n\n\u003c!-- images --\u003e\n[python]: readme/python.png\n[scrapy]: readme/scrapy.png\n[selenium]: readme/selenium.png\n[docker]: readme/docker.png\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffantaso%2Fscrapy-tradefest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffantaso%2Fscrapy-tradefest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffantaso%2Fscrapy-tradefest/lists"}