{"id":22083792,"url":"https://github.com/sekharsunkara6/twitter-web-scraping","last_synced_at":"2026-05-07T05:37:15.823Z","repository":{"id":242912169,"uuid":"810917594","full_name":"SekharSunkara6/Twitter-Web-Scraping","owner":"SekharSunkara6","description":"Web scraping with Selenium and ProxyMesh, storing the data in  MongoDB, showing a list on a webpage. ","archived":false,"fork":false,"pushed_at":"2024-06-05T15:53:59.000Z","size":77,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-29T04:45:59.563Z","etag":null,"topics":["flask","gunicorn","mongodb","proxymesh","pymongo","python","selenium","webdriver"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SekharSunkara6.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-05T15:39:59.000Z","updated_at":"2024-06-05T15:57:07.000Z","dependencies_parsed_at":"2024-06-05T18:03:46.638Z","dependency_job_id":null,"html_url":"https://github.com/SekharSunkara6/Twitter-Web-Scraping","commit_stats":null,"previous_names":["sekharsunkara6/twitter-web-scraping"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SekharSunkara6%2FTwitter-Web-Scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SekharSunkara6%2FTwitter-Web-Scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SekharSunkara6%2FTwitter-Web-Scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SekharSunkara6%2FTwitter-Web-Scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SekharSunkara6","download_url":"https://codeload.github.com/SekharSunkara6/Twitter-Web-Scraping/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245171125,"owners_count":20572191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flask","gunicorn","mongodb","proxymesh","pymongo","python","selenium","webdriver"],"created_at":"2024-12-01T00:17:41.907Z","updated_at":"2026-05-07T05:37:10.769Z","avatar_url":"https://github.com/SekharSunkara6.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Twitter Web-Scraping App\r\nIn this repository it was created by using Selenium script that can read the Twitter home page (on your local computer) and fetch the top 5 trending topics under “What’s Happening” section from the homepage. \r\n\r\n## Technologies Used:\r\n1. Selenium: Selenium is a popular tool for automating web browsers. In this project, it is used for web scraping to interact with the Twitter website and extract trending topics.\r\n2. MongoDB: MongoDB is a NoSQL database program. It is used to store the scraped data (trending topics) in a database for persistence.\r\n3. HTML and Jinja: HTML is used to structure the web pages, while Jinja is a templating engine for Python that is used with Flask to generate dynamic HTML content. \r\n4. Python-Dotenv: Python-Dotenv is a Python library that manages environment variables stored in a .env file. It is used in this project to securely store sensitive information such as credentials.\r\n5. Flask: Flask is a micro web framework written in Python. It is used to build the web application and define routes for handling HTTP requests.\r\n\r\n## Features:\r\n1. Web Scraping: Utilizes Selenium and BeautifulSoup for web scraping to extract trending topics from Twitter's homepage.\r\n2. MongoDB Integration: Stores scraped data in a MongoDB database for persistence.\r\n3. Flask Web Interface: Provides a simple web interface for triggering the scraping process and displaying the scraped data.\r\n\r\n## Files Included: \r\n1. Web.py: Flask application script containing routes for triggering the scraping process and displaying scraped data.\r\n2. twitter_scrapping.py: Python script for scraping Twitter's homepage and storing trending topics in MongoDB.\r\n3. requirements.txt: File containing a list of Python dependencies required for running the project.\r\n4. index.html: HTML template file for rendering the web interface.\r\n5. .env: Environment variable file containing sensitive information like credentials.\r\n\r\n### Installation\r\nClone the repository to your local machine:\r\n\r\n```bash\r\ngit clone https://github.com/SekharSunkara6/Twitter-Web-Scraping.git\r\n```\r\nNavigate to the project directory:\r\n\r\n```bash\r\ncd twitter-scraping-app\r\n```\r\nInstall the required dependencies using pip:\r\n\r\n```bash\r\npip install -r requirements.txt\r\n```\r\nSet up your environment variables by creating a .env file in the project directory and adding the necessary credentials:\r\n\r\n```makefile\r\nproxymesh_user=your_proxymesh_username\r\nproxymesh_pass=your_proxymesh_password\r\nX_EMAIL=your_twitter_email\r\nX_PASS=your_twitter_password\r\nX_USER=your_twitter_username\r\nMONGODB_URI=your_mongodb_uri\r\n```\r\nUsage\r\nRun the Flask application:\r\n\r\n```bash\r\npython Web.py\r\n```\r\nOpen your web browser and go to http://localhost:5000 to access the web interface.\r\n\r\nClick the \"Scrape Data\" button to trigger the scraping process.\r\n\r\nOnce the scraping is complete, the trending topics will be displayed on the webpage.\r\n\r\n## Live Hosted\r\n![Project picture](./image/image.png)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsekharsunkara6%2Ftwitter-web-scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsekharsunkara6%2Ftwitter-web-scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsekharsunkara6%2Ftwitter-web-scraping/lists"}