{"id":23260082,"url":"https://github.com/sxryxnshs5/python-playwright-stockscraper-api","last_synced_at":"2026-04-17T04:32:35.641Z","repository":{"id":267053289,"uuid":"899977907","full_name":"SxryxnshS5/Python-Playwright-StockScraper-API","owner":"SxryxnshS5","description":"A Flask-based API that scrapes stock data using Playwright and BeautifulSoup, returning the results as downloadable CSV files. Note: The API is hosted on Render.com's free tier, which may lead to a loading time of 3-4 minutes.","archived":false,"fork":false,"pushed_at":"2025-04-01T08:17:25.000Z","size":4232,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-01T09:26:43.969Z","etag":null,"topics":["nifty","playwright","python","scraping","scraping-api","scraping-python","stock-data"],"latest_commit_sha":null,"homepage":"https://python-scraper-api-r1uj.onrender.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SxryxnshS5.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-07T14:35:57.000Z","updated_at":"2025-04-01T08:17:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"e44ff7f2-155c-4c63-9625-de57195acd27","html_url":"https://github.com/SxryxnshS5/Python-Playwright-StockScraper-API","commit_stats":null,"previous_names":["sxryxnshs5/python-playwright-stockscraper-api"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SxryxnshS5%2FPython-Playwright-StockScraper-API","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SxryxnshS5%2FPython-Playwright-StockScraper-API/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SxryxnshS5%2FPython-Playwright-StockScraper-API/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SxryxnshS5%2FPython-Playwright-StockScraper-API/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SxryxnshS5","download_url":"https://codeload.github.com/SxryxnshS5/Python-Playwright-StockScraper-API/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247439991,"owners_count":20939192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nifty","playwright","python","scraping","scraping-api","scraping-python","stock-data"],"created_at":"2024-12-19T13:14:07.691Z","updated_at":"2026-04-17T04:32:35.633Z","avatar_url":"https://github.com/SxryxnshS5.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"*Under maintenance*\n\n# Flask Stock Data Scraper API\n\n**Note:** The API offers two ways to retrieve stock data:\n\n1. **Live Scraped Data Endpoint**: Takes approximately **3-4 minutes** to return the CSV data due to the use of a free-tier deployment on Render and the time required for headless browsers to fully load and render the source website during scraping.\n2. **Automated Scraped Data Endpoint**: Returns data in just a **few seconds** by using pre-scraped data generated twice a day through GitHub Actions *(It may still take 30 seconds to 1 minute for Render to wake up from sleep mode.)*.\n\nThis project provides a RESTful API to scrape stock data and return it as a CSV file. The application is built with Flask, Playwright, and BeautifulSoup. It is deployed on [Render](https://render.com) and containerized using Docker.\n\n⚠️ **This project is for educational purposes only.**\n\n---\n\n## Base URL\n\n```\nhttps://python-scraper-api-r1uj.onrender.com\n```\n\n---\n\n## Available API Endpoints\n\n### 1. Live Scraped Data Endpoint\n\n**Endpoint**: `/api/\u003csymbol\u003e`\n\nThis endpoint performs live scraping of stock data from the source website when requested. Since it uses a headless browser via Playwright, it may take **3-4 minutes** to complete.\n\n- **Example Requests**:\n\n  ```\n  GET /api/NIFTY\n  GET /api/BANKNIFTY\n  GET /api/NIFTYIT\n  GET /api/NIFTY500\n  GET /api/NIFTYMIDCAP\n  GET /api/NIFTYAUTO\n  ```\n\n- **How it Works**:\n  - Initiates a live scrape of the stock data for the specified symbol.\n  - Returns a CSV file with the latest stock data once the scraping is complete.\n\n- **Use Case**:\n  - When you need real-time, up-to-date stock data.\n\n### 2. Automated Scraped Data Endpoint\n\n**Endpoint**: `/api-asd/\u003csymbol\u003e`\n\nThis endpoint serves pre-scraped stock data stored in CSV files. The data is automatically updated **twice daily** via GitHub Actions, making the response time just a **few seconds**.\n\n- **Example Requests**:\n\n  ```\n  GET /api-asd/NIFTY\n  GET /api-asd/BANKNIFTY\n  GET /api-asd/NIFTYIT\n  GET /api-asd/NIFTY500\n  GET /api-asd/NIFTYMIDCAP\n  GET /api-asd/NIFTYAUTO\n  ```\n\n- **How it Works**:\n  - Checks if a pre-scraped CSV file for the requested symbol exists in the `Scraped data` folder.\n  - Returns the CSV file if available. If not, returns an error message indicating no data is available.\n\n- **Use Case**:\n  - When you need quick access to recently updated stock data without waiting for live scraping.\n\n---\n\n## Supported Symbols\n\n**Note:** All supported symbols are listed in `symbols.txt`.\n\nSome commonly used symbols include:\n\n- **NIFTY**\n- **BANKNIFTY**\n- **NIFTYIT**\n- **NIFTY500**\n- **NIFTYMIDCAP**\n- **NIFTYAUTO**\n\n---\n\n## Example Response\n\nWhen you call the endpoint `/api/NIFTY` or `/api-asd/NIFTY`, the API returns a CSV file with the following format:\n\n```\nname,price,change\nRELIANCE,2490.50,2.45\nTCS,3245.65,-1.12\nINFY,1500.75,0.89\n```\n\n---\n\n## How GitHub Actions Automate Scraping\n\nThe project uses a **GitHub Actions workflow** to automate the scraping process:\n\n1. **Scheduled Execution**:\n   - Runs **twice daily** at **1 AM** and **1 PM IST** (7:30 PM and 7:30 AM UTC).\n   - Ensures the stock data remains fresh and up-to-date.\n\n2. **Workflow Steps**:\n   - **Checkout Repository**: Pulls the latest code.\n   - **Create Directory**: Ensures the `Scraped data` directory exists.\n   - **Scrape Data**: Triggers the scraping process for all supported symbols.\n   - **Save CSVs**: Downloads the scraped data and saves it as CSV files in the `Scraped data` folder.\n   - **Commit and Push**: Updates the repository with the new CSVs.\n\n3. **Manual Trigger**:\n   - The workflow can also be triggered manually via GitHub Actions.\n\n---\n\n## Running the Application Locally\n\n1. **Clone the repository**:\n\n   ```bash\n   git clone https://github.com/yourusername/stock-scraper-api.git\n   cd stock-scraper-api\n   ```\n\n2. **Build the Docker image**:\n\n   ```bash\n   docker build -t stock-scraper-api .\n   ```\n\n3. **Run the Docker container**:\n\n   ```bash\n   docker run -p 5000:5000 stock-scraper-api\n   ```\n\n4. **Access the API**:\n\n   Visit `http://localhost:5000/api/\u003csymbol\u003e` for live scraping or `http://localhost:5000/api-asd/\u003csymbol\u003e` for automated scraped data.\n\n---\n\n## Deployment\n\nThis application is deployed on [Render](https://render.com). The Docker image is used for deployment, ensuring consistent behavior across environments.\n\n---\n\n## Technologies Used\n\n- **Flask**: Web framework for creating API endpoints.\n- **Playwright**: For automating browser interactions to perform live scraping.\n- **BeautifulSoup**: For parsing HTML content and extracting stock data.\n- **Docker**: For containerizing the application.\n- **Render**: For deploying the API.\n- **GitHub Actions**: For automating the scraping process and maintaining up-to-date data.\n\n---\n\n## 🤝 Contributing\n\nWe’d love your help! Check out our [Contributing Guide](CONTRIBUTING.md) to get started. If you have questions or want to brainstorm ideas, visit the [Discussions](https://github.com/yourusername/repo/discussions) page.\n\n**Let’s build this together!** 🚀\n\n---\n\n## License\n\nMIT License. This project is for educational purposes only.\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsxryxnshs5%2Fpython-playwright-stockscraper-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsxryxnshs5%2Fpython-playwright-stockscraper-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsxryxnshs5%2Fpython-playwright-stockscraper-api/lists"}