{"id":23425417,"url":"https://github.com/jannepaakkari/openai-web-scraper","last_synced_at":"2026-04-29T20:34:51.576Z","repository":{"id":268111450,"uuid":"902402795","full_name":"jannepaakkari/OpenAI-Web-Scraper","owner":"jannepaakkari","description":".NET based web scraper that uses OpenAI to analyze/predict categories from the topics of a selected website","archived":false,"fork":false,"pushed_at":"2024-12-20T14:39:32.000Z","size":119,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-24T18:40:15.701Z","etag":null,"topics":["csharp","dotnet","openai","sqlite","webscraper"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jannepaakkari.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-12T13:54:14.000Z","updated_at":"2024-12-20T14:39:35.000Z","dependencies_parsed_at":"2025-04-09T11:56:11.352Z","dependency_job_id":null,"html_url":"https://github.com/jannepaakkari/OpenAI-Web-Scraper","commit_stats":null,"previous_names":["jannepaakkari/openai-webscraper","jannepaakkari/openai-web-scraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jannepaakkari/OpenAI-Web-Scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jannepaakkari%2FOpenAI-Web-Scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jannepaakkari%2FOpenAI-Web-Scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jannepaakkari%2FOpenAI-Web-Scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jannepaakkari%2FOpenAI-Web-Scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jannepaakkari","download_url":"https://codeload.github.com/jannepaakkari/OpenAI-Web-Scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jannepaakkari%2FOpenAI-Web-Scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32443563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T20:22:27.477Z","status":"ssl_error","status_checked_at":"2026-04-29T20:22:26.507Z","response_time":110,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csharp","dotnet","openai","sqlite","webscraper"],"created_at":"2024-12-23T05:12:28.599Z","updated_at":"2026-04-29T20:34:51.559Z","avatar_url":"https://github.com/jannepaakkari.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n### Web scraper Implementation with OpenAI\n\nThis Web scraper application is designed to scrape selected websites and collect topics. It uses AI to analyze and predict the categorization of the collected topics. The analysis and scraped data are stored in an SQLite database. Additionally, the application provides an API that can be used to extract data for frontend applications and other purposes. This app is primarily designed to run locally and operates via the command line.\n\nPotential use cases include detecting ongoing discussions and identifying emerging trends. With minimal to no refactoring, you can also filter results to track specific topics or keywords, such as 'X' or 'Y', to see if they are being mentioned.\n\n## Technologies\n- C# (.NET 9.0)\n- Web scraping (HTML Agility Pack)\n- SQLite database\n- OpenAI\n- APIs\n- Github Actions\n- Unit tests\n\n## Installation\n\n1. Clone the repository:\n```bash\n   git clone https://github.com/jannepaakkari/Web-scraper.git\n   cd app/WebScraper\n```\n\nSet up .env file at the root of the WebScraper folder and add:\n```bash\n    OPENAI_API_KEY='your_key'\n    AllowedCorsOrigin=\"your_allowed_cors\" # optional add if needed with API\n```\n\n- Modify `appsettings.json` at the root of the WebScraper folder:\n```bash\n    \"Url\": Set url you want to scrape,\n    \"RunAPI\": Set true you want to run APIs, not neccessary for scraper itself,\n    \"ScrapingNodes\": Nodes you want to scrape, depends on site what you should add here, by default we scrape headers,\n```\n\n3. Restore dependencies:\n```bash\ndotnet restore\n```\n\n4. Build the project:\n```bash\ndotnet build\n```\n\n5. Run the application:\n```bash\ndotnet run\n```\n\n6. To apply the latest database migrations, use the following command:\n```bash\ndotnet ef database update\n```\n\n## Usage\n\n1. After you have successfully run the application (`dotnet run` in the `cd app/WebScraper` directory), you can view the content from `HeadersDatabase.db`.\n2. You can also use the API endpoint `/api/content` to view the content. Make sure `RunAPI` is set to `true` in `appsettings.json` before running the app.\n3. You can run tests at the root using:\n```bash\ndotnet test App.sln\n```\n\n## Screenshots\n![Small example of scraped content](screenshots/webscraper0.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjannepaakkari%2Fopenai-web-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjannepaakkari%2Fopenai-web-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjannepaakkari%2Fopenai-web-scraper/lists"}