{"id":21124844,"url":"https://github.com/bitartisan1/netdigger","last_synced_at":"2025-04-13T07:29:41.629Z","repository":{"id":248920673,"uuid":"830182725","full_name":"bitArtisan1/netDigger","owner":"bitArtisan1","description":"A .NET 8.0 C# WPF desktop application for web scraping data into structured databases with a modern UI, comprehensive logging and optimized high performance.","archived":false,"fork":false,"pushed_at":"2024-08-06T17:40:35.000Z","size":384,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-20T04:18:37.910Z","etag":null,"topics":["csharp","data","data-scraper","data-scraping","database","desktop","dotnet","internet","logging","scraper","ui","url","web-scraper","web-scrapers","web-scraping","web-scrapping"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bitArtisan1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-17T19:01:17.000Z","updated_at":"2024-08-12T13:10:07.000Z","dependencies_parsed_at":"2024-08-03T22:21:46.783Z","dependency_job_id":"d05e316e-d2b9-44db-b153-63c6817b2f70","html_url":"https://github.com/bitArtisan1/netDigger","commit_stats":null,"previous_names":["git-aamar/netdigger","bitartisan1/netdigger"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitArtisan1%2FnetDigger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitArtisan1%2FnetDigger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitArtisan1%2FnetDigger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitArtisan1%2FnetDigger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bitArtisan1","download_url":"https://codeload.github.com/bitArtisan1/netDigger/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234899846,"owners_count":18904139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csharp","data","data-scraper","data-scraping","database","desktop","dotnet","internet","logging","scraper","ui","url","web-scraper","web-scrapers","web-scraping","web-scrapping"],"created_at":"2024-11-20T04:18:15.954Z","updated_at":"2025-01-21T05:25:12.137Z","avatar_url":"https://github.com/bitArtisan1.png","language":"C#","funding_links":["https://www.buymeacoffee.com/bitArtisan","https://img.buymeacoffee.com/button-api/?text=Buy"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"right\"\u003e\n    \u003ca href=\"https://www.buymeacoffee.com/bitArtisan\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?style=flat-square\u0026logo=buy-me-a-coffee\u0026logoColor=black\" alt=\"Buy Me a Coffee Pls!\" /\u003e\n    \u003c/a\u003e\n\u003c/div\u003e\n\n\n# netDigger\n\nnetDigger is a web scraping application built using .NET 8.0 and C# WPF in Visual Studio. It collects various types of data and exports them into organized and structured databases. The application features a modern UI design with detailed and comprehensive logging.\n\u003ccenter\u003e\n\u003cdiv\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/513cba1d-d0a3-4461-9422-c67cd6528ec4\" alt=\"netDigger Logo.\" style=\"width: 1000px; height: 200px;\" /\u003e\n  \u003cp align=\"center\"\u003e\u003cstrong\u003e\u003cem\u003eA Powerful In-Depth Web Scraping Application.\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003c/div\u003e\n\u003c/center\u003e\n\n\n\n## Features\n\n- **Asynchronous Web Scraping**: Efficiently scrape web pages using asynchronous tasks with minimized latency and multi-threading for parallel processing.\n- **Data Collection**: Collects data such as PDFs, CSVs, DOCX, XLS, PPTX, TXT, Images, Videos, JSON, DBSQL, XML, HTML, PHP, JS, Archives, and Miscellaneous files.\n- **Comprehensive JSON, XML, and HTML Parsing**: Utilizes advanced parsing techniques to extract valuable information from JSON, XML, and HTML documents, including finding and processing hidden element data and meta data.\n- **Database Integration**: Organizes scraped URLs into SQLite databases based on their file types.\n- **Modern UI Design**: User-friendly WPF interface with rich text logging.\n- **Detailed Logging**: Comprehensive log messages with timestamps, log levels, and thread IDs.\n- **Export Options**: Export scraped data to database files, CSV, and TXT formats.\n- **Multi OS Support**: Compatible with Windows x64/x86/ARM, Linux and MacOS.\n\n## Technologies Used\n\n- **.NET 8.0**\n- **C#**\n- **WPF (Windows Presentation Foundation)**\n- **AngleSharp** for HTML parsing\n- **PuppeteerSharp**: A headless browser automation library for .NET.\n- **Newtonsoft.Json (Json.NET)**: A popular library for working with JSON in .NET.\n- **SQLite** for database management\n- **Concurrent Collections** for thread-safe operations\n\n## Prerequisites\n\n- .NET 8.0 Desktop Runtime or SDK Framework.\n- Visual Studio 2022. (In case you want to build it yourself).\n\n## Installation\n\n1. Clone the repository:\n   ```sh\n   git clone https://github.com/your-username/netDigger.git\n   cd netDigger\n   ```\n2. Open the solution file (netDigger.sln) in Visual Studio.\n\n3. Build the project:\n\nSelect `Build \u003e Build Solution`.\nRun the application:\n\nSelect `Debug \u003e Start Debugging` or press `F5`.\n\n## Contribution\n1. Fork the repository.\n2. Create a new branch (git checkout -b feature-branch).\n3. Commit your changes (git commit -m 'Add new feature').\n4. Push to the branch (git push origin feature-branch).\n5. Create a new Pull Request.\n\n## License\nThis project is licensed under the __GNU Affero General Public License v3.0__. See the __LICENSE__ file for more details.\n\n## Support Me\nIf you find RepoUp useful, consider supporting me by:\n\n- Starring the repository on GitHub\n- Sharing the tool with others\n- Providing feedback and suggestions\n- Follow me for more :)\n\n\u003ca href=\"https://www.buymeacoffee.com/bitArtisan\"\u003e\u003cimg src=\"https://img.buymeacoffee.com/button-api/?text=Buy me a coffee\u0026emoji=\u0026slug=bitArtisan\u0026button_colour=CBC3E3\u0026font_colour=000000\u0026font_family=Cookie\u0026outline_colour=000000\u0026coffee_colour=ffffff\" /\u003e\u003c/a\u003e\n    \n---\nFor any issues or feature requests, please open an issue on GitHub. Happy coding!\n\u003ccenter\u003e\n\u003cdiv style=\"text-align: center;\"\u003e\n  \u003cp align=\"center\"\u003e\n   \u003cimg src=\"https://github.com/user-attachments/assets/13333030-bd33-4cf6-a2b5-d1dacab12642\" alt=\"octodance\" style=\"margin-right: 10px;\"/\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\u003c/center\u003e\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitartisan1%2Fnetdigger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitartisan1%2Fnetdigger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitartisan1%2Fnetdigger/lists"}