{"id":24407625,"url":"https://github.com/is-leeroy-jenkins/koolaid","last_synced_at":"2026-04-29T02:07:33.842Z","repository":{"id":272367419,"uuid":"860202487","full_name":"is-leeroy-jenkins/Koolaid","owner":"is-leeroy-jenkins","description":"Koolaid is a WPF-based web crawler written in C-Sharp.","archived":false,"fork":false,"pushed_at":"2025-01-14T00:17:24.000Z","size":37087,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-14T01:21:08.177Z","etag":null,"topics":["web","web-crawler"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/is-leeroy-jenkins.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-20T02:30:28.000Z","updated_at":"2025-01-14T00:17:27.000Z","dependencies_parsed_at":"2025-01-14T01:31:29.581Z","dependency_job_id":null,"html_url":"https://github.com/is-leeroy-jenkins/Koolaid","commit_stats":null,"previous_names":["is-leeroy-jenkins/koolaid"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/is-leeroy-jenkins%2FKoolaid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/is-leeroy-jenkins%2FKoolaid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/is-leeroy-jenkins%2FKoolaid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/is-leeroy-jenkins%2FKoolaid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/is-leeroy-jenkins","download_url":"https://codeload.github.com/is-leeroy-jenkins/Koolaid/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243366721,"owners_count":20279411,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["web","web-crawler"],"created_at":"2025-01-20T05:11:59.954Z","updated_at":"2025-12-29T02:24:55.047Z","avatar_url":"https://github.com/is-leeroy-jenkins.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"#### Koolaid is a WPF-based web crawler written in C-Sharp.\n![](https://github.com/is-leeroy-jenkins/Koolaid/blob/master/Resources/Assets/GithubImages/repo-image.png)\n\n\n\n\n\n\n\n## ![](https://github.com/is-leeroy-jenkins/Koolaid/blob/main/Resources/Assets/GitHubImages/features.png)  Features\n\nWeb crawlers, also known as spiders, are most often used by search engines such as Google, Bing DuckDuckGo, \nto go through web pages and read the content on them. Web crawlers work by giving them an initial address to \nstart from and from there it will find new addresses to browse through. Web crawlers benefit a lot from concurrency,\nsince usually the highest amount of time is spent loading the page. By using some concurrent method such as using multiple \nthreads to load and read through the pages, performance of the crawler can be substantially increased..\n\n\n\n## ![](https://github.com/is-leeroy-jenkins/Koolaid/blob/main/Resources/Assets/GitHubImages/system_requirements.png)  System requirements\n\n- You need [VC++ 2019 Runtime](https://aka.ms/vs/17/release/vc_redist.x64.exe) 32-bit and 64-bit versions\n\n- You will need .NET 8.\n\n- You need to install the version of VC++ Runtime that Baby Browser needs. \n## Implementation\n\nI implemented the solution by creating a class named ``Crawler``.\nThe class packs inside itself \n* ``HttpClient`` to make requests\n* ``ConcurrentQueue`` to hold urls to visit next\n* ``HashSet`` for keeping track of all visited urls and urls found. \n* ``Semaphore`` for protecting variables from data races\n* ``List`` of tasks that do all the work\n* ``SiteMap`` (my own implementation) that holds recursive data structure to map out the web pages \n\nCrawler is started by calling Run method, which creates desired amount of Tasks.\n``CancellationToken`` is also passed in case the user wants to stop the execution. Each task will start by dequeing an \nurl from the queue. If the queue is empty, task will try dequeing item until it will timeout (5 sec)\nor it will dequeue url succesfully. Queue uses ConcurrentQueue class so it is thread safe.\nUrl is then used to create a http request. First status code is checked to be ok and then the header content is checked to be text/html.\nIf this is ok, page is loaded and passed to helper function which finds all urls by looking for strings in the html \nwhich start with ``href=\"`` and end with ``\"``.\n\nThese links are then added to a HashSet which is basically an unordered list. I will call it a list for now the keep things simple. \nElements in this list are then added to a list which contains all urls, if it is not already in it, and it will also get added to \nqueue if the url has not been yet visited and is not invalid file type such as .exe or .jpeg.\nAfter parsing the urls from the body of html, the current url is added to a list which holds all visited urls.\nAfter that, new url is dequeued and same process is repeated.\n\n\n## ![](https://github.com/is-leeroy-jenkins/Koolaid/blob/main/Resources/Assets/GitHubImages/documentation.png) Documentation\n\n- [User Guide](Resources/Github/Users.md) - How to get Koolaid\n- [Compilation Guide](Resources/Github/Compilation.md) - How to compile koolaid.\n- [Configuration Guide](Resources/Github/Configuration.md) - Configuration settings used in koolaid.\n- [Distribution Guide](Resources/Github/Distribution.md) - Create a new installer package\n\n\n## ![](https://github.com/is-leeroy-jenkins/Koolaid/blob/main/Resources/Assets/GitHubImages/csharp.png) Code\n\n- Built on NET 8\n- Supports AnyCPU as well as x86/x64 specific builds\n- [Crawler](https://github.com/is-leeroy-jenkins/Koolaid/blob/master/Data/Crawler.cs) - the web crawler class.\n- [UI](https://github.com/is-leeroy-jenkins/Koolaid/blob/master/UI) - main UI layer and associated controls and related functionality.\n- [Assets](https://github.com/is-leeroy-jenkins/Koolaid/tree/master/Resources/Assets) - resources used in koolaid.\n\n\n\n## [Report](./REPORT.md)\n\n\n![](./img/2.png)\n\n## [Example output](./Map.txt)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fis-leeroy-jenkins%2Fkoolaid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fis-leeroy-jenkins%2Fkoolaid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fis-leeroy-jenkins%2Fkoolaid/lists"}