{"id":18732503,"url":"https://github.com/gsscoder/pickall","last_synced_at":"2025-04-12T18:31:14.669Z","repository":{"id":142612439,"uuid":"225642071","full_name":"gsscoder/pickall","owner":"gsscoder","description":".NET agile and extensible web searching API","archived":false,"fork":false,"pushed_at":"2025-01-30T19:19:28.000Z","size":488,"stargazers_count":29,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-26T12:51:10.410Z","etag":null,"topics":["api","csharp","dotnet","dotnet-core","library","scraping","searching","web"],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gsscoder.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-03T14:38:12.000Z","updated_at":"2023-09-20T21:01:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"0807ebe8-89f2-4603-9477-5413d7f97a32","html_url":"https://github.com/gsscoder/pickall","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gsscoder%2Fpickall","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gsscoder%2Fpickall/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gsscoder%2Fpickall/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gsscoder%2Fpickall/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gsscoder","download_url":"https://codeload.github.com/gsscoder/pickall/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248613325,"owners_count":21133492,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","csharp","dotnet","dotnet-core","library","scraping","searching","web"],"created_at":"2024-11-07T15:06:27.497Z","updated_at":"2025-04-12T18:31:14.653Z","avatar_url":"https://github.com/gsscoder.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://dev.azure.com/gsscoder/pickall/_apis/build/status/gsscoder.pickall?branchName=master)](https://dev.azure.com/gsscoder/pickall/_build/latest?definitionId=2\u0026branchName=master)\n[![NuGet](https://img.shields.io/nuget/dt/pickall.svg)](https://nuget.org/packages/pickall)\n[![NuGet](https://img.shields.io/nuget/vpre/pickall.svg)](https://www.nuget.org/packages/pickall)\n[![Join the Gitter chat!](https://badges.gitter.im/gsscoder/pickall.svg)](https://gitter.im/pickallwebsearcher/community#)\n\n# PickAll\n\n![alt text](/assets/icon.png \"SharpX Logo\")\n\n.NET agile and extensible web searching API. Built with [AngleSharp](https://anglesharp.github.io/).\n\n## Philosophy\n\nPickAll is primarily designed to collect a limited amount of results (possibly the more relavant) from different sources and process these in a chain of steps. Results are essentially URLs and descriptions, but more data can be handled.\n\n## Documentation\n\nDocumentation is available in the project [wiki](https://github.com/gsscoder/pickall/wiki).\n\n## Targets\n\n- .NET Standard 2.0\n- .NET Core 3.1\n- .NET 5.0\n\n## Install via NuGet\n\n```sh\n$ dotnet add package PickAll --version 1.3.1\n  Determining projects to restore...\n  ...\n```\n\n## Build and sample\n\n```sh\n# clone the repository\n$ git clone https://github.com/gsscoder/pickall.git\n\n# build the package\n$ cd pickall/src/PickAll\n$ dotnet build -c release\n\n# execute sample\n$ cd pickall/samples/PickAll.Sample\n$ dotnet build -c release\n$ cd ../../artifacts/PickAll.Sample/Release/netcoreapp3.0/PickAll.Sample\n./PickAll.Sample \"Steve Jobs\" -e bing:duckduckgo\nSearching 'Steve Jobs' ...\n[0] Bing: \"Steve Jobs - Wikipedia\": \"https://it.wikipedia.org/wiki/Steve_Jobs\"\n[0] DuckDuckGo: \"Steve Jobs - Wikipedia\": \"https://en.wikipedia.org/wiki/Steve_Jobs\"\n[1] DuckDuckGo: \"Steve Jobs - Apple, Family \u0026 Death - Biography\": \"https://www.biography.com/business-figure/steve-jobs\"\n[2] Bing: \"CC-BY-SA licenza\": \"http://creativecommons.org/licenses/by-sa/3.0/\"\n[2] DuckDuckGo: \"Steve Jobs - IMDb\": \"https://www.imdb.com/name/nm0423418/\"\n[3] Bing: \"Biografia di Steve Jobs - Biografieonline\": \"https://biografieonline.it/biografia.htm?BioID=1560\u0026biografia=Steve+Jobs\"\n```\n\n## Test\n\n```sh\n# change to tests directory\n$ cd pickall/tests/PickAll.Specs\n\n# build with debug configuration\n$ dotnet build -c debug\n...\n\n# execute tests\n$ dotnet test\n...\n```\n\n## At a glance\n\n**CSharp:**\n```csharp\nusing PickAll;\n\nvar context = new SearchContext()\n    .WithEvents()\n    .With\u003cGoogle\u003e() // search on google.com\n    .With\u003cYahoo\u003e() // search on yahoo.com\n    .With\u003cUniqueness\u003e() // remove duplicates\n    .With\u003cOrder\u003e() // prioritize results\n    // match Levenshtein distance with maximum of 15\n    .With\u003cFuzzyMatch\u003e(new FuzzyMatchSettings { Text = \"mechanics\", MaximumDistance = 15 });\n    // repeat a search using more frequent words of previous results\n    .With\u003cImprove\u003e(new ImproveSettings { WordCount = 2, NoiseLength = 3 })\n    // scrape result pages and extract all text\n    .With\u003cTextify\u003e(new TextifySettings { IncludeTitle = true, NoiseLength = 3 });\n// attach events\ncontext.ResultCreated += (sender, e) =\u003e Console.WriteLine($\"Result created from {e.Result.Originator}\");\n// execute services (order of addition)\nvar results = await context.SearchAsync(\"quantum physics\");\n// do anything you need with LINQ\nvar scientific = results.Where(result =\u003e result.Url.Contains(\"wikipedia\"));\nforeach (var result in scientific) {\n    Console.WriteLine($\"{result.Url} {result.Description}\");\n}\n```\n\n**FSharp:**\n```fsharp\nlet context = new SearchContext(typeof\u003cGoogle\u003e,\n                                typeof\u003cDuckDuckGo\u003e,\n                                typeof\u003cYahoo\u003e)\nlet results = context.SearchAsync(\"quantum physics\")\n              |\u003e Async.AwaitTask\n              |\u003e Async.RunSynchronously\n\nresults |\u003e Seq.iter (fun x -\u003e printfn \"%s %s\" x.Url x.Description)\n```\n\n## Libraries\n\n- [AngleSharp](https://github.com/AngleSharp/AngleSharp)\n- [AngleSharp.Io](https://github.com/AngleSharp/AngleSharp.Io)\n- [SharpX](https://github.com/gsscoder/sharpx)\n- [CommandLineParser](https://github.com/commandlineparser/commandline)\n- [xUnit.net](https://github.com/xunit/xunit)\n- [FluentAssertions](https://github.com/fluentassertions/fluentassertions)\n- [WaffleGenerator](https://github.com/SimonCropp/WaffleGenerator)\n- [Bogus](https://github.com/bchavez/Bogus)\n\n## Tools\n\n- [Paket](https://github.com/fsprojects/Paket)\n\n## Icon\n\n- [Search Engine](https://thenounproject.com/search/?q=search%20engine\u0026i=2054907) icon designed by Vectors Market from [The Noun Project](https://thenounproject.com/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgsscoder%2Fpickall","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgsscoder%2Fpickall","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgsscoder%2Fpickall/lists"}