{"id":19596288,"url":"https://github.com/peroxy/avtonetscraper","last_synced_at":"2025-04-23T08:16:59.274Z","repository":{"id":54666566,"uuid":"263852102","full_name":"peroxy/AvtoNetScraper","owner":"peroxy","description":"Scrapes car information from website avto.net and stores it into a local sqlite database","archived":false,"fork":false,"pushed_at":"2021-02-04T20:15:31.000Z","size":86,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-23T08:16:53.545Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/peroxy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-14T07:59:36.000Z","updated_at":"2024-08-28T18:24:25.000Z","dependencies_parsed_at":"2022-08-13T23:20:15.133Z","dependency_job_id":null,"html_url":"https://github.com/peroxy/AvtoNetScraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peroxy%2FAvtoNetScraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peroxy%2FAvtoNetScraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peroxy%2FAvtoNetScraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peroxy%2FAvtoNetScraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/peroxy","download_url":"https://codeload.github.com/peroxy/AvtoNetScraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250395286,"owners_count":21423400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T08:52:40.552Z","updated_at":"2025-04-23T08:16:59.256Z","avatar_url":"https://github.com/peroxy.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AvtoNetScraper\nScrapes car information from Slovenian car website [AVTO.NET](https://avto.net) and stores it into a local sqlite database. \nThis was developed to store all avto.net car ads into a database and analyze them. This was not meant to be used when buying a car and getting notifications when a new car pops up. It can definitely be modified for that purpose though, would not require much work.\n\n## How to run\n### Requirements\nBefore you can run this project you will need these:\n- [.NET core SDK](https://dotnet.microsoft.com/download)\n- [dotnet ef](https://docs.microsoft.com/en-us/ef/core/miscellaneous/cli/dotnet) (can be installed via `dotnet tool install --global dotnet-ef`)\n\n### Setting up config\nOpen `appsettings.template.json` and setup custom urls, request intervals in milliseconds and local image download directory. URL values should be copied from avto.net after you have specified search filters. After you are done customizing the file, rename or save it as `appsettings.json`.\n\nPlease be aware of avto.net limitation that only shows a maximum of 1000 car ads in their search - even if more should be included in the filter. It is suggested to limit your search in those cases to be always under 1000 or they won't get scraped.\n\nHere is an example of appsettings.json file:\n\n```\n{\n  \"appSettings\": {\n    \"searchFilterUrls\": [\n      \"https://www.avto.net/Ads/results.asp?znamka=Ford\u0026model=\u0026modelID=\u0026tip=katerikoli%20tip\u0026znamka2=\u0026model2=\u0026tip2=katerikoli%20tip\u0026znamka3=\u0026model3=\u0026tip3=katerikoli%20tip\u0026cenaMin=0\u0026cenaMax=999999\u0026letnikMin=0\u0026letnikMax=2090\u0026bencin=0\u0026starost2=999\u0026oblika=0\u0026ccmMin=0\u0026ccmMax=99999\u0026mocMin=\u0026mocMax=\u0026kmMin=0\u0026kmMax=9999999\u0026kwMin=0\u0026kwMax=999\u0026motortakt=\u0026motorvalji=\u0026lokacija=0\u0026sirina=\u0026dolzina=\u0026dolzinaMIN=\u0026dolzinaMAX=\u0026nosilnostMIN=\u0026nosilnostMAX=\u0026lezisc=\u0026presek=\u0026premer=\u0026col=\u0026vijakov=\u0026EToznaka=\u0026vozilo=\u0026airbag=\u0026barva=\u0026barvaint=\u0026EQ1=1000000000\u0026EQ2=1000000000\u0026EQ3=1000000000\u0026EQ4=100000000\u0026EQ5=1000000000\u0026EQ6=1000000000\u0026EQ7=1110100120\u0026EQ8=1010000001\u0026EQ9=100000000\u0026KAT=1010000000\u0026PIA=\u0026PIAzero=\u0026PSLO=\u0026akcija=\u0026paketgarancije=\u0026broker=\u0026prikazkategorije=\u0026kategorija=\u0026zaloga=\u0026arhiv=\u0026presort=\u0026tipsort=\u0026stran=\",\n      \"https://www.avto.net/Ads/results.asp?znamka=Ford\u0026model=\u0026modelID=\u0026tip=katerikoli%20tip\u0026znamka2=\u0026model2=\u0026tip2=katerikoli%20tip\u0026znamka3=\u0026model3=\u0026tip3=katerikoli%20tip\u0026cenaMin=0\u0026cenaMax=999999\u0026letnikMin=0\u0026letnikMax=2090\u0026bencin=0\u0026starost2=999\u0026oblika=0\u0026ccmMin=0\u0026ccmMax=99999\u0026mocMin=\u0026mocMax=\u0026kmMin=0\u0026kmMax=9999999\u0026kwMin=0\u0026kwMax=999\u0026motortakt=\u0026motorvalji=\u0026lokacija=0\u0026sirina=\u0026dolzina=\u0026dolzinaMIN=\u0026dolzinaMAX=\u0026nosilnostMIN=\u0026nosilnostMAX=\u0026lezisc=\u0026presek=\u0026premer=\u0026col=\u0026vijakov=\u0026EToznaka=\u0026vozilo=\u0026airbag=\u0026barva=\u0026barvaint=\u0026EQ1=1000000000\u0026EQ2=1000000000\u0026EQ3=1000000000\u0026EQ4=100000000\u0026EQ5=1000000000\u0026EQ6=1000000000\u0026EQ7=1110100120\u0026EQ8=1010000001\u0026EQ9=100000000\u0026KAT=1010000000\u0026PIA=\u0026PIAzero=\u0026PSLO=\u0026akcija=\u0026paketgarancije=\u0026broker=\u0026prikazkategorije=\u0026kategorija=\u0026zaloga=\u0026arhiv=\u0026presort=\u0026tipsort=\u0026stran=\"\n    ],\n    \"requestIntervalMs\": 10000,\n    \"imagesDirectory\": \"C:\\\\Pictures\"\n  }\n}\n```\n\n### Restoring packages and database\nGo to project directory and run commands: \n1. `dotnet restore` - this will restore all NuGet project packages\n2. `dotnet ef database update` - this will create a local sqlite database file, make sure the build action is set to copy to output directory\n\n### Running application\n`dotnet run` - this will run the console application with default configuration (by using `-urls -cars` arguments) and fill up your local sqlite database.\nOptional arguments: \n1. `dotnet run -- -urls` - will only parse car urls and insert them into database, will not make a request to each url and parse each car's info\n2. `dotnet run -- -cars` - will read the `Url` table from local database and make a request to each car url and parse each car's info into database\n3. `dotnet run -- -images` - will read the `Car` table from local database and make a request to each car url image and download it to a local path specified in appsettings\n\n## Development\n### Updating database model\nModify Database/Model.cs file and run commands:\n1. `dotnet ef migrations add Changelog`\n2. `dotnet ef database update`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperoxy%2Favtonetscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperoxy%2Favtonetscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperoxy%2Favtonetscraper/lists"}