{"id":28527208,"url":"https://github.com/welaika/sputnik","last_synced_at":"2025-07-05T22:31:05.360Z","repository":{"id":57551603,"uuid":"113530102","full_name":"welaika/sputnik","owner":"welaika","description":"Crawling since 1957","archived":false,"fork":false,"pushed_at":"2021-10-11T14:15:12.000Z","size":519,"stargazers_count":30,"open_issues_count":0,"forks_count":3,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-06-09T12:12:29.040Z","etag":null,"topics":["elixir"],"latest_commit_sha":null,"homepage":"https://hexdocs.pm/sputnik/","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/welaika.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-08T04:06:15.000Z","updated_at":"2025-05-16T22:24:09.000Z","dependencies_parsed_at":"2022-09-26T18:41:33.035Z","dependency_job_id":null,"html_url":"https://github.com/welaika/sputnik","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/welaika/sputnik","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welaika%2Fsputnik","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welaika%2Fsputnik/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welaika%2Fsputnik/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welaika%2Fsputnik/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/welaika","download_url":"https://codeload.github.com/welaika/sputnik/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welaika%2Fsputnik/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263818364,"owners_count":23516084,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir"],"created_at":"2025-06-09T12:12:15.650Z","updated_at":"2025-07-05T22:31:05.355Z","avatar_url":"https://github.com/welaika.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](https://github.com/spawnfest/sputnik/blob/master/static/sputnik_logo_mini.png)\n\n# Sputnik\nby weLaika\n\nSputnik is a website crawler written in Elixir.\n\nIt crawls a website following all internal links and makes a report of all pages' status codes.\n\nWith query flags you can pass one ore more css selector to produce pages report about that.\n\n## Build\n\nSputnik can be built with:\n\n```\nmix deps.get\nmix escript.build\n```\n\n## Usage\n\nSputnik takes the url to crawl and optional query to perform on the crawled pages:\n\n### Options\n\n- query: valid css selectors, separated by commas, that you want to analyze all over the website\n- connections: max number of concurrent HTTP connections (default is 10)\n\n```\nsputnik [--query \u003cQ\u003e --query \u003cQ1\u003e ...] [--connections \u003cN\u003e] \u003curl\u003e\n```\n\n## Examples\n\nrunning\n\n```\n./sputnik \"http://spawnfest.github.io\" --query \"div\" --query \"a\" --query \"h1,h2,h3,h4,h5,h6\" --connections 10\n```\n\nproduces the following output\n\n```\n#################### Pages ####################\nPages found: 19\nstatus_code 200: 12\nstatus_code 301: 7\n\n\n#################### Queries ####################\n## query `a` ##\n327 result(s)\nMin 18 result(s) per page\nMax 57 result(s) per page\n## query `div` ##\n347 result(s)\nMin 13 result(s) per page\nMax 53 result(s) per page\n## query `h1,h2,h3,h4,h5,h6` ##\n95 result(s)\nMin 0 result(s) per page\nMax 31 result(s) per page\n\n```\n\nand it opens the browser with a page like this\n\n![](https://github.com/spawnfest/sputnik/blob/gh-pages/assets/screenshot.png)\n\n## Requirements\n\nDocumentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc)\nand published on [HexDocs](https://hexdocs.pm). Once published, the docs can\nbe found at [https://hexdocs.pm/sputnik](https://hexdocs.pm/sputnik).\n\n## Testing\n\nTo run tests:\n\n```bash\n$ mix test --cover\n```\n\nTo run credo:\n\n```bash\n$ mix credo\n```\n\n## Documentation\n\nTo generate the documentation:\n\n```bash\n$ mix docs \u0026\u0026 open doc/index.html\n```\n\n## Releasing\n\nBump the version in `mix.exs`, commit \u0026\u0026 push, and run `mix hex.publish`\nPlease read [https://hex.pm/docs/publish](https://hex.pm/docs/publish) for help.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwelaika%2Fsputnik","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwelaika%2Fsputnik","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwelaika%2Fsputnik/lists"}