{"id":20822125,"url":"https://github.com/internetarchive/strainer","last_synced_at":"2026-04-19T16:06:49.644Z","repository":{"id":66163212,"uuid":"379321894","full_name":"internetarchive/strainer","owner":"internetarchive","description":"Heritrix frontier files manipulation tool.","archived":false,"fork":false,"pushed_at":"2021-06-23T13:51:08.000Z","size":39,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-12-27T01:59:11.839Z","etag":null,"topics":["crawling","frontier","heritrix"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/internetarchive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-06-22T15:44:06.000Z","updated_at":"2024-08-22T12:48:23.000Z","dependencies_parsed_at":"2023-03-10T23:41:52.323Z","dependency_job_id":null,"html_url":"https://github.com/internetarchive/strainer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/internetarchive/strainer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Fstrainer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Fstrainer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Fstrainer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Fstrainer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/internetarchive","download_url":"https://codeload.github.com/internetarchive/strainer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Fstrainer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32012832,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawling","frontier","heritrix"],"created_at":"2024-11-17T22:14:00.979Z","updated_at":"2026-04-19T16:06:49.638Z","avatar_url":"https://github.com/internetarchive.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![made-with-Go](https://img.shields.io/badge/Made%20with-Go-1f425f.svg)](http://golang.org)\n[![GitHub go.mod Go version of a Go module](https://img.shields.io/github/go-mod/go-version/internetarchive/strainer.svg)](https://github.com/internetarchive/strainer)\n[![Go Report Card](https://goreportcard.com/badge/github.com/internetarchive/strainer)](https://goreportcard.com/report/github.com/internetarchive/strainer)\n[![GitHub license](https://img.shields.io/github/license/internetarchive/strainer.svg)](https://github.com/internetarchive/strainer/blob/master/LICENSE)\n\n# Strainer\n\nParse Heritrix frontiers, build seeds list, exclude hosts, crawl better.\n\n## What's strainer?\n\nStrainer is a tool built to manipulate Heritrix frontier files.\nIt takes one or multiple (GZIP or not) frontier files and build new seeds list based on them.\n\nIt can exclude specific hosts, limit the number of occurrence of hosts in the final seeds list, and give you interesting statistics about your frontiers.\n\nIt's still a WIP, but it's usable right now.\n\n## Usage\n\n```\nusage: strainer [-h|--help] -f|--file \"\u003cvalue\u003e\" [-f|--file \"\u003cvalue\u003e\" ...]\n                [-m|--max-host-occurrence \u003cinteger\u003e] [-e|--excluded-hosts\n                \"\u003cvalue\u003e\" [-e|--excluded-hosts \"\u003cvalue\u003e\" ...]] [--temp-dir\n                \"\u003cvalue\u003e\"]\n\n                manipulate Heritrix frontier files\n\nArguments:\n\n  -h  --help                 Print help information\n  -f  --file                 Frontier file(s) to process, can be .gz files.\n  -m  --max-host-occurrence  Max number of a occurrence of a given host to\n                             accept in the final seed list. If an host is\n                             parsed more than X times, new occurrences of that\n                             host past that limit will be excluded. -1 value\n                             means no limit. Default: -1\n  -e  --excluded-hosts       Specific hosts to exclude from the final seed\n                             list.\n      --temp-dir             Temporary directory to use for the key/value\n                             database. Default: /tmp\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finternetarchive%2Fstrainer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finternetarchive%2Fstrainer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finternetarchive%2Fstrainer/lists"}