{"id":15647753,"url":"https://github.com/spk/maman","last_synced_at":"2025-10-07T20:40:50.836Z","repository":{"id":57636506,"uuid":"57924270","full_name":"spk/maman","owner":"spk","description":"Rust Web Crawler saving pages on Redis","archived":false,"fork":false,"pushed_at":"2021-06-13T12:14:10.000Z","size":208,"stargazers_count":44,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-28T22:41:27.268Z","etag":null,"topics":["crawler","http","spider","web","web-crawler"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spk.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-05-02T22:04:44.000Z","updated_at":"2025-02-07T10:44:59.000Z","dependencies_parsed_at":"2022-08-30T09:40:49.755Z","dependency_job_id":null,"html_url":"https://github.com/spk/maman","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spk%2Fmaman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spk%2Fmaman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spk%2Fmaman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spk%2Fmaman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spk","download_url":"https://codeload.github.com/spk/maman/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248610360,"owners_count":21132924,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","http","spider","web","web-crawler"],"created_at":"2024-10-03T12:20:53.903Z","updated_at":"2025-10-07T20:40:45.790Z","avatar_url":"https://github.com/spk.png","language":"Rust","readme":"# Maman\n\nMaman is a Rust Web Crawler saving pages on Redis.\n\nPages are send to list `\u003cMAMAN_ENV\u003e:queue:maman` using\n[Sidekiq job format](https://github.com/mperham/sidekiq/wiki/Job-Format)\n\n``` json\n{\n\"class\": \"Maman\",\n\"jid\": \"b4a577edbccf1d805744efa9\",\n\"retry\": true,\n\"created_at\": 1461789979, \"enqueued_at\": 1461789979,\n\"args\": {\n    \"document\":\"\u003chtml\u003e\u003cbody\u003e\u003ca href='#' /\u003e\u003ca href='/new' /\u003e\u003c/html\u003e\",\n    \"urls\": [\"https://example.net/new\"],\n    \"headers\": {\"content-type\": \"text/html\"},\n    \"url\": \"https://example.net/\"\n    }\n}\n```\n\n## Dependencies\n\n* [Redis](http://redis.io/)\n\n## Installation\n\n### With cargo\n\n```\ncargo install maman\n```\n\n### With [just](https://github.com/casey/just)\n\n```\nPREFIX=~/.local just install\n```\n\n## Usage\n\n```\nmaman URL [LIMIT] [MIME_TYPES]\n```\n\n`LIMIT` must be an integer or `0` is the default, meaning no limit.\n\n## Environment variables\n\n### Defaults\n\n* MAMAN_ENV=development\n* REDIS_URL=\"redis://127.0.0.1/\"\n\n### Others\n\n* RUST_LOG=maman=info\n\n## LICENSE\n\nThe MIT License\n\nCopyright (c) 2016-2021 Laurent Arnoud \u003claurent@spkdev.net\u003e\n\n---\n[![Build](https://img.shields.io/github/workflow/status/spk/maman/CI/master.svg)](https://github.com/spk/maman/actions)\n[![Version](https://img.shields.io/crates/v/maman.svg)](https://crates.io/crates/maman)\n[![Documentation](https://img.shields.io/badge/doc-rustdoc-blue.svg)](https://docs.rs/maman/)\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT \"MIT\")\n[![Dependency status](https://deps.rs/repo/github/spk/maman/status.svg)](https://deps.rs/repo/github/spk/maman)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspk%2Fmaman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspk%2Fmaman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspk%2Fmaman/lists"}