{"id":19794183,"url":"https://github.com/openzim/devdocs","last_synced_at":"2025-05-01T02:30:58.838Z","repository":{"id":247355008,"uuid":"824462895","full_name":"openzim/devdocs","owner":"openzim","description":"devdocs.io to ZIM scraper","archived":false,"fork":false,"pushed_at":"2024-11-14T08:22:38.000Z","size":178,"stargazers_count":3,"open_issues_count":10,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-21T06:00:24.479Z","etag":null,"topics":["devdocs","scraper","zim"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openzim.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-05T07:29:57.000Z","updated_at":"2025-03-24T16:37:18.000Z","dependencies_parsed_at":"2024-07-08T09:53:29.435Z","dependency_job_id":"0dbeaa5a-b252-4bfc-8952-8ec1dd239725","html_url":"https://github.com/openzim/devdocs","commit_stats":null,"previous_names":["openzim/devdocs"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openzim%2Fdevdocs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openzim%2Fdevdocs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openzim%2Fdevdocs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openzim%2Fdevdocs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openzim","download_url":"https://codeload.github.com/openzim/devdocs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251812307,"owners_count":21647884,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["devdocs","scraper","zim"],"created_at":"2024-11-12T07:12:26.613Z","updated_at":"2025-05-01T02:30:58.831Z","avatar_url":"https://github.com/openzim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Devdocs scraper\n\nThis scraper downloads [devdocs.io](https://devdocs.io/) documentation databases and puts them in ZIM files,\na clean and user friendly format for storing content for offline usage.\n\n[![CodeFactor](https://www.codefactor.io/repository/github/openzim/devdocs/badge)](https://www.codefactor.io/repository/github/openzim/devdocs)\n[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)\n[![codecov](https://codecov.io/gh/openzim/devdocs/branch/main/graph/badge.svg)](https://codecov.io/gh/openzim/devdocs)\n[![PyPI version shields.io](https://img.shields.io/pypi/v/devdocs2zim.svg)](https://pypi.org/project/devdocs2zim/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/devdocs2zim.svg)](https://pypi.org/project/devdocs2zim)\n[![Docker](https://ghcr-badge.egpl.dev/openzim/devdocs/latest_tag?label=docker)](https://ghcr.io/openzim/devdocs)\n\n\n## Installation\n\nThere are three main ways to install and use `devdocs2zim` from most recommended to least:\n\n\u003cdetails\u003e\n\u003csummary\u003eInstall using a pre-built container\u003c/summary\u003e\n\n\n1. Download the image using `docker`:\n\n    ```sh\n    docker pull ghcr.io/openzim/devdocs\n    ```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003eBuild your own container\u003c/summary\u003e\n\n1. Clone the repository locally:\n\n    ```sh\n    git clone https://github.com/openzim/devdocs.git \u0026\u0026 cd devdocs\n    ```\n\n1. Build the image:\n\n    ```sh\n    docker build -t ghcr.io/openzim/devdocs .\n    ```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003eRun the software locally using Hatch\u003c/summary\u003e\n\n1. Clone the repository locally:\n\n    ```sh\n    git clone https://github.com/openzim/devdocs.git \u0026\u0026 cd devdocs\n    ```\n\n1. Install [Hatch](https://hatch.pypa.io/):\n\n    ```sh\n    pip3 install hatch\n    ```\n\n1. Start a hatch shell to install software and dependencies in an isolated virtual environment.\n\n    ```sh\n    hatch shell\n    ```\n\n1. Run the `devdocs2zim` command:\n\n    ```sh\n    devdocs2zim --help\n    ```\n\n\u003c/details\u003e\n\n## Usage\n\n\u003e [!WARNING]\n\u003e This project is still a work in progress and isn't ready for use yet, the commands below are examples only.\n\n\n```sh\n# Usage\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim [--all|--slug=SLUG|--first=N]\n\n# Fetch all documents\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --all\n\n# Fetch all documents except Ansible\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --all --skip-slug-regex \"^ansible.*\"\n\n# Fetch Vue related documents\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --slug vue~3 --slug vue_router~4\n\n# Fetch the docs for the two most recent versions of each software\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --first=2\n```\n\n\n**One of the following flags is required:**\n\n* `--all`: Fetch all Devdocs resources, and produce one ZIM per resource.\n* `--slug SLUG`: Fetch the provided Devdocs resource. Slugs are the first path entry in the Devdocs URL.\n    For example, the slug for: `https://devdocs.io/gcc~12/` is `gcc~12`. Use --slug several times to add multiple.\n* `--first N`: Fetch the first number of items per slug as shown in the DevDocs UI.\n\n\n**Optional Flags:**\n\n*  `--skip-slug-regex REGEX`: Skips slugs matching the given regular expression.\n*  `--output OUTPUT_FOLDER`: Output folder for ZIMs. Default: /output\n*  `--creator CREATOR`: Name of content creator. Default: 'DevDocs'\n*  `--publisher PUBLISHER`: Custom publisher name. Default: 'openZIM'\n*  `--name-format FORMAT`: Custom name format for individual ZIMs.\n    Default: 'devdocs_{slug_without_version}_{version}'\n*  `--title-format FORMAT`: Custom title format for individual ZIMs.\n    Value will be truncated to 30 chars. Default: '{full_name} Documentation'\n*  `--description-format FORMAT`: Custom description format for individual ZIMs.\n    Value will be truncated to 80 chars. Default: '{full_name} Documentation'\n*  `--long-description-format FORMAT`: Custom long description format for your ZIM.\n    Value will be truncated to 4000 chars.Default: '{full_name} documentation by DevDocs'\n*  `--tag TAG`: Add tag to the ZIM. Use --tag several times to add multiple.\n    Formatting is supported. Default: ['devdocs', '{slug_without_version}']\n*  `--logo-format FORMAT`: URL/path for the ZIM logo in PNG, JPG, or SVG format.\n    Formatting placeholders are supported. If unset, a DevDocs logo will be used.\n\n**Formatting Placeholders**\n\nThe following formatting placeholders are supported:\n\n* `{name}`: Human readable name of the resource e.g. `Python`.\n* `{full_name}`: Name with optional version for the resource e.g. `Python 3.12`.\n* `{slug}`: Devdocs slug for the resource e.g. `python~3.12`.\n* `{clean_slug}`: Slug with non alphanumeric/period characters replaced with `-` e.g. `python-3.12`.\n* `{slug_without_version}`: Devdocs slug for the resource without the version e.g. `python`.\n* `{version}`: Shortened version displayed in devdocs, if any e.g. `3.12`.\n* `{release}`: Specific release of the software the documentation is for, if any e.g. `3.12.1`.\n* `{attribution}`: License and attribution information about the resource.\n* `{home_link}`: Link to the project's home page, if any: e.g. `https://python.org`.\n* `{code_link}`: Link to the project's source, if any: e.g. `https://github.com/python/cpython`.\n* `{period}`: The current date in `YYYY-MM` format e.g. `2024-02`.\n\n## Developing\n\nUse the commands below to set up the project once:\n\n```sh\n# Install hatch if it isn't installed already.\n❯ pip install hatch\n\n# Local install (in default env) / re-sync packages\n❯ hatch run pip list\n\n# Set-up pre-commit\n❯ pre-commit install\n```\n\nThe following commands can be used to build and test the scraper:\n\n```sh\n# Show scripts\n❯ hatch env show\n\n# linting, testing, coverage, checking\n❯ hatch run lint:all\n❯ hatch run lint:fixall\n\n# run tests on all matrixed' envs\n❯ hatch run test:run\n\n# run tests in a single matrixed' env\n❯ hatch env run -e test -i py=3.12 coverage\n\n# run static type checks\n❯ hatch env run check:all\n\n# building packages\n❯ hatch build\n```\n\n\n### Contributing\n\nThis project adheres to openZIM's [Contribution Guidelines](https://github.com/openzim/overview/wiki/Contributing).\n\nThis project has implemented openZIM's [Python bootstrap, conventions and policies](https://github.com/openzim/_python-bootstrap/docs/Policy.md) **v1.0.3**.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenzim%2Fdevdocs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenzim%2Fdevdocs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenzim%2Fdevdocs/lists"}