{"id":30694356,"url":"https://github.com/danterolle/repo","last_synced_at":"2026-05-18T03:34:09.704Z","repository":{"id":209709214,"uuid":"724756844","full_name":"danterolle/repo","owner":"danterolle","description":"Repository Extraction and Parsing Operations. These scripts allow you to download the Packages file from any Debian repository (including ParrotOS), format it as needed to get a JSON format easily usable in the web.","archived":false,"fork":false,"pushed_at":"2023-12-04T13:14:57.000Z","size":65,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-02T06:11:22.217Z","etag":null,"topics":["debian","parrotsec","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danterolle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-11-28T18:24:43.000Z","updated_at":"2024-01-25T18:04:28.000Z","dependencies_parsed_at":"2023-12-04T13:58:55.289Z","dependency_job_id":null,"html_url":"https://github.com/danterolle/repo","commit_stats":null,"previous_names":["danterolle/repo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/danterolle/repo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danterolle%2Frepo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danterolle%2Frepo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danterolle%2Frepo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danterolle%2Frepo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danterolle","download_url":"https://codeload.github.com/danterolle/repo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danterolle%2Frepo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33163756,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T22:39:12.733Z","status":"online","status_checked_at":"2026-05-18T02:00:06.436Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["debian","parrotsec","python3"],"created_at":"2025-09-02T06:07:14.136Z","updated_at":"2026-05-18T03:34:09.699Z","avatar_url":"https://github.com/danterolle.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# REPO (Repository Extraction and Parsing Operations)\n\nThis repository is a reimplementation written in Python 3 of [the following project for ParrotOS](https://github.com/danterolle/packages-filter).\n\nIt is all been redesigned and written again to cope with the new structural change in the ParrotOS repositories, particularly because of the Debian 12-based version 6.0.\n\nIt works better, it is better managed, and will be further improved.\n\nSo, these scripts allow you to download the Packages file from any Debian repository (including ParrotOS), format it as needed to get a JSON format easily usable in the web.\n\n## Getting started\n\nIn this project, each script has a specific task. A dedicated Makefile will probably be created in the future or everything will be rearranged with a better project structure.\n\nEach script should be executed in exactly the order in which they are shown in this README.\n\nIn addition, each time these scripts are used, a log file is created in a temporary folder called `tmp` that will contain their execution status \n\n### `repo_downloader.py`\n \nThis script is designed to download \"Packages\" files from a specific Debian repository, using information specified by the user through command-line arguments.\n\n\u003cdetails\u003e\n  \u003csummary\u003eCommand line arguments\u003c/summary\u003e\n\n  `--base-url` allows the user to specify a custom base URL for the Debian repository. The default URL is **https://deb.parrot.sh/parrot/dists/**\n\n  `--codename` allows the user to specify one or more Parrot/Debian name codes for which to download \"Packages\" files. In Parrot, could be **lory**.\n\n  `--branch` allows the user to specify one or more repository branches, for instance: \n    \n    main\n    contrib\n    non-free\n    non-free-firmware\n\n  `--architecture` allows the user to specify one or more architectures (it is necessary to keep this format as it is standard for the repository itself):\n    \n    binary-amd64\n    binary-arm64\n    binary-armhf\n    binary-i386\n\n\u003c/details\u003e\n\n#### Usage example\n\n```\n$ python3 repo_downloader.py --codename lory --branch main contrib --architecture arm64 i386\n```\n\nYou can also have a helper printed on terminal by typing:\n\n```\n$ python3 repo_downloader.py -h\n```\n\n```\nusage: repo_downloader.py [-h] [--base-url BASE_URL] [--codename CODENAME [CODENAME ...]]\n                          [--branch BRANCH [BRANCH ...]]\n                          [--architecture ARCHITECTURE [ARCHITECTURE ...]]\n\nDownload Parrot Packages files from a specified repository.\n\noptions:\n  -h, --help            show this help message and exit\n  --base-url BASE_URL   Specify a custom base URL.\n  --codename CODENAME [CODENAME ...]\n                        Specify codenames to download Packages for.\n  --branch BRANCH [BRANCH ...]\n                        Specify branches to download Packages for.\n  --architecture ARCHITECTURE [ARCHITECTURE ...]\n                        Specify architectures to download Packages for.\n```\n\n### `format_packages.py`\n\nThis script performs formatting operations on Parrot/Debian Packages files. It processes each package block within the input files, updates the description and tag fields, and saves the modified content to new output files.\n\n[What problem does it solve?](https://github.com/danterolle/repo/blob/781619acb1f3cff23c4b4247006e5bd3e339f487/format_packages.py#L5C7-L5C7)\n\n\u003cdetails\u003e\n  \u003csummary\u003eCommand line arguments\u003c/summary\u003e\n\n  `input_directory` allows the user to select a directory where the correction is to take place. It is recursive.\n\n  `output_directory` allows the user to select the output directory where the processed files will be created.\n\n\u003c/details\u003e\n\n#### Usage example\n\n```\n$ python3 format_packages.py lory/ lory/\n```\n\nYou can also have a helper printed on terminal by typing:\n\n```\nusage: format_packages.py [-h] input_directory output_directory\n\nFormat Parrot/Debian Packages files in a specified directory.\n\npositional arguments:\n  input_directory   Specify the input directory containing Packages files.\n  output_directory  Specify the output directory for processed Packages files.\n\noptions:\n  -h, --help        show this help message and exit\n```\n\n### `json_parser.py`\n\nThis script (recursively) processes a specified root directory, identifies \"Packages\" files within it, extracts information from these files, and saves the parsed data in JSON format. The resulting JSON files are organized in a specified output directory maintaining the directory structure of the input. \n\n\u003cdetails\u003e\n  \u003csummary\u003eCommand line arguments\u003c/summary\u003e\n\n  `input_directory` allows the user to select a directory where the Packages files are located.\n\n  `output_directory` allows the user to select the output directory where the *Packages.json* files will be placed.\n\n\u003c/details\u003e\n\n#### Usage example\n\n```\n$ python3 json_parser.py --recursive lory/ output/ \n```\n\nYou can also have a helper printed on terminal by typing:\n\n```\nusage: json_parser.py [-h] [--recursive] input_directory output_directory\n\nParse multiple Packages files and save JSON outputs.\n\npositional arguments:\n  input_directory   Path to the root directory containing Packages files.\n  output_directory  Path to the directory for saving JSON outputs.\n\noptions:\n  -h, --help        show this help message and exit\n  --recursive       Recursively process subdirectories.\n```\n\n### `server.py`\n\nThis script implements a FastAPI server that provides an API endpoint for retrieving information about packages from a repository, based on user-provided query parameters.\n\nIt's still being evaluated and from a security point of view it could be improved.\n\n#### Usage example\n\nStart the app\n```\n$ uvicorn server:app --reload\n```\n\nReturn the complete list of packages for a given branch and architecture\n```\n$ curl -X GET \"http://127.0.0.1:8000/packages/?architecture=amd64\u0026branch=main\"\n```\n\nReturn only the information of a specific package\n```\ncurl -X GET \"http://127.0.0.1:8000/packages/?package_name=0ad\u0026architecture=amd64\u0026branch=main\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanterolle%2Frepo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanterolle%2Frepo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanterolle%2Frepo/lists"}