{"id":22802637,"url":"https://github.com/periareon/req-compile","last_synced_at":"2026-02-14T02:01:54.073Z","repository":{"id":45328734,"uuid":"168985863","full_name":"periareon/req-compile","owner":"periareon","description":"Python requirements compilation","archived":false,"fork":false,"pushed_at":"2026-02-08T21:05:39.000Z","size":1001,"stargazers_count":15,"open_issues_count":12,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-02-09T02:47:13.031Z","etag":null,"topics":["bazel","bazel-rules","python","python3","starlark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/periareon.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-02-03T19:45:00.000Z","updated_at":"2026-02-08T21:05:42.000Z","dependencies_parsed_at":"2024-01-17T01:26:16.577Z","dependency_job_id":"1cd6404a-c58b-49aa-a5ac-a9fb26e3e77f","html_url":"https://github.com/periareon/req-compile","commit_stats":{"total_commits":322,"total_committers":2,"mean_commits":161.0,"dds":0.2142857142857143,"last_synced_commit":"3801c1a09dd38821d3f75ce43350fd4f450d42e2"},"previous_names":["sputt/qer","periareon/req-compile"],"tags_count":36,"template":false,"template_full_name":null,"purl":"pkg:github/periareon/req-compile","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/periareon%2Freq-compile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/periareon%2Freq-compile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/periareon%2Freq-compile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/periareon%2Freq-compile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/periareon","download_url":"https://codeload.github.com/periareon/req-compile/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/periareon%2Freq-compile/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29431593,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T22:20:51.549Z","status":"online","status_checked_at":"2026-02-14T02:00:07.626Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bazel","bazel-rules","python","python3","starlark"],"created_at":"2024-12-12T09:06:49.071Z","updated_at":"2026-02-14T02:01:54.067Z","avatar_url":"https://github.com/periareon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Req-Compile\n===========\n\n.. image:: https://img.shields.io/pypi/v/req-compile.svg\n    :alt: PyPI package version\n    :target: https://pypi.python.org/pypi/req-compile\n\n.. image:: https://github.com/periareon/req-compile/actions/workflows/build.yml/badge.svg\n    :alt: Github build status\n    :target: https://github.com/periareon/req-compile\n\nReq-Compile is a Python requirements compiler for large projects. It produces fully pinned,\nauditable solutions and supports local sources, constraint-only inputs, and cached downloads\nfor offline installs.\n\nHighlights\n----------\n* Produce an output file consisting of fully constrained exact versions of your requirements\n* Identify sources of constraints on your requirements\n* Constrain your output requirements using requirements that will not be included in the output\n* Save distributions that are downloaded while compiling in a configurable location\n* Use a current solution as a source of requirements to compile subsets without re-downloading\n\nBazel integration\n-----------------\n\nAlso supplied by this repository is the Bazel module ``rules_req_compile``, which provides a binary\nto compile a requirements solution and install third-party distributions from that solution. This\nproject differs from ``rules_python`` in that the URLs for the distributions are written to the\nsolution, enabling deterministic and repo-cacheable downloads and installs of third-party dependencies.\n\nTo get started with Bazel, add the following to your ``MODULE.bazel`` file:\n\n.. code-block:: python\n\n    # Find {version} on the req-compile GitHub releases page.\n    bazel_dep(name = \"rules_req_compile\", version = \"{version}\")\n\n    requirements = use_extension(\"//extensions:python.bzl\", \"requirements\")\n    requirements.parse(\n        name = \"my_pip_deps\",\n        requirements_locks = {\n            \"//3rdparty:requirements.linux.txt\": \"@platforms//os:linux\",\n            \"//3rdparty:requirements.macos.txt\": \"@platforms//os:macos\",\n            \"//3rdparty:requirements.windows.txt\": \"@platforms//os:windows\",\n        },\n    )\n    use_repo(requirements, \"my_pip_deps\")\n\n\nWhy use it?\n-----------\n**pip** and **pip-tools** are missing features and lack usability for some important workflows:\n\n* Using a previous solution as an input file to avoid hitting the network\n* pip-compile can't consider constraints that are not included in the final output. While pip accepts a constraints file, there is no way to stop at the \"solving\" phase, which would be used to push a fully solved solution to your repo\n* Track down where conflicting constraints originate\n* Treating source directories recursively as sources of requirements, like with --find-links\n* Configuring a storage location for downloaded distributions. Finding a fresh solution to a set of input requirements always requires downloading distributions\n\nA common workflow that is difficult to achieve with other tools:\n\nYou have a project with requirements ``requirements.txt`` and test requirements ``test-requirements.txt``. You want\nto produce a fully constrained output of ``requirements.txt`` to use to deploy your application. Easy, right? Just\ncompile ``requirements.txt``. However, if your test requirements will in any way constrain packages you need,\neven those needed transitively, it means you will have tested with different versions than you'll ship.\n\nFor this reason, you can use Req-Compile to compile ``requirements.txt`` using ``test-requirements.txt`` as constraints.\n\nThe Basics\n----------\n\nInstall and run\n~~~~~~~~~~~~~~~\nReq-Compile can be simply installed by running::\n\n    pip install req-compile\n\nTwo entrypoint scripts are provided::\n\n    req-compile \u003cinput reqfile1\u003e ... \u003cinput_reqfileN\u003e [--constraints constraint_file] [repositories, such as --index-url https://...]\n    req-candidates [requirement] [repositories, such as --index-url https://...]\n\nProducing output requirements\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nTo produce a fully constrained set of requirements for a given number of input requirements files, pass requirements\nfiles to req-compile::\n\n    \u003e cat requirements.txt\n    astroid \u003e= 2.0.0\n    isort \u003e= 4.2.5\n    mccabe\n\n    \u003e req-compile req-compile requirements.txt\n    astroid==2.9.0            # requirements.txt (\u003e=2.0.0)\n    isort==5.10.1             # requirements.txt (\u003e=4.2.5)\n    lazy-object-proxy==1.7.1  # astroid (\u003e=1.4.0)\n    mccabe==0.6.1             # requirements.txt\n    setuptools==60.0.1        # astroid (\u003e=20.0)\n    typed-ast==1.5.1          # astroid (\u003c2.0,\u003e=1.4.0)\n    typing_extensions==4.0.1  # astroid (\u003e=3.10)\n    wrapt==1.13.3             # astroid (\u003c1.14,\u003e=1.11)\n\n\nOutput is always emitted to stdout. Possible inputs include::\n\n    \u003e req-compile\n    \u003e req-compile .\n    # Compiles the current directory (looks for a setup.py or pyproject.toml)\n\n    \u003e req-compile subdir/project\n    # Compiles the project in the subdir/project directory\n\n    \u003e req-candidates --paths-only | req-compile\n    # Search for candidates and compile them piped in via stdin\n\n    \u003e echo flask | req-compile\n    # Compile the requirement 'flask' using the default remote index (PyPI)\n\n    \u003e req-compile . --extra test\n    # Compiles the current directory with the extra \"test\"\n\n\nSpecifying source of distributions\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nReq-Compile supports obtaining python distributions from multiple sources, each of which can be specified more than once.\nThese are referred to as repositories. If a candidate can be found in a provided solution or source directory, the remaining\nrepositories will not be considered. This is important for \"lazy\" requirement updates (e.g. only updating what is necessary\nto find a solution, and otherwise keep the existing solution) and ensure that source directories\ntake precedence over remote repositories.\n\nThe following repositories can be specified:\n\n* ``--solution``\n\n  Load a previous solution and use it as a source of distributions. This will allow a full\n  recompilation of a working solution without requiring any other source. If the\n  solution file can't be found, a warning will be emitted but not cause a failure\n* ``--source``\n\n  Use a local filesystem with source python packages to compile from. This will search the entire\n  tree specified at the source directory, until an __init__.py is reached. ``--remove-source`` can\n  be supplied to remove results that were obtained from source directories. You may want to do\n  this if compiling for a project and only third party requirements compilation results need to be saved.\n* ``--find-links``\n\n  Read a directory to load distributions from. The directory can contain anything\n  a remote index would, wheels, zips, and source tarballs. This matches pip's command line.\n* ``--index-url``\n\n  URL of a remote index to search for packages in. When compiling, it's necessary to download\n  a package to determine its requirements. ``--wheel-dir`` can be supplied to specify where to save\n  these distributions. Otherwise they will be deleted after compilation is complete. When specified,\n  replaces the default index that is located in pip.conf/pip.ini on your system.\n* ``--extra-index-url``\n\n  Extra remote index to search. Same semantics as index-url, but searched afterward. Additionally,\n  does not replace the default index URL so it can be used as a supplemental source of requirements\n  without knowing (or recording in the solution) the default index URL.\n\nAll options can be repeated multiple times, with the resolution order within solution and source matching what\nwas passed on the commandline.\n\nBy default, PyPI (https://pypi.org/) or the default pip index is added as a default repository. It can be removed by passing\n``--no-index`` on the commandline or passing a different index via ``--index-url``.\n\nIdentifying source of constraints\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nWhy did I just get version 1.11.0 of ``six``? Find out by examining the output::\n\n    six==1.11.0  # astroid, pathlib2, pymodbus (==1.11.0), pytest (\u003e=1.10.0), more_itertools (\u003c2.0.0,\u003e=1.0.0)\n\n\nIn the above output, the (==1.11.0) indicates that pymodbus, the requirement name listed before the\nparenthesis, specifically requested version 1.11.0 of six.\n\nConstraining output\n~~~~~~~~~~~~~~~~~~~\nConstrain production outputs with test requirements using the ``--constraints`` flag. More than one file can be\npassed::\n\n    \u003e cat requirements.txt\n    astroid\n\n    \u003e cat test-requirements.txt\n    pylint\u003c1.6\n\n    \u003e req-compile requirements.txt --constraints test-requirements.txt\n    astroid==1.4.9            # pylint (\u003c1.5.0,\u003e=1.4.5), requirements.txt\n    lazy-object-proxy==1.7.1  # astroid\n    six==1.16.0               # astroid, pylint\n    wrapt==1.13.3             # astroid\n\n\nNote that astroid is constrained by ``pylint``, even though ``pylint`` is not included in the output.\n\nIf a passed constraints file is fully pinned, Req-Compile will not attempt to find a solution for\nthe requirements passed in the constraints files. This behavior only occurs if ALL of the requirements\nlisted in the constraints files are pinned. This is because pinning a single requirement may\nstill bring in transitive requirements that would affect the final solution. The heuristic of\nchecking that all requirements are pinned assumes that you are providing a full solution.\n\nAdvanced Features\n-----------------\nCompiling a constrained subset\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nInput can be supplied via stdin as well as via as through files. For example, to supply a full\nsolution through a second compilation in order to obtain a subset of requirements, the\nfollowing cmdline might be used::\n\n    \u003e req-compile requirements.txt --constraints compiled-requirements.txt\n\nor, for example to consider two projects together::\n\n    \u003e req-compile /some/other/project /myproject | req-compile /myproject --solution -\n\nwhich is equivalent to::\n\n    \u003e req-compile /myproject --constraints /some/other/project\n\nResolving constraint conflicts\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nConflicts will automatically print the source of each conflicting requirement::\n\n    \u003e cat projectreqs.txt\n    astroid\u003c1.6\n    pylint\u003e=1.5\n\n    \u003e req-compile projectreqs.txt\n    No version of astroid could possibly satisfy the following requirements (astroid\u003c1.6,\u003c3,\u003e=2.3.0):\n      projectreqs.txt -\u003e astroid\u003c1.6\n      projectreqs.txt -\u003e pylint 2.4.1 -\u003e astroid\u003c3,\u003e=2.3.0\n\nSaving distributions\n~~~~~~~~~~~~~~~~~~~~\nFiles downloading during the compile process can be saved for later install. This can optimize\nthe execution times of builds when a separate compile step is required::\n\n    \u003e req-compile projectreqs.txt --wheel-dir .wheeldir \u003e compiledreqs.txt\n    \u003e pip install -r compiledreqs.txt --find-links .wheeldir --no-index\n\nEnvironment variables\n---------------------\nThe following environment variables control compile behavior. All are optional and use the\ndefaults shown if not set.\n\nREQ_COMPILE_MAX_DEPTH\n  Maximum recursion depth while resolving dependencies. Default: 80.\n\nREQ_COMPILE_MAX_DOWNGRADE\n  Maximum number of version downgrades or walkback attempts used to resolve conflicts.\n  Default: 20.\n\nREQ_COMPILE_WHEEL_TIMEOUT\n  Timeout in seconds for building a wheel to extract metadata from a source distribution.\n  Default: 30.0.\n\nREQ_COMPILE_EGG_INFO_TIMEOUT\n  Timeout in seconds for running ``setup.py egg_info`` during metadata extraction.\n  Default: 15.0.\n\nCookbook\n--------\nSome useful patterns for projects are outlined below.\n\nCompile, then install\n~~~~~~~~~~~~~~~~~~~~~\nAfter requirements are compiled, the usual next step is to install them\ninto a virtualenv.\n\nA script for test might run::\n\n    \u003e req-compile --extra test --solution compiled-requirements.txt --wheel-dir .wheeldir \u003e compiled-requirements.txt\n    \u003e pip-sync compiled-requirement.txt --find-links .wheeldir --no-index\n    or\n    \u003e pip install -r compiled-requirements.txt --find-links .wheeldir --no-index\n\nThis would produce an environment containing all of the requirements and test requirements for the project\nin the current directory (as defined by a setup.py). This is a *stable* set, in that only changes to\nthe requirements and constraints would produce a new output. To produce a totally fresh compilation,\ndon't pass in a previous solution.\n\nThe find-links parameter to the sync or pip install will *reuse* the wheels already downloaded by Req-Compile during\nthe compilation phase. This will make the installation step entirely offline.\n\nWhen taking this environment to deploy, trim down the set to the install requirements::\n\n    \u003e req-compile --solution compiled-requirements.txt --no-index \u003e install-requirements.txt\n\ninstall-requirements.txt will contain the pinned requirements that should be installed in your\ntarget environment. The reason for this extra step is that you don't want to distribute\nyour test requirements, and you also want your installed requirements to be the same\nversions that you've tested with. In order to get all of your explicitly declared\nrequirements and all of the transitive dependencies, you can use the prior solution to\nextract a subset. Passing the ``--no-index`` makes it clear that this command will not\nhit the remote index at all (though this would naturally be the case as solution files\ntake precedence over remote indexes in repository search order).\n\nCompile for a group of projects\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nReq-Compile can discover requirements that are grouped together on the filesystem. The\n``req-candidates`` command will print discovered projects and with the ``--paths-only`` options\nwill dump their paths to stdout. This allows recursive discovery of projects that you\nmay want to compile together.\n\nFor example, consider a filesystem with this layout::\n\n    solution\n      \\_ utilities\n      |   \\_ network_helper\n      |_ integrations\n      |   \\_ github\n      \\_ frameworks\n          |_ neural_net\n          \\_ cluster\n\nIn each of the leaf nodes, there is a setup.py and full python project. To compile these\ntogether and ensure that their requirements will all install into the same environment::\n\n    \u003e cd solution\n    \u003e req-candidates --paths-only\n    /home/user/projects/solution/utilities/network_helper\n    /home/user/projects/solution/integrations/github\n    /home/user/projects/solution/frameworks/neural_net\n    /home/user/projects/solution/frameworks/cluster\n\n    \u003e req-candidates --paths-only | req-compile --extra test --solution compiled-requirements.txt --wheel-dir .wheeldir \u003e compiled-requirements.txt\n    .. all reqs and all test reqs compiled together...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperiareon%2Freq-compile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperiareon%2Freq-compile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperiareon%2Freq-compile/lists"}