{"id":15442882,"url":"https://github.com/thoth-station/adviser","last_synced_at":"2025-04-13T15:50:47.171Z","repository":{"id":30753761,"uuid":"123548968","full_name":"thoth-station/adviser","owner":"thoth-station","description":"The recommendation engine for Python software stacks and Dependency Monkey in project Thoth.","archived":false,"fork":false,"pushed_at":"2023-10-18T00:35:20.000Z","size":13737,"stargazers_count":34,"open_issues_count":24,"forks_count":13,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-10-18T13:16:40.104Z","etag":null,"topics":["artificial-intelligence","hacktoberfest","machine-learning","machinelearning","monte-carlo-tree-search","python","python3","reinforcement-learning","resolver","temporal-difference","thoth"],"latest_commit_sha":null,"homepage":"https://thoth-station.github.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thoth-station.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-03-02T07:59:45.000Z","updated_at":"2023-11-29T22:23:46.000Z","dependencies_parsed_at":"2023-02-14T15:15:30.781Z","dependency_job_id":"23ea9270-ef47-47ec-a8fd-908629059c4c","html_url":"https://github.com/thoth-station/adviser","commit_stats":{"total_commits":2329,"total_committers":37,"mean_commits":"62.945945945945944","dds":0.695148132245599,"last_synced_commit":"de8cf66e01ab4aacec8c69f505388ea14be7df01"},"previous_names":[],"tags_count":86,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoth-station%2Fadviser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoth-station%2Fadviser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoth-station%2Fadviser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoth-station%2Fadviser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thoth-station","download_url":"https://codeload.github.com/thoth-station/adviser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248741147,"owners_count":21154249,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","hacktoberfest","machine-learning","machinelearning","monte-carlo-tree-search","python","python3","reinforcement-learning","resolver","temporal-difference","thoth"],"created_at":"2024-10-01T19:31:10.758Z","updated_at":"2025-04-13T15:50:47.149Z","avatar_url":"https://github.com/thoth-station.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Thoth Adviser\n-------------\n\nWelcome to Thoth's adviser repository.\n\nThis repository provides sources for a component called \"thoth-adviser\" which\nserves the following purposes:\n\n1. Recommendation engine for `project Thoth \u003chttps://thoth-station.ninja\u003e`__.\n\n2. A `tool called \"Dependency Monkey\" \u003chttps://developers.redhat.com/articles/2021/09/17/resolve-python-dependencies-thoth-dependency-monkey\u003e`__\n   that can generate all the possible software stacks for a project respecting\n   dependency resolution in the Python ecosystem following programmable rules.\n\n3. Check provenance of installed Python artifacts based on package source\n   indexes used.\n\nIf you would like to interact with Thoth from user's perspective, check\n`Thamos repository \u003chttps://github.com/thoth-station/thamos\u003e`__.\n\nIf you would like to browse technical documentation, visit\n`thoth-adviser section\n\u003chttps://thoth-station.ninja/docs/developers/adviser/\u003e`__ at\n`thoth-station.ninja \u003chttps://thoth-station.ninja\u003e`_.\n\n.. image:: https://github.com/thoth-station/adviser/blob/master/docs/source/_static/rl_video.png?raw=true\n   :alt: A reinforcement learning based dependency resolution.\n   :align: center\n   :target:  https://www.youtube.com/watch?v=WEJ65Rvj3lc\n\nSoftware stack resolution pipeline\n==================================\n\nThe software stack generation is shared for `Dependency Monkey\n\u003chttps://thoth-station.ninja/docs/developers/adviser/dependency_monkey.html\u003e`__\nas well as for the recommendation engine. The core principle of the software\nstack generation lies in an abstraction called \"software stack resolution\npipeline\". This pipeline is made out of multiple units of different type that\nform atomic pieces to score packages that can occur in a software stack based\non the dependency resolution.\n\n.. image:: https://github.com/thoth-station/adviser/blob/master/docs/source/_static/pipeline_builder.gif?raw=true\n   :alt: Building a resolution pipeline.\n   :align: center\n\nAs can be seen in the animation shown above, the pipeline units that form the\nsoftware stack resolution pipeline are included in the pipeline dynamically on\nthe pipeline creation. A set of units included form a pipeline configuration.\n\nThe pipeline configuration is built by \"pipeline builder\" which asks each\npipeline unit for inclusion in the pipeline configuration. Each pipeline unit\ncan decide when and whether it should be included in the pipeline configuration\nconsidering aspects for the user software stacks, such as:\n\n* hardware available in the runtime environment when running the\n  application (e.g. CPU, GPU)\n\n* operating system and it's version used in the runtime environment where the\n  application is supposed to be run\n\n* software provided by the operating system, such as Python interpreter\n  version, CUDA version (for GPU computation) and other native dependencies\n  (e.g. glibc version, Intel MKL libraries, ...) and their ABI\n\n* user's intention with the software built - e.g. building a computational\n  intensive application, an application which should be secure for production\n  environments, latest-greatest software, ...\n\n* `type of inspections for the software quality checks\n  \u003chttps://github.com/thoth-station/amun-api/\u003e`__ - Dependency Monkey scenario\n\n* ...\n\nAll these vectors stated above form a \"context\" for pipeline builder (the robot\nin the animation) that creates the pipeline configuration (a set of pipeline units).\n\n**See also:**\n\n* `dev.to: How to beat Python’s pip: Software stack resolution pipelines \u003chttps://dev.to/fridex/how-to-beat-python-s-pip-software-stack-resolution-pipelines-19kg\u003e`__\n* `YouTube: Pipeline units in a software stack resolution process \u003chttps://www.youtube.com/watch?v=OCX8JQDXP9s\u003e`__\n* `Jupyter Notebook: Pipeline units in a software stack resolution process \u003chttps://github.com/thoth-station/notebooks/blob/master/notebooks/development/Pipeline%20units.ipynb\u003e`__\n\nPredictor guided resolution process\n===================================\n\nOnce the pipeline configuration is constructed, it is used to resolve software\nstacks meeting desired quality and purpose.\n\n.. image:: https://github.com/thoth-station/adviser/blob/master/docs/source/_static/pipeline.gif?raw=true\n   :alt: A resolution pipeline run.\n   :align: center\n\nResolver (the fairy in the animation) can resolve and walk through the\ndependency graph respecting Python packaging based on the `pre-aggregated data\nfrom dependency solver runs \u003chttps://github.com/thoth-station/solver/\u003e`__. The\nresolution process is guided by an abstraction called \"Predictor\" (shown as a\nmagician in the animation). Predictor decides which packages in the dependency\ngraph should be resolved by Resolver and thus be included in the resulting\nsoftware stacks. Packages that are resolved go through the software stack\nresolution pipeline which scores packages (positively, negatively or completely\ndiscard a package from a software stack resolved). The resolution pipeline can:\n\n* inject new packages or new package versions to the dependency graph based on\n  packages resolved (e.g. a package accidentally not stated as a dependency of\n  a library, dependency underpinning issues, ...)\n\n* remove a dependency in a specific version or the whole dependency from the\n  dependency graph (e.g. a package accidentally stated a a dependency, missing\n  ABI symbols in the runtime environment, dependency overpinning issues, ...)\n\n* score a package occurring in the dependency graph positively - prioritize\n  resolution of a specific package in the dependency graph (e.g. positive\n  performance aspect of a package in a specific version/build)\n\n* score a package in a specific version occurring in the dependency graph\n  negatively - prioritize resolution of other versions (e.g. a security\n  vulnerability present in a specific release)\n\n* prevent resolving a specific package in a specific version so that resolver\n  tries to find a different resolution path, if any (e.g. buggy package releases)\n\nThe pipeline units present in the pipeline configuration can take into account\n\"context\" as stated above - pipeline units can take into account\ncharacteristics of the runtime environment used (software and hardware\navailable), purpose of the application, ...\n\nPipeline units are of different types - Boots, Pseudonyms, Sieves, Steps,\nStrides and Wraps. `Follow the online documentation for more info\n\u003chttps://thoth-station.ninja/docs/developers/adviser/index.html#pipeline-units\u003e`__.\n\nPredictor can be switched and the type of predictor can help with the\ndesired resolution process. For recommending high quality software stacks,\nreinforcement learning algorithms, such as `MCTS\n\u003chttps://en.wikipedia.org/wiki/Monte_Carlo_tree_search\u003e`__ or `TD-learning\n\u003chttps://en.wikipedia.org/wiki/Temporal_difference_learning\u003e`__ are used.\n\n**See also:**\n\n* `dev.to: How to beat Python’s pip: Reinforcement learning-based dependency resolution \u003chttps://dev.to/fridex/how-to-beat-python-s-pip-reinforcement-learning-based-dependency-resolution-2he2\u003e`__\n* `YouTube: Reinforcement learning-based dependency resolution \u003chttps://www.youtube.com/watch?v=WEJ65Rvj3lc\u003e`__\n\nResolution process as Markov Decision Process\n=============================================\n\nThe whole resolution process can be modeled as a Markov Decision Process (MDP)\nthus the reinforcement learning (RL) principles stated above can apply.\nRecommending the best possible set of packages than corresponds to solving the\ngiven MDP.  See `Thoth's documentation for more info\n\u003chttps://thoth-station.ninja/docs/developers/adviser/introduction.html\u003e`__.\n\nTo obey terms often used in the reinforcement learning terminology, Predictor\ncan be seen as an agent. Resolver and Software stack resolution pipeline can be\nseen as entities that interact with the environment. Names used in Thoth\nintentionally do not correspond to RL terminology as RL based resolution is\njust one of the possible resolutions that can be implemented (others can be\nhill-climbing, random walks in the dependency graph, ...).\n\n**See also:**\n\n* `dev.to: How to beat Python’s pip: Reinforcement learning-based dependency resolution \u003chttps://dev.to/fridex/how-to-beat-python-s-pip-reinforcement-learning-based-dependency-resolution-2he2\u003e`__\n* `YouTube: Reinforcement learning-based dependency resolution \u003chttps://www.youtube.com/watch?v=WEJ65Rvj3lc\u003e`__\n\nDependency Monkey\n=================\n\nSee `the following article for in-depth explanation with\na video \u003chttps://developers.redhat.com/articles/2021/09/17/resolve-python-dependencies-thoth-dependency-monkey\u003e`__.\n\nThe primary use-case for Dependency Monkey is to generate software stacks that\nare subsequently validated and scored in the `Amun\n\u003chttps://github.com/thoth-station/amun-api\u003e`__ service. Simply, when generating\nall the possible software stacks, we can find the best software stack for an\napplication by validating it in a CI (or Amun in case of Thoth), running the\napplication in the specific runtime environment (e.g. Fedora 33 with installed\nnative packages - RPMs) on some specific hardware configuration. Generating and\nscoring all the possible software stacks is, however, most often not doable in\na reasonable time. For this purpose, Dependency Monkey can create a sample of\nsoftware stacks that can be taken as representatives. These representatives are\nscored and aggregated data are used for predicting the best application stack\n(again, generated and run through CI/Amun to make predictions more accurate by\nlearning over time).\n\nSee `Dependency Monkey documentation\n\u003chttps://thoth-station.ninja/docs/developers/adviser/dependency_monkey.html\u003e`_\nfor more info.\n\n**See also:**\n\n* `Resolve Python dependencies with Thoth Dependency Monkey \u003chttps://developers.redhat.com/articles/2021/09/17/resolve-python-dependencies-thoth-dependency-monkey\u003e`__\n* `Developers Red Hat: AI software stack inspection with Thoth and TensorFlow \u003chttps://developers.redhat.com/blog/2020/09/30/ai-software-stack-inspection-with-thoth-and-tensorflow/?sc_cid=7013a000002gbzfAAA\u003e`__\n* `dev.to: How to beat Python’s pip: Inspecting the quality of machine learning software \u003chttps://dev.to/fridex/how-to-beat-python-s-pip-inspecting-the-quality-of-machine-learning-software-1pkp\u003e`__\n* `YouTube: Thoth Amun API: Inspecting the quality of software \u003chttps://www.youtube.com/watch?v=yeBjnZpdMwY\u003e`__\n* `dev.to: How to beat Python’s pip: Dependency Monkey inspecting the quality of TensorFlow dependencies \u003chttps://dev.to/fridex/how-to-beat-python-s-pip-dependency-monkey-inspecting-the-quality-of-tensorflow-dependencies-6fc\u003e`__\n* `YouTube: Dependency Monkey inspecting Python dependencies of TensorFlow \u003chttps://www.youtube.com/watch?v=S3hFn8KRsKc\u003e`__\n\nAdvises and Recommendations\n===========================\n\nIn Thoth's terminology, advises and recommendations are the same. Based on\nthe aggregated knowledge stored in the database, provide the best application\nstack with reasoning on why the given software stack is used. Pipeline units\npresent in the pipeline configuration score packages resolved and provide such\nreasoning. The reasoning is called \"justification\" in Thoth's terminology. See\nThoth's pages to `see some of them\n\u003chttps://thoth-station.ninja/justifications\u003e`__.\n\nProvenance Checks\n=================\n\nAs Thoth aggregates information about packages available, it can verify\na user's stack against its knowledge base. See `Provenance Checks\n\u003chttps://thoth-station.ninja/docs/developers/adviser/provenance_checks.html\u003e`_\nfor more info.\n\nInstallation and deployment\n===========================\n\nAdviser is built using OpenShift Source-to-Image and deployed\nautomatically with Thoth's deployment available in the\n`thoth-station/thoth-application repository\n\u003chttps://github.com/thoth-station/thoth-application\u003e`__.\n\nIn a Thoth deployment, adviser is run based on requests coming to the `user API\n\u003chttps://github.com/thoth-station/user-api\u003e`__ - each deployed adviser is run\nper a user request. You can run adviser locally as well by installing it and\nusing its command line interface:\n\n::\n\n  pip3 install thoth-adviser\n  thoth-adviser --help\n  # Or use git repo directly for the latest code:\n  # pip3 install git+https://github.com/thoth-station/adviser\n\nNote a database needs to be available.  See `thoth-storages repository\n\u003chttps://github.com/thoth-station/storages\u003e`__ on how to run Thoth's knowledge\ngraph locally and example `notebooks\n\u003chttps://github.com/thoth-station/notebooks\u003e`__ with experiments.\n\nWhen thoth-adviser is scheduled in a deployment, it is actually executed as a\nCLI with arguments passed via environment variables.\n\nBlocking a bogus pipeline unit\n==============================\n\nAdviser also considers environment variable ``THOTH_ADVISER_BLOCKED_UNITS`` that\nstates a comma separated list of pipeline units that should not be added to\nthe pipeline. This can be handy if an issue with a unit arises in a deployment\n- Thoth operator can remove pipeline unit by adjusting environment variable in\nthe adviser deployment manifest and provide this configuration without a need\nto deploy a new version of adviser.\n\nDisabling pipeline unit validation\n==================================\n\nFor prod-like deployments, you can disable pipeline unit validation. By doing\nso, the pipeline unit configuration can be constructed faster. Provide\n``THOTH_ADVISER_VALIDATE_UNIT_CONFIGURATION_SCHEMA=0`` environment variable to\ndisable pipeline unit configuration validation.\n\nRunning adviser locally\n=======================\n\nOften, it is useful to run adviser locally to experiment or verify your changes\nin implementation. You can do so easily by running:\n\n.. code-block:: console\n\n  pipenv install --dev\n  PYTHONPATH=. pipenv run ./thoth-adviser --help\n\nThis command will run adviser locally - adviser will try to connect to a local\nPostgreSQL instance and compute recommendations. `Browse docs here\n\u003chttps://github.com/thoth-station/thoth-storages\u003e`__ to see how to setup a local\nPostgreSQL instance. Also, follow the developer's guide to get `more\ninformation about developer's setup\n\u003chttps://thoth-station.ninja/docs/developers/adviser/developers_guide.html\u003e`__.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthoth-station%2Fadviser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthoth-station%2Fadviser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthoth-station%2Fadviser/lists"}