{"id":13645096,"url":"https://github.com/blackhc/llm-strategy","last_synced_at":"2025-04-04T16:13:47.266Z","repository":{"id":65571262,"uuid":"580595594","full_name":"BlackHC/llm-strategy","owner":"BlackHC","description":"Directly Connecting Python to LLMs via Strongly-Typed Functions, Dataclasses, Interfaces \u0026 Generic Types","archived":false,"fork":false,"pushed_at":"2024-03-05T13:07:53.000Z","size":1137,"stargazers_count":377,"open_issues_count":5,"forks_count":21,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-05-19T14:34:36.760Z","etag":null,"topics":["gpt","langchain","llm","openai","pydantic","python","strongly-typed"],"latest_commit_sha":null,"homepage":"https://blackhc.github.io/llm-strategy/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BlackHC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-21T00:23:47.000Z","updated_at":"2024-08-01T16:37:24.302Z","dependencies_parsed_at":"2023-11-11T19:30:53.082Z","dependency_job_id":"62dcbc3f-82c6-4e58-a964-62decf10218f","html_url":"https://github.com/BlackHC/llm-strategy","commit_stats":{"total_commits":23,"total_committers":1,"mean_commits":23.0,"dds":0.0,"last_synced_commit":"5381b934ae76e4b30ab749323b9c5dc3bf982b0b"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackHC%2Fllm-strategy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackHC%2Fllm-strategy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackHC%2Fllm-strategy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackHC%2Fllm-strategy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BlackHC","download_url":"https://codeload.github.com/BlackHC/llm-strategy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208140,"owners_count":20901570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpt","langchain","llm","openai","pydantic","python","strongly-typed"],"created_at":"2024-08-02T01:02:26.834Z","updated_at":"2025-04-04T16:13:47.246Z","avatar_url":"https://github.com/BlackHC.png","language":"Python","funding_links":[],"categories":["Langchain"],"sub_categories":[],"readme":"# llm-strategy\n\n[![Release](https://img.shields.io/github/v/release/blackhc/llm-strategy)](https://img.shields.io/github/v/release/blackhc/llm-strategy)\n[![Build status](https://img.shields.io/github/actions/workflow/status/blackhc/llm-strategy/main.yml?branch=main)](https://github.com/blackhc/llm-strategy/actions/workflows/main.yml?query=branch%3Amain)\n[![codecov](https://codecov.io/gh/blackhc/llm-strategy/branch/main/graph/badge.svg)](https://codecov.io/gh/blackhc/llm-strategy)\n[![Commit activity](https://img.shields.io/github/commit-activity/m/blackhc/llm-strategy)](https://img.shields.io/github/commit-activity/m/blackhc/llm-strategy)\n[![License](https://img.shields.io/github/license/blackhc/llm-strategy)](https://img.shields.io/github/license/blackhc/llm-strategy)\n\nImplementing the Strategy Pattern using LLMs.\n\nAlso, please see https://blog.blackhc.net/2022/12/llm_software_engineering/ for a wider perspective on why this could be important in the future.\n\nThis package adds a decorator `llm_strategy` that connects to an LLM (such as OpenAI’s GPT-3) and uses the LLM to \"implement\" abstract methods in interface classes. It does this by forwarding requests to the LLM and converting the responses back to Python data using Python's `@dataclasses`.\n\nIt uses the doc strings, type annotations, and method/function names as prompts for the LLM, and can automatically convert the results back into Python types (currently only supporting `@dataclasses`). It can also extract a data schema to send to the LLM for interpretation. While the `llm-strategy` package still relies on some Python code, it has the potential to reduce the need for this code in the future by using additional, cheaper LLMs to automate the parsing of structured data.\n\n- **Github repository**: \u003chttps://github.com/blackhc/llm-strategy/\u003e\n- **Documentation** \u003chttps://blackhc.github.io/llm-strategy/\u003e\n\n## Research Example\n\nThe latest version also includes a package for hyperparameter tracking and collecting traces from LLMs.\n\nThis for example allows for meta optimization. See examples/research for a simple implementation using Generics.\n\nYou can find an example WandB trace at: https://wandb.ai/blackhc/blackboard-pagi/reports/Meta-Optimization-Example-Trace--Vmlldzo3MDMxODEz?accessToken=p9hubfskmq1z5yj1uz7wx1idh304diiernp7pjlrjrybpaozlwv3dnitjt7vni1j\n\nThe prompts showing off the pattern using Generics are straightforward:\n```python\nT_TaskParameters = TypeVar(\"T_TaskParameters\")\nT_TaskResults = TypeVar(\"T_TaskResults\")\nT_Hyperparameters = TypeVar(\"T_Hyperparameters\")\n\n\nclass TaskRun(BaseModel, Generic[T_TaskParameters, T_TaskResults, T_Hyperparameters]):\n    \"\"\"\n    The task run. This is the 'data' we use to optimize the hyperparameters.\n    \"\"\"\n\n    task_parameters: T_TaskParameters = Field(..., description=\"The task parameters.\")\n    hyperparameters: T_Hyperparameters = Field(\n        ...,\n        description=\"The hyperparameters used for the task. We optimize these.\",\n    )\n    all_chat_chains: dict = Field(..., description=\"The chat chains from the task execution.\")\n    return_value: T_TaskResults | None = Field(\n        ..., description=\"The results of the task. (None for exceptions/failure.)\"\n    )\n    exception: list[str] | str | None = Field(..., description=\"Exception that occurred during the task execution.\")\n\n\nclass TaskReflection(BaseModel):\n    \"\"\"\n    The reflections on the task.\n\n    This contains the lessons we learn from each task run to come up with better\n    hyperparameters to try.\n    \"\"\"\n\n    feedback: str = Field(\n        ...,\n        description=(\n            \"Only look at the final results field. Does its content satisfy the \"\n            \"task description and task parameters? Does it contain all the relevant \"\n            \"information from the all_chains and all_prompts fields? What could be improved \"\n            \"in the results?\"\n        ),\n    )\n    evaluation: str = Field(\n        ...,\n        description=(\n            \"The evaluation of the outputs given the task. Is the output satisfying? What is wrong? What is missing?\"\n        ),\n    )\n    hyperparameter_suggestion: str = Field(\n        ...,\n        description=\"How we want to change the hyperparameters to improve the results. What could we try to change?\",\n    )\n    hyperparameter_missing: str = Field(\n        ...,\n        description=(\n            \"What hyperparameters are missing to improve the results? What could \"\n            \"be changed that is not exposed via hyperparameters?\"\n        ),\n    )\n\n\nclass TaskInfo(BaseModel, Generic[T_TaskParameters, T_TaskResults, T_Hyperparameters]):\n    \"\"\"\n    The task run and the reflection on the experiment.\n    \"\"\"\n\n    task_parameters: T_TaskParameters = Field(..., description=\"The task parameters.\")\n    hyperparameters: T_Hyperparameters = Field(\n        ...,\n        description=\"The hyperparameters used for the task. We optimize these.\",\n    )\n    reflection: TaskReflection = Field(..., description=\"The reflection on the task.\")\n\n\nclass OptimizationInfo(BaseModel, Generic[T_TaskParameters, T_TaskResults, T_Hyperparameters]):\n    \"\"\"\n    The optimization information. This is the data we use to optimize the\n    hyperparameters.\n    \"\"\"\n\n    older_task_summary: str | None = Field(\n        None,\n        description=(\n            \"A summary of previous experiments and the proposed changes with \"\n            \"the goal of avoiding trying the same changes repeatedly.\"\n        ),\n    )\n    task_infos: list[TaskInfo[T_TaskParameters, T_TaskResults, T_Hyperparameters]] = Field(\n        ..., description=\"The most recent tasks we have run and our reflections on them.\"\n    )\n    best_hyperparameters: T_Hyperparameters = Field(..., description=\"The best hyperparameters we have found so far.\")\n\n\nclass OptimizationStep(BaseModel, Generic[T_TaskParameters, T_TaskResults, T_Hyperparameters]):\n    \"\"\"\n    The next optimization steps. New hyperparameters we want to try experiments and new\n    task parameters we want to evaluate on given the previous experiments.\n    \"\"\"\n\n    best_hyperparameters: T_Hyperparameters = Field(\n        ...,\n        description=\"The best hyperparameters we have found so far given task_infos and history.\",\n    )\n    suggestion: str = Field(\n        ...,\n        description=(\n            \"The suggestions for the next experiments. What could we try to \"\n            \"change? We will try several tasks next and several sets of hyperparameters. \"\n            \"Let's think step by step.\"\n        ),\n    )\n    task_parameters_suggestions: list[T_TaskParameters] = Field(\n        ...,\n        description=\"The task parameters we want to try next.\",\n        hint_min_items=1,\n        hint_max_items=4,\n    )\n    hyperparameter_suggestions: list[T_Hyperparameters] = Field(\n        ...,\n        description=\"The hyperparameters we want to try next.\",\n        hint_min_items=1,\n        hint_max_items=2,\n    )\n\n\nclass ImprovementProbability(BaseModel):\n    considerations: list[str] = Field(..., description=\"The considerations for potential improvements.\")\n    probability: float = Field(..., description=\"The probability of improvement.\")\n\n\nclass LLMOptimizer:\n    @llm_explicit_function\n    @staticmethod\n    def reflect_on_task_run(\n        language_model,\n        task_run: TaskRun[T_TaskParameters, T_TaskResults, T_Hyperparameters],\n    ) -\u003e TaskReflection:\n        \"\"\"\n        Reflect on the results given the task parameters and hyperparameters.\n\n        This contains the lessons we learn from each task run to come up with better\n        hyperparameters to try.\n        \"\"\"\n        raise NotImplementedError()\n\n    @llm_explicit_function\n    @staticmethod\n    def summarize_optimization_info(\n        language_model,\n        optimization_info: OptimizationInfo[T_TaskParameters, T_TaskResults, T_Hyperparameters],\n    ) -\u003e str:\n        \"\"\"\n        Summarize the optimization info. We want to preserve all relevant knowledge for\n        improving the hyperparameters in the future. All information from previous\n        experiments will be forgotten except for what this summary.\n        \"\"\"\n        raise NotImplementedError()\n\n    @llm_explicit_function\n    @staticmethod\n    def suggest_next_optimization_step(\n        language_model,\n        optimization_info: OptimizationInfo[T_TaskParameters, T_TaskResults, T_Hyperparameters],\n    ) -\u003e OptimizationStep[T_TaskParameters, T_TaskResults, T_Hyperparameters]:\n        \"\"\"\n        Suggest the next optimization step.\n        \"\"\"\n        raise NotImplementedError()\n\n    @llm_explicit_function\n    @staticmethod\n    def probability_for_improvement(\n        language_model,\n        optimization_info: OptimizationInfo[T_TaskParameters, T_TaskResults, T_Hyperparameters],\n    ) -\u003e ImprovementProbability:\n        \"\"\"\n        Return the probability for improvement (between 0 and 1).\n\n        This is your confidence that your next optimization steps will improve the\n        hyperparameters given the information provided. If you think that the\n        information available is unlikely to lead to better hyperparameters, return 0.\n        If you think that the information available is very likely to lead to better\n        hyperparameters, return 1. Be concise.\n        \"\"\"\n        raise NotImplementedError()\n```\n\n## Application Example\n\n```python\nfrom dataclasses import dataclass\nfrom llm_strategy import llm_strategy\nfrom langchain.llms import OpenAI\n\n\n@llm_strategy(OpenAI(max_tokens=256))\n@dataclass\nclass Customer:\n    key: str\n    first_name: str\n    last_name: str\n    birthdate: str\n    address: str\n\n    @property\n    def age(self) -\u003e int:\n        \"\"\"Return the current age of the customer.\n\n        This is a computed property based on `birthdate` and the current year (2022).\n        \"\"\"\n\n        raise NotImplementedError()\n\n\n@dataclass\nclass CustomerDatabase:\n    customers: list[Customer]\n\n    def find_customer_key(self, query: str) -\u003e list[str]:\n        \"\"\"Find the keys of the customers that match a natural language query best (sorted by closeness to the match).\n\n        We support semantic queries instead of SQL, so we can search for things like\n        \"the customer that was born in 1990\".\n\n        Args:\n            query: Natural language query\n\n        Returns:\n            The index of the best matching customer in the database.\n        \"\"\"\n        raise NotImplementedError()\n\n    def load(self):\n        \"\"\"Load the customer database from a file.\"\"\"\n        raise NotImplementedError()\n\n    def store(self):\n        \"\"\"Store the customer database to a file.\"\"\"\n        raise NotImplementedError()\n\n\n@llm_strategy(OpenAI(max_tokens=1024))\n@dataclass\nclass MockCustomerDatabase(CustomerDatabase):\n    def load(self):\n        self.customers = self.create_mock_customers(10)\n\n    def store(self):\n        pass\n\n    @staticmethod\n    def create_mock_customers(num_customers: int = 1) -\u003e list[Customer]:\n        \"\"\"\n        Create mock customers with believable data (our customers are world citizens).\n        \"\"\"\n        raise NotImplementedError()\n```\n\nSee [examples/mock_app/customer_database_search.py](examples/mock_app/customer_database_search.py) for a full example.\n\n![Customer Database Viewer](examples/mock_app/app.svg)\n\n![Searching for a Customer](examples/mock_app/search1.svg)\n\n![Searching for a Customer](examples/mock_app/search2.svg)\n\n## Getting started with contributing\n\nClone the repository first. Then, install the environment and the pre-commit hooks with \n\n```bash\nmake install\n```\n\nThe CI/CD\npipeline will be triggered when you open a pull request, merge to main,\nor when you create a new release.\n\nTo finalize the set-up for publishing to PyPi or Artifactory, see\n[here](https://fpgmaas.github.io/cookiecutter-poetry/features/publishing/#set-up-for-pypi).\nFor activating the automatic documentation with MkDocs, see\n[here](https://fpgmaas.github.io/cookiecutter-poetry/features/mkdocs/#enabling-the-documentation-on-github).\nTo enable the code coverage reports, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/codecov/).\n\n## Releasing a new version\n\n- Create an API Token on [Pypi](https://pypi.org/).\n- Add the API Token to your projects secrets with the name `PYPI_TOKEN` by visiting \n[this page](https://github.com/blackhc/llm-strategy/settings/secrets/mock_app/actions/new).\n- Create a [new release](https://github.com/blackhc/llm-strategy/releases/new) on Github. \nCreate a new tag in the form ``*.*.*``.\n\nFor more details, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/cicd/#how-to-trigger-a-release).\n\n---\n\nRepository initiated with [fpgmaas/cookiecutter-poetry](https://github.com/fpgmaas/cookiecutter-poetry).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackhc%2Fllm-strategy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblackhc%2Fllm-strategy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackhc%2Fllm-strategy/lists"}