{"id":26060477,"url":"https://github.com/tensorzero/llmgym","last_synced_at":"2025-04-11T08:00:16.805Z","repository":{"id":279265365,"uuid":"916815084","full_name":"tensorzero/llmgym","owner":"tensorzero","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-01T23:19:30.000Z","size":1555,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-01T23:30:33.996Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tensorzero.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-14T20:09:02.000Z","updated_at":"2025-03-31T17:56:10.000Z","dependencies_parsed_at":"2025-04-01T23:26:15.487Z","dependency_job_id":null,"html_url":"https://github.com/tensorzero/llmgym","commit_stats":null,"previous_names":["tensorzero/llmgym"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorzero%2Fllmgym","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorzero%2Fllmgym/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorzero%2Fllmgym/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorzero%2Fllmgym/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tensorzero","download_url":"https://codeload.github.com/tensorzero/llmgym/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248360154,"owners_count":21090657,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-08T14:08:28.043Z","updated_at":"2025-04-11T08:00:16.770Z","avatar_url":"https://github.com/tensorzero.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e [!IMPORTANT]\n\u003e\n\u003e **This repository is still under active development. Expect breaking changes.**\n\n# LLM Gym\n\nLLM Gym is a unified environment interface for developing and benchmarking LLM applications that learn from feedback. Think [gym](https://gymnasium.farama.org/) for LLM agents.\n\nAs the space of benchmarks rapidly grows, fair and comprehensive comparisons are getting trickier, so we aim to make that easier for you. The vision is an intuitive interface for a suite of environments you can seamlessly swap out for research and development purposes.\n\n## Installation\n\nFollow these steps to set up the development environment for LLM Gym using uv for virtual environment management and Hatch (with Hatchling) for building and packaging.\n\n### Prerequisites\n\n- Python 3.10 (or a compatible version, e.g., \u003e=3.10, \u003c4.0)\n- [uv](https://docs.astral.sh/uv/getting-started/installation/) – an extremely fast Python package manager and virtual environment tool\n\n### Steps\n\n#### 1. Clone the Repository\nClone the repository to your local machine:\n```bash\ngit clone git@github.com:tensorzero/gym-scratchpad.git\ncd llmgym\n```\n\n#### 2. Create and Activate a Virtual Environment\nUse uv to create a virtual environment. This command will create a new environment (by default in the .venv directory) using Python 3.10:\n```bash\nuv venv --python 3.10\n```\nActivate the virtual environment:\n```bash\nsource .venv/bin/activate\n```\n\n#### 3. Install Project Dependencies\nInstall the project in editable mode along with its development dependencies:\n```bash\nuv pip install -e .\n```\n\n#### 4. Verify the Installation\nTo ensure everything is set up correctly, you can run the tests or simply import the package in Python.\n\nRun tests:\n```bash\nuv run pytest\n```\n\nImport the package in Python:\n```bash\npython\n\u003e\u003e\u003e import llmgym\n\u003e\u003e\u003e llmgym.__version__\n'0.0.0'\n```\n\n## Setting Environment Variables\n\nTo set the `OPENAI_API_KEY` environment variable, run the following command:\n```bash\nexport OPENAI_API_KEY=\"your_openai_api_key\"\n```\n\nWe recommend using [direnv](https://direnv.net/) and creating a local `.envrc` file to manage environment variables. For example, the `.envrc` file might look like this:\n```bash\nexport OPENAI_API_KEY=\"your_openai_api_key\"\n```\n\nand then run `direnv allow` to load the environment variables.\n\n## Quickstart\n\nStart ipython with async support.\n```bash\nipython --async=True\n```\nRun an episode of the 21-questions environment.\n```python\nimport logging\n\nimport llmgym\nfrom llmgym.logs import get_logger\nfrom llmgym.agents import OpenAIAgent\n\nlogger = get_logger(\"llmgym\")\nlogger.setLevel(logging.INFO)\n\nenv  = llmgym.make(\"21_questions_v0\")\n\nagent = llmgym.agents.OpenAIAgent(\n    model_name=\"gpt-4o-mini\",\n    function_configs=env.functions,\n    tool_configs=env.tools,\n)\n# Get default horizon\nmax_steps = env.horizon\n\n# Reset the environment\nreset_data = await env.reset()\nobs = reset_data.observation\n\n# Run the episode\nfor _step in range(max_steps):\n    # Get action from agent\n    action = await agent.act(obs)\n\n    # Step the environment\n    step_data = await env.step(action)\n    obs = step_data.observation\n\n    # Check if the episode is done\n    done = step_data.terminated or step_data.truncated\n    if done:\n        break\nenv.close()\n```\n\nThis can also be run in the [Quickstart Notebook](examples/quickstart.ipynb).\n\n\n## Tutorial\n\nFor a full tutorial, see the [Tutorial Notebook](examples/tutorial.ipynb).\n\nTo see how to run multiple episodes concurrently, see the [Tau Bench](examples/tau_bench.ipynb) or [21 Questions](examples/21_questions.ipynb) notebooks.\n\nFor a supervised finetuning example, see the [Supervised Finetuning Notebook](examples/supervised_fine_tuning.ipynb).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorzero%2Fllmgym","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftensorzero%2Fllmgym","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorzero%2Fllmgym/lists"}