{"id":29035890,"url":"https://github.com/aleph-alpha/pharia-studio-sdk","last_synced_at":"2025-08-10T04:37:45.365Z","repository":{"id":300992993,"uuid":"1000068631","full_name":"Aleph-Alpha/pharia-studio-sdk","owner":"Aleph-Alpha","description":null,"archived":false,"fork":false,"pushed_at":"2025-08-05T10:54:54.000Z","size":2429,"stargazers_count":0,"open_issues_count":6,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-05T11:26:25.987Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://pharia-studio-sdk.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aleph-Alpha.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-11T08:09:49.000Z","updated_at":"2025-08-05T09:16:38.000Z","dependencies_parsed_at":"2025-08-05T11:15:56.780Z","dependency_job_id":"bbc01eaa-c32c-433e-a26f-c31b94d8569e","html_url":"https://github.com/Aleph-Alpha/pharia-studio-sdk","commit_stats":null,"previous_names":["aleph-alpha/pharia-studio-sdk"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/Aleph-Alpha/pharia-studio-sdk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fpharia-studio-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fpharia-studio-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fpharia-studio-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fpharia-studio-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aleph-Alpha","download_url":"https://codeload.github.com/Aleph-Alpha/pharia-studio-sdk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fpharia-studio-sdk/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269677473,"owners_count":24457853,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-26T12:36:19.605Z","updated_at":"2025-08-10T04:37:45.340Z","avatar_url":"https://github.com/Aleph-Alpha.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pharia-studio-sdk\n\nFormerly the `intelligence_layer/evaluation` package.\n\n## Overview\n\nThe pharia-studio-sdk provides a set of tools for evaluating and benchmarking LLMs with the Studio applications.\nYou can access the documentation on [Read the Docs](https://pharia-studio-sdk.readthedocs.io/en/latest/).\n\n## Installation\nThe SDK is published on [PyPI](https://pypi.org/project/pharia-studio-sdk/).\n\nTo add the SDK as a dependency to an existing project managed, run\n```bash\npip install pharia-studio-sdk\n```\n\n## Example Usage\nThis example demonstrates how to define, run, and evaluate a custom task using the pharia-studio-sdk.\nIt serves as a reference point for creating your own evaluation logic with Pharia Studio.\nBefore running the example, make sure to set up the necessary configuration values: `STUDIO_URL`, `INFERENCE_URL`, and `AA_TOKEN`.\n\n```python\nfrom statistics import mean\nfrom typing import Iterable\n\nfrom aleph_alpha_client import Client\nfrom pharia_inference_sdk.core import (\n    CompleteInput,\n    ControlModel,\n    Llama3InstructModel,\n    Task,\n    TaskSpan,\n)\nfrom pydantic import BaseModel\n\nfrom pharia_studio_sdk import StudioClient\nfrom pharia_studio_sdk.evaluation import (\n    AggregationLogic,\n    Example,\n    SingleOutputEvaluationLogic,\n    StudioBenchmarkRepository,\n    StudioDatasetRepository,\n)\n\n# Studio Configuration\nPROJECT_NAME = \"My first project\"\nBENCHMARK_NAME = \"My first benchmark\"\nSTUDIO_URL = \"\u003cstudio_url\u003e\"\n\n# Inference Configuration\nINFERENCE_URL = \"\u003cinference_url\u003e\"\nAA_TOKEN = \"\u003caa_token\u003e\"\n\n# Define a task with the `pharia-inference-sdk` module\nclass ExtractIngredientsTaskInput(BaseModel):\n    recipe: str\n\nclass ExtractIngredientsTaskOutput(BaseModel):\n    ingredients: list[str]\n\nclass ExtractIngredientsTask(Task[ExtractIngredientsTaskInput, ExtractIngredientsTaskOutput]):\n    PROMPT_TEMPLATE = \"\"\"Given the following recipe, extract the list of ingredients. Write one ingredient per line, do not add the quantity, do not add any text besides the list.\n\nRecipe:\n{recipe}\n\"\"\"    \n    def __init__(self, model: ControlModel):\n        self._model = model\n\n\n    def do_run(\n            self, input: ExtractIngredientsTaskInput, task_span: TaskSpan\n        ) -\u003e ExtractIngredientsTaskOutput:\n            \n            prompt = self._model.to_instruct_prompt(\n                self.PROMPT_TEMPLATE.format(recipe=input.recipe)\n            )\n\n            completion_input = CompleteInput(\n                 prompt=prompt,\n                 model=self._model)\n            completion = self._model.complete(completion_input, tracer=task_span)\n            ingredients = completion.completions[0].completion.split('\\n')\n            return ExtractIngredientsTaskOutput(ingredients=ingredients)\n\n# Define a logic for evaluation\nclass IngredientsEvaluation(BaseModel):\n    correct_number_of_ingredients: bool\n\nclass IngredientsAggregatedEvaluation(BaseModel):\n    avg_result: float\n\nclass IngredientsEvaluationLogic(\n    SingleOutputEvaluationLogic[\n        ExtractIngredientsTaskInput,\n        ExtractIngredientsTaskOutput,\n        ExtractIngredientsTaskOutput,\n        IngredientsEvaluation,\n    ]\n):\n    def do_evaluate_single_output(\n        self,\n        example: Example[ExtractIngredientsTaskInput, ExtractIngredientsTaskOutput],\n        output: ExtractIngredientsTaskOutput,\n    ) -\u003e IngredientsEvaluation:\n        return IngredientsEvaluation(\n            correct_number_of_ingredients=len(output.ingredients) == len(example.expected_output.ingredients),\n        )\nclass IngredientsAggregationLogic(\n    AggregationLogic[IngredientsEvaluation, IngredientsAggregatedEvaluation]\n):\n    def aggregate(\n        self, evaluations: Iterable[IngredientsEvaluation]\n    ) -\u003e IngredientsAggregatedEvaluation:\n        evaluation_list = list(evaluations)\n        return IngredientsAggregatedEvaluation(\n            avg_result=mean(evaluation.correct_number_of_ingredients for evaluation in evaluation_list)\n        )\n\naa_client = Client(token=AA_TOKEN, host=INFERENCE_URL)\nstudio_client = StudioClient(PROJECT_NAME,studio_url=STUDIO_URL, auth_token=AA_TOKEN, create_project=True)\nstudio_benchmark_repository = StudioBenchmarkRepository(studio_client=studio_client)\nstudio_dataset_repository = StudioDatasetRepository(studio_client=studio_client)\n\nevaluation_logic = IngredientsEvaluationLogic()\naggregation_logic = IngredientsAggregationLogic()\n\n# Create a dataset with example inputs and expected outputs\nexamples = [   Example(\n        input=ExtractIngredientsTaskInput(\n            recipe=\"\"\"# Pike Burger\n\n- Pike (the bigger the better)\n- Breadcrumbs\n- Onions\n- Garlic \n- Eggs\n- Mustard\n- Flour\n- Spices\n- Salt\n- Pepper\n- Oil\n\n1. Fish pike\n2. Filet fish into pieces\n3. Grind the fish and onions into a paste\n4. Mix paste with all other ingredients beside the flour and oil, add breadcrumbs until the consistency is right for forming patties\n5. Form the paste into patties and coat them in flour\n6. Let them rest in the fridge for 30min to firm up\n7. Shallow fry them in a pan with a lot of oil\n\"\"\"\n        ),\n        expected_output=ExtractIngredientsTaskOutput(\n            ingredients=[\n               \"Pike\", \"Breadcrumbs\", \"Onions\", \"Garlic\", \"Eggs\", \"Mustard\", \"Flour\", \"Spices\", \"Salt\", \"Pepper\", \"Oil\"\n            ]\n        ),\n    )]\ndataset = studio_dataset_repository.create_dataset(\n    examples=examples,\n    dataset_name=\"My first dataset\",\n    metadata={\"description\": \"dataset_description\"},\n)\n\nmodel = Llama3InstructModel(name=\"llama-3.1-8b-instruct\", client=aa_client)\ntask = ExtractIngredientsTask(model=model)\n# Create and run a benchmark\nbenchmark = studio_benchmark_repository.create_benchmark(\n    dataset_id=dataset.id,\n    eval_logic=evaluation_logic,\n    aggregation_logic=aggregation_logic,\n    name=BENCHMARK_NAME,\n    metadata={\"key\": \"value\"},\n)\nbenchmark.execute(\n    task=task,\n    name=BENCHMARK_NAME,\n)\n\n# Go see the benchmark result in PhariaStudio\n```\n\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](https://github.com/Aleph-Alpha/pharia-studio-sdk/blob/main/CONTRIBUTING.md) for details on how to set up the development environment and submit changes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fpharia-studio-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faleph-alpha%2Fpharia-studio-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fpharia-studio-sdk/lists"}