{"id":20172994,"url":"https://github.com/humansignal/label-studio-evalme","last_synced_at":"2025-04-10T03:16:04.193Z","repository":{"id":37923636,"uuid":"221198400","full_name":"HumanSignal/label-studio-evalme","owner":"HumanSignal","description":"Evaluation metrics package","archived":false,"fork":false,"pushed_at":"2025-04-09T09:19:04.000Z","size":374,"stargazers_count":8,"open_issues_count":4,"forks_count":7,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-10T03:15:41.288Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HumanSignal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-12T11:09:42.000Z","updated_at":"2025-04-03T18:23:55.000Z","dependencies_parsed_at":"2023-10-16T23:21:03.034Z","dependency_job_id":"0e1f3216-a59b-4134-b411-471f070d3936","html_url":"https://github.com/HumanSignal/label-studio-evalme","commit_stats":null,"previous_names":["humansignal/label-studio-evalme","heartexlabs/label-studio-evalme"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanSignal%2Flabel-studio-evalme","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanSignal%2Flabel-studio-evalme/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanSignal%2Flabel-studio-evalme/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanSignal%2Flabel-studio-evalme/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HumanSignal","download_url":"https://codeload.github.com/HumanSignal/label-studio-evalme/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248148245,"owners_count":21055548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T01:33:04.893Z","updated_at":"2025-04-10T03:16:04.165Z","avatar_url":"https://github.com/HumanSignal.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# label-studio-evalme\nEvaluation metrics package\n\n## Installation\n\nSimple installation from PyPI\n```bash\npip install label-studio-evalme\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eOther installation methods\u003c/summary\u003e\n  \n\tPip from source\n\t```bash\n\t# with git\n\tpip install git+https://github.com/heartexlabs/label-studio-evalme.git@master\n\t```\n\n\u003c/details\u003e\n\n## What is Evalme?\nEvalme is a collection of Label Studio evaluation metric implementations and an easy-to-use API to create custom metrics. It offers:\n\n* A standardized interface to increase reproducibility\n* Reduced boilerplate\n* Optimized metrics for Label Studio\n\n## Get started with Evalme\nYou can use Evalme with any Label Studio versions or with Label Studio Enterprise.\n\n### Load existing data from Label Studio\n\nUse the Label Studio REST API to load existing data from your instance of Label Studio or Label Studio Enterprise.\n\nSpecify your Label Studio URL, access token and project ID in the parameters:\n``` python\nfrom evalme.matcher import Matcher\n\nloader = Matcher(url=\"http://127.0.0.1:8000\",\n                 token=\"ACCESS_TOKEN\",\n                 project='1')\nloader.refresh()\n```\n\nYou can also load data from exported annotation files from Label Studio, exported using [the API](https://labelstud.io/guide/api.html#Export-annotations) or the [Label Studio UI](https://labelstud.io/guide/export.html):\n``` python\nfrom evalme.matcher import Matcher\n\nloader = Matcher()\nloader.load('your_filename')\n```\n\nAfter you load data, it is available in the `_raw_data` field. \n\n### Built-in metrics\n\nBy default there is a naive metric object. It evaluates annotation differences with a naive approach:\nif an object is fully equal to another one, the evaluation method returns 1,\notherwise it returns 0.\n\nTo use the built-in metrics, do the following:\n\n``` python\nfrom evalme.matcher import Matcher\n\nloader = Matcher()\nloader.load('your_filename')\n# Run agreement_matrix method to get matrix for all your annotations\nmatrix = loader.agreement_matrix()\n# print result\nprint(matrix)\n```\n\n### Implement your own metric\n\nYou can implement your own metric by creating an evaluation function and registering it in Metrics class. \n\nFor example, create an evaluation function with 2 parameters for compared objects:\n\n```python\nfrom evalme.matcher import Matcher\n# write your own evaluation function or use existing one\ndef naive(x, y):\n\t\"\"\"\n    Naive comparison of annotations\n    \"\"\"\n    if len(x) != len(y):\n        result = 0\n    else:\n        for i in range(len(x)):\n            if x[i]['value'] != y[i]['value']:\n                result = 0\n                break\n        else:\n            result = 1\n    return result\n# Register it in Metrics object\nMetrics.register(\n    name='naive',\n    form=None,\n    tag='all',\n    func=naive,\n    desc='Naive comparison of result dict'\n)\n# create Matcher object from previous example\nloader = Matcher()\nloader.load('your_filename')\nmatrix = loader.agreement_matrix(metric_name='naive')\n# print result\nprint(matrix)\n```\n\n## Contribute!\nThe Label Studio team is hard at work adding even more metrics, but we're looking for incredible contributors like you to submit new metrics and improve existing ones!\n\nJoin our [Slack community](https://slack.labelstud.io) to get help becoming a contributor!\n\n## Community\nFor help or questions, join our huge community on [Slack](https://slack.labelstud.io)!\n\n## License\nPlease observe the MIT License that is listed in this repository. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumansignal%2Flabel-studio-evalme","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhumansignal%2Flabel-studio-evalme","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumansignal%2Flabel-studio-evalme/lists"}