{"id":15683701,"url":"https://github.com/ma7555/evalify","last_synced_at":"2025-05-07T05:51:02.751Z","repository":{"id":43467398,"uuid":"459380466","full_name":"ma7555/evalify","owner":"ma7555","description":"Evaluate your biometric verification models literally in seconds.","archived":false,"fork":false,"pushed_at":"2024-11-08T23:54:49.000Z","size":3193,"stargazers_count":19,"open_issues_count":1,"forks_count":20,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-05T03:54:07.557Z","etag":null,"topics":["evaluation","evaluation-framework","evaluation-metrics","face-recognition","face-verification","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ma7555.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-02-15T01:06:53.000Z","updated_at":"2024-11-09T00:06:59.000Z","dependencies_parsed_at":"2024-11-16T10:03:11.538Z","dependency_job_id":null,"html_url":"https://github.com/ma7555/evalify","commit_stats":{"total_commits":54,"total_committers":3,"mean_commits":18.0,"dds":"0.11111111111111116","last_synced_commit":"1baacd88562080bb39bb1572a6364fd1cb8ad878"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ma7555%2Fevalify","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ma7555%2Fevalify/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ma7555%2Fevalify/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ma7555%2Fevalify/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ma7555","download_url":"https://codeload.github.com/ma7555/evalify/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252823693,"owners_count":21809709,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation","evaluation-framework","evaluation-metrics","face-recognition","face-verification","python"],"created_at":"2024-10-03T17:08:14.447Z","updated_at":"2025-05-07T05:51:02.723Z","avatar_url":"https://github.com/ma7555.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# evalify\n\n\u003cp align=\"center\"\u003e\n\n\u003cimg src=\"https://user-images.githubusercontent.com/7144929/154332210-fa1fee34-faae-4567-858a-49fa53e99a2b.svg\" width=\"292\" height=\"120\" alt=\"Logo\"/\u003e\n\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\n\u003ca href=\"https://github.com/ma7555/evalify/blob/main/LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/license/ma7555/evalify\"\n        alt = \"License\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://doi.org/10.5281/zenodo.6181723\"\u003e\u003cimg src=\"https://zenodo.org/badge/DOI/10.5281/zenodo.6181723.svg\" alt=\"DOI\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.python.org/downloads/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/python-3.9 | 3.10 | 3.11 | 3.12-blue.svg\"\n        alt = \"Python 3.7 | 3.8 | 3.9 | 3\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://pypi.python.org/pypi/evalify\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/v/evalify.svg\"\n        alt = \"Release Status\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://github.com/ma7555/evalify/actions\"\u003e\n    \u003cimg src=\"https://github.com/ma7555/evalify/actions/workflows/dev.yml/badge.svg?branch=main\" alt=\"CI Status\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://ma7555.github.io/evalify/\"\u003e\n    \u003cimg src=\"https://img.shields.io/website/https/ma7555.github.io/evalify/index.html.svg?label=docs\u0026down_message=unavailable\u0026up_message=available\" alt=\"Documentation Status\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://github.com/astral-sh/ruff\"\u003e\n    \u003cimg src=\"https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json\" alt=\"Code style: Ruff\"\u003e\n\u003c/a\u003e\n\n\u003ca href=\"https://codecov.io/gh/ma7555/evalify\"\u003e\n  \u003cimg src=\"https://codecov.io/gh/ma7555/evalify/branch/main/graph/badge.svg\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/evalify/\"\u003e\u003cimg alt=\"PyPI Downloads/Month\" src=\"https://img.shields.io/pypi/dm/evalify\"\u003e\n\u003c/a\u003e\n\n\u003c/p\u003e\n\n**Evaluate Biometric Authentication Models Literally in Seconds.**\n\n## Installation\n#### Stable release:\n```bash\npip install evalify\n```\n#### Bleeding edge:\n```bash\npip install git+https://github.com/ma7555/evalify.git\n```\n## Used for\nEvaluating all biometric authentication models, where the model output is a high-level embeddings known as feature vectors for visual or behaviour biometrics or d-vectors for auditory biometrics.\n\n## Usage\n\n```python\nimport numpy as np\nfrom evalify import Experiment\n\nrng = np.random.default_rng()\nnphotos = 500\nemb_size = 32\nnclasses = 10\nX = rng.random((self.nphotos, self.emb_size))\ny = rng.integers(self.nclasses, size=self.nphotos)\n\nexperiment = Experiment()\nexperiment.run(X, y)\nexperiment.get_roc_auc()\nprint(experiment.roc_auc)\nprint(experiment.find_threshold_at_fpr(0.01))\n```\n## How it works\n* When you run an experiment, evalify tries all the possible combinations between individuals for authentication based on the `X` and `y` parameters and returns the results including FPR, TPR, FNR, TNR and ROC AUC. `X` is an array of embeddings and `y` is an array of corresponding targets.\n* Evalify can find the optimal threshold based on your agreed FPR and desired similarity or distance metric.\n\n## Documentation: \n* \u003chttps://ma7555.github.io/evalify/\u003e\n\n\n## Features\n\n* Blazing fast implementation for metrics calculation through optimized einstein sum and vectorized calculations.\n* Many operations are dispatched to canonical BLAS, cuBLAS, or other specialized routines.\n* Smart sampling options using direct indexing from pre-calculated arrays with total control over sampling strategy and sampling numbers.\n* Supports most evaluation metrics:\n    - `cosine_similarity`\n    - `pearson_similarity`\n    - `cosine_distance`\n    - `euclidean_distance`\n    - `euclidean_distance_l2`\n    - `minkowski_distance`\n    - `manhattan_distance`\n    - `chebyshev_distance`\n* Computation time for 4 metrics 4.2 million samples experiment is **24 seconds vs 51 minutes** if looping using `scipy.spatial.distance` implemntations.\n\n## TODO\n* Safer memory allocation. I did not have issues but if you ran out of memory please manually set the `batch_size` argument.\n\n## Contribution\n* Contributions are welcomed, and they are greatly appreciated! Every little bit helps, and credit will always be given.\n* Please check [CONTRIBUTING.md](https://github.com/ma7555/evalify/blob/main/CONTRIBUTING.md) for guidelines.\n\n## Citation\n* If you use this software, please cite it using the metadata from [CITATION.cff](https://github.com/ma7555/evalify/blob/main/CITATION.cff)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fma7555%2Fevalify","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fma7555%2Fevalify","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fma7555%2Fevalify/lists"}