{"id":21053892,"url":"https://github.com/vinid/quica","last_synced_at":"2025-05-15T22:33:52.862Z","repository":{"id":56413864,"uuid":"310316819","full_name":"vinid/quica","owner":"vinid","description":"quica is a tool to run inter coder agreement pipelines in an easy and effective ways. Multiple measures are run and results are collected in a single table than can be easily exported in Latex","archived":false,"fork":false,"pushed_at":"2020-11-09T18:29:13.000Z","size":115,"stargazers_count":23,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-03T15:52:37.886Z","etag":null,"topics":["evaluation-framework","evaluation-metrics","inter-coder-agreement","inter-rater-agreement","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vinid.png","metadata":{"files":{"readme":"README.rst","changelog":"HISTORY.rst","contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-05T14:02:29.000Z","updated_at":"2023-08-27T22:38:45.000Z","dependencies_parsed_at":"2022-08-15T18:20:09.035Z","dependency_job_id":null,"html_url":"https://github.com/vinid/quica","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinid%2Fquica","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinid%2Fquica/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinid%2Fquica/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinid%2Fquica/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vinid","download_url":"https://codeload.github.com/vinid/quica/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254433360,"owners_count":22070464,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation-framework","evaluation-metrics","inter-coder-agreement","inter-rater-agreement","python"],"created_at":"2024-11-19T16:09:44.744Z","updated_at":"2025-05-15T22:33:47.840Z","avatar_url":"https://github.com/vinid.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"======================================\nQuick Inter Coder Agreement in Python\n======================================\n\n\nQuica (Quick Inter Coder Agreement in Python) is a tool to run inter coder agreement pipelines in an easy and effective way.\nMultiple measures are run and results are collected in a single table than can be easily exported in Latex.\nquica supports binary or multiple coders.\n\n.. image:: https://raw.githubusercontent.com/vinid/quica/master/logo.svg\n\n.. image:: https://img.shields.io/pypi/v/quica.svg\n        :target: https://pypi.python.org/pypi/quica\n\n.. image:: https://github.com/vinid/quica/workflows/Python%20package/badge.svg\n        :target: https://github.com/vinid/quica/actions\n\n.. image:: https://readthedocs.org/projects/quica/badge/?version=latest\n        :target: https://quica.readthedocs.io/en/latest/?badge=latest\n        :alt: Documentation Status\n\n.. image:: https://img.shields.io/badge/License-MIT-blue.svg\n        :target: https://lbesson.mit-license.org/\n        :alt: License\n\nQuick Inter Coder Agreement in Python\n\n\n* Free software: MIT license\n* Documentation: https://quica.readthedocs.io.\n\nInstallation\n------------\n\n.. code-block:: bash\n\n    pip install -U quica\n\nJump start Tutorial\n-------------------\n\n.. |colab1| image:: https://colab.research.google.com/assets/colab-badge.svg\n    :target: https://colab.research.google.com/drive/14x6OOQ09Ojn54mNH8JY83bztrJ61yg0o?usp=sharing\n    :alt: Open In Colab\n\n\n+----------------------------------------------------------------+--------------------+\n| Name                                                           | Link               |\n+================================================================+====================+\n| Different possible usages of QUICA                             | |colab1|           |\n+----------------------------------------------------------------+--------------------+\n\nGet Quick Agreement\n-------------------\n\nIf you already have a python dataframe you can run Quica with few liens of code! Let's assume you have two\ncoders; we will create a pandas dataframe just to show how to use the library. As for now, we support only integer values\nand we still have not included weighting.\n\n.. code-block:: python\n\n    from quica.quica import Quica\n    import pandas as pd\n\n    coder_1 = [0, 1, 0, 1, 0, 1]\n    coder_3 = [0, 1, 0, 1, 0, 0]\n\n    dataframe = pd.DataFrame({\"coder1\" : coder_1,\n                  \"coder3\" : coder_3})\n\n    quica = Quica(dataframe=dataframe)\n    print(quica.get_results())\n\nThis is the expected output:\n\n.. code-block:: python\n\n    Out[1]:\n                 score\n    names\n    krippendorff  0.685714\n    fleiss        0.666667\n    scotts        0.657143\n    raw           0.833333\n    mace          0.426531\n    cohen         0.666667\n\nIt was pretty easy to get all the scores, right? What if we do not have a pandas dataframe? what if we want to directly get\nthe latex table to put into the paper? worry not, my friend: it's easier done than said!\n\n.. code-block:: python\n\n    from quica.measures.irr import *\n    from quica.dataset.dataset import IRRDataset\n    from quica.quica import Quica\n\n    coder_1 = [0, 1, 0, 1, 0, 1]\n    coder_3 = [0, 1, 0, 1, 0, 0]\n\n    disagreeing_coders = [coder_1, coder_3]\n    disagreeing_dataset = IRRDataset(disagreeing_coders)\n\n    quica = Quica(disagreeing_dataset)\n\n    print(quica.get_results())\n    print(quica.get_latex())\n\nyou should get this in output, note that the latex table requires the booktabs package:\n\n.. code-block:: python\n\n\n    Out[1]:\n                 score\n    names\n    krippendorff  0.685714\n    fleiss        0.666667\n    scotts        0.657143\n    raw           0.833333\n    mace          0.426531\n    cohen         0.666667\n\n    Out[2]:\n\n    \\begin{tabular}{lr}\n    \\toprule\n    {} \u0026     score \\\\\n    names        \u0026           \\\\\n    \\midrule\n    krippendorff \u0026  0.685714 \\\\\n    fleiss       \u0026  0.666667 \\\\\n    scotts       \u0026  0.657143 \\\\\n    raw          \u0026  0.833333 \\\\\n    mace         \u0026  0.426531 \\\\\n    cohen        \u0026  0.666667 \\\\\n    \\bottomrule\n    \\end{tabular}\n\nFeatures\n--------\n\n.. code-block:: python\n\n    from quica.measures.irr import *\n    from quica.dataset.dataset import IRRDataset\n    from quica.quica import Quica\n\n    coder_1 = [0, 1, 0, 1, 0, 1]\n    coder_2 = [0, 1, 0, 1, 0, 1]\n    coder_3 = [0, 1, 0, 1, 0, 0]\n\n    agreeing_coders = [coder_1, coder_2]\n    agreeing_dataset = IRRDataset(agreeing_coders)\n\n    disagreeing_coders = [coder_1, coder_3]\n    disagreeing_dataset = IRRDataset(disagreeing_coders)\n\n    kri = Krippendorff()\n    cohen = CohensK()\n\n    assert kri.compute_irr(agreeing_dataset) == 1\n    assert kri.compute_irr(agreeing_dataset) == 1\n    assert cohen.compute_irr(disagreeing_dataset) \u003c 1\n    assert cohen.compute_irr(disagreeing_dataset) \u003c 1\n\nSupported Algorithms\n--------------------\n\n+ **MACE** (Multi-Annotator Competence Estimation)\n     + Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., \u0026 Hovy, E. (2013, June). Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1120-1130).\n\n     + We define the inter coder agreeement as the average competence of the users.\n+ Krippendorff's Alpha\n+ Cohens' K\n+ Fleiss' K\n+ Scotts' PI\n+ Raw Agreement: Standard Accuracy\n\nCredits\n-------\n\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template. Thanks to Pietro Lesci and Dirk Hovy\nfor their implementation of MACE.\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvinid%2Fquica","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvinid%2Fquica","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvinid%2Fquica/lists"}