{"id":23440852,"url":"https://github.com/living-with-machines/zoonyper","last_synced_at":"2025-07-04T15:37:27.867Z","repository":{"id":103706543,"uuid":"483235795","full_name":"Living-with-machines/zoonyper","owner":"Living-with-machines","description":"Code to make it easy to import and process Zooniverse annotations and their metadata in Python/Jupyter Notebooks","archived":false,"fork":false,"pushed_at":"2025-02-12T00:15:55.000Z","size":1714,"stargazers_count":0,"open_issues_count":3,"forks_count":2,"subscribers_count":0,"default_branch":"v0.1.0","last_synced_at":"2025-07-04T15:37:17.398Z","etag":null,"topics":["crowdsourcing","data","data-processing","data-science","python","zooniverse"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Living-with-machines.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-04-19T12:24:39.000Z","updated_at":"2024-10-17T14:23:11.000Z","dependencies_parsed_at":"2025-04-13T09:13:18.657Z","dependency_job_id":"f631c2dd-cc1f-402d-b5aa-e22d7b34c0a5","html_url":"https://github.com/Living-with-machines/zoonyper","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Living-with-machines/zoonyper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fzoonyper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fzoonyper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fzoonyper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fzoonyper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Living-with-machines","download_url":"https://codeload.github.com/Living-with-machines/zoonyper/tar.gz/refs/heads/v0.1.0","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fzoonyper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263568099,"owners_count":23481614,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crowdsourcing","data","data-processing","data-science","python","zooniverse"],"created_at":"2024-12-23T16:19:21.655Z","updated_at":"2025-07-04T15:37:27.828Z","avatar_url":"https://github.com/Living-with-machines.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zoonyper: Processing Zooniverse Annotations and Metadata\n\n[![Zenodo record for repository](https://zenodo.org/badge/DOI/10.5281/zenodo.13946466.svg)](https://doi.org/10.5281/zenodo.13946466)\n[![All Contributors](https://img.shields.io/github/all-contributors/Living-with-machines/zoonyper?color=ee8449\u0026style=flat-square)](#contributors)\n\n**Zoonyper** is a Python library, designed to make it easy for users to import and process Zooniverse annotations and their metadata in your own Python code. It is especially designed for use in [Jupyter Notebooks](https://jupyter.org/).\n\n## Purpose\n\nThe [Zooniverse citizen science platform's Project Builder](https://www.zooniverse.org/lab) allows anyone to create crowdsourced tasks using uploaded or [imported images](https://blogs.bl.uk/digital-scholarship/2022/04/importing-images-into-zooniverse-with-a-iiif-manifest-introducing-an-experimental-feature.html) and other media. However, its flexibility means that the data created can be difficult to process.\n\nZoonyper can help process the output files from the Zooniverse citizen science platform, and facilitate data wrangling, compression, and output into JSON and CSV files. The output files can then be more easily used in e.g. Observable visualisations, Excel and other tools.\n\n## Background\n\nThe library was created as part of the [Living with Machines project](https://livingwithmachines.ac.uk), a research project developing historical and data science methods to study the effects of the mechanisation on the lives of ordinary people during the long nineteenth century.\n\nAs part of that work, we used digitised historical newspapers at scale. We chose crowdsourcing as a method for some of this work so that we could invite the public to actively contribute to our research, observe how training data is created and annotated for machine learning, and to view the source material we were using across the project. We used the Zooniverse project builder as it is designed for citizen science projects where volunteers contribute to scientific research projects by annotating and categorizing images or other data. The annotations created by volunteers are collected as \"classifications\" in the Zooniverse system.\n\nWe queried digitised newspapers for keywords related to our research topics, uploaded the images, automatically transcribed text (OCR) and metadata about the selected articles to Zooniverse, then asked volunteers to help us with classifications or transcriptions (typing in text) of those articles. The final goal for the research overall was to use the annotations to study the content of these historical newspapers and gain insights into the events and trends of the past.\n\n### Previous work\n\n- ['Zooniverse to Observable' notebook by @kallewesterling](https://observablehq.com/d/55d968557378768e)\n- [Earlier work on a sample Zoonyper Notebook by @GiorgiatolfoBL and @kallewesterling](https://colab.research.google.com/github/Living-with-machines/zooniverse-analysis-workshop/blob/main/process-zooniverse-data.ipynb)\n\n## Getting started\n\nHere's how you can use Zoonyper in your own project:\n\n1. **Install the repository**: First, you'll need to install the repository. You can do this by cloning the repository or installing it using [the instructions below](#installation).\n\n2. **Import the `Project` class**: Once you've installed the repository, you can import the `Project` class into your own Python code. You can do this by adding the following line to the top of your code:\n\n   ```py\n   from zoonyper import Project\n   ```\n\n3. **Initialize a `Project` object**: To start using the Project class, you'll need to create a Project object. You can do this by calling the constructor and passing in the path to the directory that contains all the files from your Zooniverse project lab\\*:\n\n   ```py\n   project = Project(\"\u003cpath to the directory with all necessary files\u003e\")\n   ```\n\n4. **Access the project's data and metadata**: Once you have a `Project` object, you can access its annotations by using the `.classifications` attribute. This attribute is a Pandas DataFrame, where each row contains information about a single classification, including annotations.\n\n5. **Process the data and metadata**: Because the data structures in Zoonyper are Pandas DataFrames, you can process the classifications, subjects, and annotations in any way you like, using the tools and techniques that you're familiar with. For example, you might want to calculate statistics about the annotations, or create plots to visualize the data.\n\n## Preparing your Zooniverse files\n\nVia Zooniverse's web 'Lab' interface, go to the Data Exports page. Request and download these exports:\n\n- classification export\n- subject export\n- workflow export\n- talk comments\n\nThey should be named \"classifications.csv\", \"subjects.csv\", \"workflows.csv\", and \"comments.json\" and \"tags.json\" respectively, placed in a folder. This folder's path is what should be passed to the `Project` constructor.\n\n## Installation\n\n\u003c!--\nInstalling through `pip`:\n\n```sh\n$ pip install zoonyper\n```\n--\u003e\n\nBecause this project is in **active development**, you need to install from the repository for the time being. In order to do so, follow [the installation instructions](docs/source/installing.rst).\n\n## Documentation\n\nYou can see the public documentation on https://living-with-machines.github.io/zoonyper.\n\nYou can contribute to the documentation using [`sphinx`](https://www.sphinx-doc.org/en/master/) to edit and render the [`docs`](docs) directory.\n\n## Data model\n\nThe Zoonyper dataframes' data model is illustrated in the following diagram.\n\n```mermaid\nerDiagram\n    workflow ||--|{ annotation : has\n    workflow }|--o{ subject_set: has\n    subject_set }|--|{ subject: contains\n    annotation ||--|| subject: on\n    user ||..|{ annotation : makes\n    user ||..|{ comment : writes\n    tag }|--|{ comment : in\n    comment ||--o{ subject : \"written about\"\n```\n\n## Contributors\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://www.westerling.nu\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/7298727?v=4?s=100\" width=\"100px;\" alt=\"Kalle Westerling\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eKalle Westerling\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/Living-with-machines/zoonyper/commits?author=kallewesterling\" title=\"Documentation\"\u003e📖\u003c/a\u003e \u003ca href=\"#ideas-kallewesterling\" title=\"Ideas, Planning, \u0026 Feedback\"\u003e🤔\u003c/a\u003e \u003ca href=\"#projectManagement-kallewesterling\" title=\"Project Management\"\u003e📆\u003c/a\u003e \u003ca href=\"https://github.com/Living-with-machines/zoonyper/pulls?q=is%3Apr+reviewed-by%3Akallewesterling\" title=\"Reviewed Pull Requests\"\u003e👀\u003c/a\u003e \u003ca href=\"https://github.com/Living-with-machines/zoonyper/commits?author=kallewesterling\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://openobjects.org.uk\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/380763?v=4?s=100\" width=\"100px;\" alt=\"Mia\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eMia\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/Living-with-machines/zoonyper/commits?author=mialondon\" title=\"Documentation\"\u003e📖\u003c/a\u003e \u003ca href=\"#ideas-mialondon\" title=\"Ideas, Planning, \u0026 Feedback\"\u003e🤔\u003c/a\u003e \u003ca href=\"#projectManagement-mialondon\" title=\"Project Management\"\u003e📆\u003c/a\u003e \u003ca href=\"https://github.com/Living-with-machines/zoonyper/pulls?q=is%3Apr+reviewed-by%3Amialondon\" title=\"Reviewed Pull Requests\"\u003e👀\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/jmiguelv\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/1233186?v=4?s=100\" width=\"100px;\" alt=\"Miguel Vieira\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eMiguel Vieira\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/Living-with-machines/zoonyper/commits?author=jmiguelv\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliving-with-machines%2Fzoonyper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliving-with-machines%2Fzoonyper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliving-with-machines%2Fzoonyper/lists"}