{"id":28602886,"url":"https://github.com/lucaspar/eyetagger","last_synced_at":"2025-06-11T16:12:38.465Z","repository":{"id":37671872,"uuid":"262408686","full_name":"lucaspar/eyetagger","owner":"lucaspar","description":"Web-based tool for Iris Annotation workloads.","archived":false,"fork":false,"pushed_at":"2024-10-24T21:50:11.000Z","size":4284,"stargazers_count":2,"open_issues_count":2,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-16T10:42:00.613Z","etag":null,"topics":["annotation-tool","django","iris-segmentation","postgres","rest-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucaspar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-08T19:15:22.000Z","updated_at":"2024-10-03T18:40:53.000Z","dependencies_parsed_at":"2024-09-12T08:35:13.454Z","dependency_job_id":"64a9e41c-50f6-48b2-9a4d-6591cc1a1ff7","html_url":"https://github.com/lucaspar/eyetagger","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lucaspar/eyetagger","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaspar%2Feyetagger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaspar%2Feyetagger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaspar%2Feyetagger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaspar%2Feyetagger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucaspar","download_url":"https://codeload.github.com/lucaspar/eyetagger/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaspar%2Feyetagger/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259294366,"owners_count":22835786,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["annotation-tool","django","iris-segmentation","postgres","rest-api"],"created_at":"2025-06-11T16:12:34.936Z","updated_at":"2025-06-11T16:12:38.458Z","avatar_url":"https://github.com/lucaspar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EyeTagger | Iris Annotation Tool\n\n\u003cimg src=\"/src/assets/logo-iris.png\" alt=\"Annotator Logo\" width=\"50\"/\u003e\n\n![](images/demo.png )\n\n## Summary\n\n+ Dockerized application for simple deployment\n+ PostgreSQL DB \u003c=\u003e Django + Gunicorn + Nginx web server \u003c= REST API =\u003e Vue-based SPA + Vuex\n+ Django Whitenoise to serve static files, CDN Ready\n+ Annotations stored in relational database\n+ Access control / user management\n+ Vuex handles state management and persistance to never lose annotations on the front-end\n\n## 1. Getting Started\n\n### 1.1. Dependencies\n\nBefore getting started you should have the following installed and running:\n\n+ Docker \u003e= v19\n+ Docker Compose \u003e= v1.25\n\n### 1.2. Link data\n\nData upload via web interface if not possible yet, so the data needs to be mounted inside the container.\n\nIf you have the images in the same machine, just put them in the expected location `data/dataset/` by creating a symbolic link (below) or just moving your data.\n\n\u003e `ln -s    $MY_DATASET_LOCATION    $(pwd)/data/dataset`\n\nIf your dataset is remote (cloud or another computer), you might want to start using `dvc`. Check the [Integrating DVC](#5.-integrating-dvc) session below.\n\n### 1.3 Create environment\n\n```sh\n# copy all example dotenv files\nsudo apt install mmv\nmmv -c 'env/*.env.example' 'env/#1.env'\n\n# edit all env/*.env files setting the following:\n#    DJANGO_STATIC_HOST\n#    SECRET_KEY\n#    DB_PASS\n#    POSTGRES_PASSWORD (same as DB_PASS)\nfind env -name \"*.env\" -exec nano {} \\;\n```\n\n### 1.4. Run services\n\n#### Install Packages\n\n```sh\n# create the public network\ndocker network create net-nginx-proxy\n\n# build docker images and run containers\ndocker-compose up\n\n# from another terminal, run the database migrations\ndocker-compose exec web pipenv run /app/manage.py migrate\n\n# create django superuser\ndocker-compose exec web pipenv run /app/manage.py createsuperuser\n\n# access localhost:80 in your browser\n```\n\n---\n\n## 2. Management\n\n### 2.1 CLI access to services\n\n#### Django + Vue container\n\n\u003e `docker-compose exec web /bin/bash`\n\n#### Nginx container\n\n\u003e `docker-compose exec nginx /bin/sh`\n\n#### PostgreSQL container\n\n\u003e `docker-compose exec db psql --username eyetagger_admin --dbname eyetagger`\n\nMore PostgreSQL commands:\n\n```sh\n\\h  # help\n\\q  # quit\n\\l  # list databases\n\\d  # list tables / relations\n\\d api_annotation   # describe a table / relation\n\n# run a query - don't forget the semicolon:\nSELECT id, annotator_id, image_id FROM api_annotation;\n```\n\n### 2.2 Dashboards\n\n| Feature                     | Default location           | Comment                                                                          |\n| --------------------------- | -------------------------- | -------------------------------------------------------------------------------- |\n| Django REST Framework       | http://localhost/api       | Only available in development mode (_i.e._ `DEBUG=True` in `env/django_app.env`) |\n| Django Administration Panel | http://localhost/api/admin | Credentials created with `pipenv run ./manage.py createsuperuser`                |\n\n### 2.3 Template Structure\n\n| Location from project root | Contents                                |\n| -------------------------- | --------------------------------------- |\n| `backend/`                 | Django Project \u0026 Backend Config         |\n| `backend/api/`             | Django App for REST `api`               |\n| `data/`                    | Git-ignored: DB + backups               |\n| `deploy/`                  | Scripts and configuration files         |\n| `dist/`                    | Git-ignored: back+front generated files |\n| `env/`                     | Environment Files                       |\n| `public/`                  | Static Assets                           |\n| `src/`                     | Vue App                                 |\n\n### 2.4 Database\n\n#### A. Backing up a DB (dump)\n\nTo run it once:\n\n```sh\n# docker-compose up db          # if db container is not running\ndocker-compose exec db pg_dump -U eyetagger_admin eyetagger | \\\n    gzip \u003e eyetagger_bkp_$(date +\"%Y_%m_%d_%I_%M_%p\").sql.gz\n```\n\nCheck [backups.sh](./backups.sh) for a simple automated version.\n\n\u003e Tip: you can add the existing `backups.sh` to your `crontab -e` for periodic backups:\n\n```txt\nTo run it every 6 hours:\n0 */6 * * * /eyetagger/backups.sh \u003e\u003e /eyetagger/data/logs/backups.log 2\u003e\u00261\n\nOr every business day (Mon-Fri) at 6pm:\n0 18 * * 1-5 /eyetagger/backups.sh \u003e\u003e /eyetagger/data/logs/backups.log 2\u003e\u00261\n```\n\n#### B. Restoring a Backup\n\n```sh\n# replace $YOUR_DUMP_GZ by your .gz location:\n\n# let's copy the backup before moving/modifying it\ncp $YOUR_DUMP_GZ /tmp/dump.sql.gz\n\n# extract the dump\ngunzip -k /tmp/dump.sql.gz\n\n# copy to the running DB container\n# docker-compose up db          # if db container is not running\ndocker cp /tmp/dump.sql eyetagger_db_1:/dump.sql\n\n# create a new empty database\ndocker-compose exec db createdb -U eyetagger_admin -T template0 eyetagger_new\n\n# populate the empty database with the dump\ndocker-compose exec db psql -U eyetagger_admin -d eyetagger_new -f /dump.sql\n\n# swap database names\ndocker-compose exec db psql --username eyetagger_admin --dbname postgres\n\\l\nALTER DATABASE eyetagger RENAME TO eyetagger_old;\nALTER DATABASE eyetagger_new RENAME TO eyetagger;\n\\l\n\\q\n\n# get the other services up and try it out!\ndocker-compose down \u0026\u0026 docker-compose up\n\n# if successful, clean the temporary backup copies\nrm      /tmp/dump.sql.gz     /tmp/dump.sql\n```\n\n## 3. Development Deploy (Default)\n\n1. There are 2 entries `command` under `docker-compose.yaml` \u003e Service `web`. Select the \"development\" one by commenting out the alternative.\n\n2. Run `docker-compose up` (run `down` first if already up) and open `localhost:9000`. Hot reload should be enabled i.e. live changes to the front-end code will update the browser.\n\n## 4. Production Deploy (Optional)\n\n1. Adapt the environment files for the backend in `env/`.\n2. Adapt the environment file for the frontend in `vue.config.js`.\n3. Follow the [Django deployment checklist](https://docs.djangoproject.com/en/2.1/howto/deployment/checklist/) for further configuration.\n4. Deploy the dockerized application in a remote server by running it in daemon form: `docker-compose up -d \u0026\u0026 docker-compose logs -f`.\n\n## 5. Integrating DVC (Optional)\n\n1. Install `dvc` on host\n\n    \u003e `pip install dvc`\n\n2. Setup access (using a GCP below)\n\n    ```sh\n    # get provider-specific api\n    pip install 'dvc[gs]'\n\n    # create google bucket credentials\n    mkdir -p $HOME/.gcp/\n    GOOGLE_APPLICATION_CREDENTIALS=$HOME/.gcp/iris-admin.json\n\n    # paste the contents of the GCP JSON in this file\n    # see https://cloud.google.com/docs/authentication/getting-started\"\n    nano $GOOGLE_APPLICATION_CREDENTIALS\n    chmod 400 $GOOGLE_APPLICATION_CREDENTIALS\n\n    export GOOGLE_APPLICATION_CREDENTIALS\n    echo -e ' \u003e\u003e Add this to your ~/.bashrc:\\n\\n\\\n        export GOOGLE_APPLICATION_CREDENTIALS='$GOOGLE_APPLICATION_CREDENTIALS'\\n\\n\n    ```\n\n3. Then get your data from the remote.\n\n    \u003e `dvc pull`\n\n    Or add new data to the bucket\n\n    \u003e `dvc add data/dataset \u0026\u0026 dvc push`\n\n## 6. Bringing your own dataset\n\nEyetagger handles two types of data: the images - referred to as the **dataset**, and the metadata - stored in a relational **database / db** using PostgreSQL.\n\nMetadata is necessary to keep track of the annotations, who did them, when, and any other data attribute that might be useful for the annotation workload. The dataset is usually a set of images to be displayed during the annotation process.\n\nIn order to serve a custom dataset, you will need to first A. run the app creating a database (steps 1.1-1.4 above) and then B. create the metadata entries for your dataset in PostgreSQL.\n\nBelow we describe how to do this part B by using a database migration:\n\n1. **Create a migration.**\n\n    The metadata entries are created by running one or more database migrations. Let's create an empty one with:\n\n    ```bash\n    # this assumes your containers are up, make sure to run docker-compose up first\n\n    # below and onwards, \"api\" is the internal name of the Django app that we are working with\n    docker-compose exec web pipenv run /app/manage.py makemigrations api --name dataset_import --empty\n    ```\n\n    After this command will have a new Python file in the migrations' directory (e.g. `backend/api/migrations/####_dataset_import.py`).\n\n2. **Call a new and customized migration script to ingest your dataset's metadata into the relational DB.**\n\n    Change that created file to import your custom script as follows:\n\n    ```python\n    from backend.api.manual_migration import import_dataset\n\n    # down in the Migration class, paste the following:\n    class Migration(migrations.Migration):\n\n        # ...\n\n        initial = True\n\n        # import_dataset is the function that will be called when you run the migration\n        # reverse_code is the function that will be called when you rollback the migration, using a \"no-op\" function below\n        operations = [\n            migrations.RunPython(import_dataset, reverse_code=migrations.RunPython.noop)\n        ]\n\n        # ...\n\n    ```\n\n3. **Customize this migration script and ORM models to match your dataset.**\n\n    + An example of a migration script can be found in `backend/api/manual_migration.py` - you can use this as a template for your own script.\n    + All SQL code and database transactions are handled by Django's ORM, so you don't need to know SQL to populate the database.\n    + The existing migration script loads a CSV file that contains metadata for each image. Because each dataset is unique, yours might have different attributes.\n    + The `import_dataset` function in that script loads this CSV, creates all ORM objects (e.g. the `img` variable), and saves them to the database `img.save()`. The other functions help with this process.\n    + Change the Image model:\n        1. Modify `backend/api/models.py` to fit your needs.\n        2. Run `/app/manage.py makemigrations` - this compares model.py to the database, if their schemas differ it'll generate code that describes a new migration.\n        3. Run `/app/manage.py migrate` to \"run\" the necessary migrations, effectively updating the database. Django keeps track of the migrations that were run.\n    + ⚠️ The attributes of your `Image` model should be close to the columns in your CSV file. If you try to store an ORM object that deviates from the table schema, the database transaction will fail.\n\n4. **Run the migrations.**\n\n    Only the necessary (new) migrations will be run with the following command:\n\n    ```bash\n    docker-compose exec web pipenv run ./manage.py migrate\n    ```\n\n    \u003e 💡 After you create (and save) some entries like `Image` objects, you will be able to see them in the Django admin panel (see [dashboards](#22-dashboards) above).\n\n5. **Troubleshooting:** when a migration goes wrong.\n\n    Errors might happen if the migration script is not correct. If so, you can **reverse** it with:\n\n    ```bash\n    # change 0001 below\n    docker-compose exec web pipenv run ./manage.py migrate api 0001\n    ```\n\n    Where `0001` is the number of the **previous** migration (i.e. the number `####` in `backend/api/migrations/####_migration_name.py`).\n\n    Another way is to reset them all: [see scenario 2 in this guide](https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html), our \"app name\" is `api`.\n\n    \u003e **A note about migrations that change schemas:** if a migration modifies the database schema, make sure your rollback function also undoes those changes. For example, if migration `N` adds a new column to the `Image` model, and you roll back to `N-1`, this roll back function should also remove that column from the `Image` model. Otherwise, when you run `N` again, Django will try to create a column that already exists, which will fail. Because of this rollback complication, I chose to separate migrations that change the database schema (e.g. creating tables, modifying attributes) from migrations that populate the database with data (e.g. the one in `manual_migration.py`).\n\n    Above are the best ways to fix migration issues and avoid corruption or data loss. But if losing data is not an issue, you can also delete the database and start over, for example:\n\n    ```bash\n    # ⚠️ this will cause data loss\n    docker-compose exec db dropdb -U eyetagger_admin eyetagger\n    docker-compose exec db createdb -U eyetagger_admin eyetagger\n    docker-compose exec web pipenv run /app/manage.py migrate\n    ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucaspar%2Feyetagger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucaspar%2Feyetagger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucaspar%2Feyetagger/lists"}