{"id":28533454,"url":"https://github.com/epam/badgerdoc","last_synced_at":"2025-07-14T06:34:57.116Z","repository":{"id":63364876,"uuid":"558749421","full_name":"epam/badgerdoc","owner":"epam","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-04T10:19:16.000Z","size":15904,"stargazers_count":35,"open_issues_count":142,"forks_count":43,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-07-13T12:05:18.086Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/epam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-10-28T07:57:39.000Z","updated_at":"2025-06-16T11:15:26.000Z","dependencies_parsed_at":"2024-04-10T14:30:00.770Z","dependency_job_id":"6a88b302-ea1f-415f-b567-8586e25b0bb0","html_url":"https://github.com/epam/badgerdoc","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"purl":"pkg:github/epam/badgerdoc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fbadgerdoc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fbadgerdoc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fbadgerdoc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fbadgerdoc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/epam","download_url":"https://codeload.github.com/epam/badgerdoc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fbadgerdoc/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265253386,"owners_count":23735093,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-09T17:07:06.629Z","updated_at":"2025-07-14T06:34:57.047Z","avatar_url":"https://github.com/epam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![badgerdoc build](https://github.com/epam/badgerdoc/actions/workflows/badgerdoc.yml/badge.svg?branch=main)](https://github.com/epam/badgerdoc/actions/workflows/badgerdoc.yml)\n\n# What is BadgerDoc\n\n**BadgerDoc** is a ML Delivery Platform made to make delivery process of Machine Learning Solutions visible to customer,\nmanagers and ML team. The primary goal of the platform is to visualize ML model delivery cycle - data annotation,\nmodel training and result visualization.\n\nThe platform has rich functionality in access and data management, annotation setups, and pipeline composition.\nAccess management is based on Keycloak, which is integrated with Active Directory.\nData can be uploaded in batches, organized into datasets as well as uploaded as a single file.\nML pipeline can be applied to a dataset, which will trigger batch processing, or to a single document.\nBadgerDoc is capable of annotating large datasets by many annotators. It has algorithms for task distribution,\nvalidation roles, several validation setups and will have multicoverage of files by annotators in nearest future.\n\nBadgerDoc also has steady growing number of pre-trained models available for users, which can be assembled into pipelines through visual editor.\n\nHaving such a rich functionality, BadgerDoc can be used for implementing full ML development cycle,\nas well as for rapid prototyping, demonstrating EPAM expertise in ML and even for large annotation\nproject when preliminary annoation is available.\n\nFor now, BadgerDoc is working with vectorized and scanned documents, but it has capability of image annotation.\n\n# Local Setup\n\n## For mac users\nWe have tested BadgerDoc under 'colima', so this is the recommended method for a local run.\n\n## Building the base image\n\nRun the following command to build the base image:\n\n```\nmake build_base\n```\n\nAfter the base image is built, it is recommended to clean up any temporary files generated during the build process. To do this, run the following command:\n\n```\nmake clean\n```\n\n## Building microservices\n\nEasiest way to build microservices is to run `make build_all` command, right after that,\nit's possible run docker-compose to serve BadgerDoc in local mode.\n\nIf it's required to build separate microservice, just run `make build_{microservice}` command,\nfor instance: `make build_users` to build or rebuild users\n\n## Running BadgerDoc in local mode\n\nAfter all services are built, you need to create `.env` file in root folder. You may just copy from example: `cp .env.example .env`\n\nTime to run:\n\n```\ndocker-compose -f docker-compose-dev.yaml up -d\n```\n\nNow services are running, but to start using BadgerDoc, some additional configuration steps are required\n\n## Keycloak local configuration\n\n_It's a good idea to automate this section_\n\nImportant! This is not secure configuration, follow [KeyCloak best practices](https://www.keycloak.org/server/configuration-production) to setup on production environment\n\n1. Login into Keycloak using url http://127.0.0.1:8082/auth and `admin:admin` as credentials\n\n2. Go to Realm Settings -\u003e Keys and disable `RSA-OAEP` algorithm. It will help to avoid issue explainded here https://github.com/jpadilla/pyjwt/issues/722\n\n3. Add tenant attribute to `admin` user, go to Users -\u003e select `admin` -\u003e go to Attributes -\u003e create attribute `tenants:local`, and save\n\n4. Go to Clients -\u003e admin-cli -\u003e Mappers -\u003e Create and fill form with following values:\n\n| Param                      | Value          |\n| -------------------------- | -------------- |\n| Protocol                   | openid-connect |\n| Name                       | tenants        |\n| Mapper Type                | User Attribute |\n| User Attribute             | tenants        |\n| Token Claim Name           | tenants        |\n| Claim JSON Type            | string         |\n| Add to ID token            | On             |\n| Add to access token        | On             |\n| Add to userinfo            | On             |\n| Multivalued                | On             |\n| Aggregate attribute values | On             |\n\n5. Go to Client Scopes -\u003e Find `roles` -\u003e Scope and select `admin` in list  to add to Assigned Roles, then go to Mappers and ensure that only 2 mappers exists: `realm roles` and `client roles`. Delete all other mappers\n\n6. Go to Clients -\u003e Create -\u003e Fill form and save\n\n| Param           | Value              |\n| --------------- | ------------------ |\n| Client ID       | badgerdoc-internal |\n| Client Protocol | openid-connect     |\n\n7. Go to Cliens -\u003e Find `badgerdoc-internal` -\u003e change settings `Access Type: Confidential`, set `Service Accounts Enabled` to `On`, set 'Valid Redirect URIs' and 'Web Origins' to '_', then save. Now you can Credentials tab, open it and copy Secret\n\nThen `Client ID` and `Secret` must be set to `.env` as `KEYCLOAK_SYSTEM_USER_CLIENT=badgerdoc-internal` and `KEYCLOAK_SYSTEM_USER_SECRET` to copied key\n\n8. Go to Clients -\u003e Find `badgerdoc-internal` -\u003e Service Account Roles -\u003e Client Roles -\u003e master-realm -\u003e Find `view-users` and `view-identity-providers` in Available Roles and add to Assigned Roles\n\n9. Go to Roles -\u003e add roles: presenter, manager, role-annotator, annotator, engineer. Open admin role, go to Composite Roles -\u003e Realm Roles and add all these roles\n\n10. Go to Realm Settings -\u003e Tokens -\u003e Find `Access Token Lifespan` and set 1 `Days`\n\nTime to reload `docker-compose`, because `.env` was changed:\n\n```\ndocker-compose -f docker-compose-dev.yaml up -d\n```\n\n## Minio configuration\n\nIn the case of installation with Minio configuration, Minio must be accessible\nfrom the browser using the same host and port as used for internal communication.\nThe reason is that BadgerDoc displays PDFs using presigned URLs.\nIf the presigned URL generated by the `assets` microservice\nuses `S3_ENDPOINT=badgerdoc-minio:9000`, then this document will be\naccessible only from `http://badgerdoc-minio:9000`.\n\nFor a local installation, it's possible to add `127.0.0.1 badgerdoc-minio` to the `/etc/hosts` file.\nThis will solve the issue with presigned URLs.\n\nFor any other installation, we highly recommend using AWS S3 or Azure Blob Storage instead of Minio.\n\n## AWS S3 storage configuration\n\nChange your .env file as follows:\n- Set `STORAGE_PROVIDER` to `s3`\n- Set `S3_REGION` to a valid region identifier (e.g., `eu-central-1`)\n\nIt is recommended to use IAM roles (for example, via an EC2 Instance Profile) for accessing S3 buckets. If you are running BadgerDoc in Docker containers, increase the HTTP hop limit to support IMDSv2 by running:\n`aws ec2 modify-instance-metadata-options --instance-id \u003cEC2_INSTANCE_ID\u003e --http-put-response-hop-limit 2`\nIt is required for IMDSv2 support from Docker containers with an EC2 Instance Profile role.\n\nAlternatively, you can configure `S3_ACCESS_KEY` and `S3_SECRET_KEY` to use IAM user credentials for accessing an S3 bucket. However, this approach is less secure and is not recommended.\n\nFinally, ensure that the S3 bucket’s CORS settings are configured to allow access from the domain where BadgerDoc is hosted. For more details, see\n[Configuring cross-origin resource sharing (CORS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html).\n\n## Azure blob storage configuration\n\nChange `STORAGE_PROVIDER=azure` in `.env` file and set `AZURE_STORAGE_CONNECTION_STRING` to the connection string\nof your Azure Blob Storage account.\n\nAdditionally, Blob Storage CORS settings must be configured to allow access\nfrom the domain you are running BadgerDoc on.\n[Cross-Origin Resource Sharing (CORS) support for Azure Storage](https://docs.microsoft.com/en-us/rest/api/storageservices/cross-origin-resource-sharing--cors--support-for-the-azure-storage-services).\n\n## Categories\n\nBe sure that you added all possible categories via badgerdoc UI (/categories) otherwise you get undefined categories on annotations view page\n\n## Set up Airflow as a pipeline service in local mode\n\nAirflow runs using its own resources (PostgreSQL, Redis, Flower) without sharing them with BadgerDoc.\n\n1. Copy `airflow/.env.example` to `airflow/.env` running:\n```\ncp airflow/.env.example airflow/.env\n```\n\nTo setup service account you need to configure Keycloak for BadgerDoc first.\n\n2. Setup service account. Login into Keycloak using url http://127.0.0.1:8082/auth and `admin:admin` as credentials. Select Clients -\u003e badgerdoc-internal -\u003e Service Accounts Roles -\u003e Find Service Account User and click \"service-account-badgerdoc-internal\". Then select Attributes tab and add `tenants:local` attribute like you did it for `admin`.\n\n3. Go to Role Mappings and assign `admin` and `default-roles-master`\n\n4. Go to Clients -\u003e badgerdoc-internal -\u003e Mappers -\u003e Create and fill form:\n\n| Param                      | Value          |\n| -------------------------- | -------------- |\n| Protocol                   | openid-connect |\n| Name                       | tenants        |\n| Mapper Type                | User Attribute |\n| User Attribute             | tenants        |\n| Token Claim Name           | tenants        |\n| Claim JSON Type            | string         |\n| Add to ID token            | On             |\n| Add to access token        | On             |\n| Add to userinfo            | On             |\n| Multivalued                | On             |\n| Aggregate attribute values | On             |\n\n5. Copy `KEYCLOAK_SYSTEM_USER_SECRET` from Badgerdoc `.env` file into Airflow `.env` file, then run\n\n```\ndocker-compose -f airflow/docker-compose-dev.yaml up -d\n```\n\n6. Login to Airflow\n\nThis docker-compose file was downloaded from the Apache Airflow website:\nhttps://airflow.apache.org/docs/apache-airflow/2.7.0/docker-compose.yaml with only a few modifications added.\n\n\n## How to install required dependencies locally\n\n1. Install all required dependencies for a microservice using a packaging tool like Pipenv/Poetry depending on the microservice you are about to set up (we will use Pipenv and \"assets\" service for this example):\n\n```\ncd assets \u0026\u0026 pipenv install --dev\n```\n\n2. Install dependencies from \"lib\" folder:\n\n```\npipenv shell \u0026\u0026 pip install -e ../lib/filter_lib ../lib/tenants\n```\n\n## API docs (swagger)\n\nUse this URL to open the swagger of some service\n\n```\nhttp://127.0.0.1:8080/{service_name}/docs\n```\n\nFor example: http://127.0.0.1:8080/users/docs\n\n## Contributors\n\n\u003ca href=\"https://github.com/epam/badgerdoc/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=epam/badgerdoc\"\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepam%2Fbadgerdoc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fepam%2Fbadgerdoc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepam%2Fbadgerdoc/lists"}