{"id":28495380,"url":"https://github.com/bmw-innovationlab/sordi-data-pipeline-reader","last_synced_at":"2025-07-02T08:31:24.587Z","repository":{"id":64249313,"uuid":"560788239","full_name":"BMW-InnovationLab/SORDI-Data-Pipeline-Reader","owner":"BMW-InnovationLab","description":"SORDI dataset has per frame annotation file in json format. Following tools create a COCO style annotation out of it. Thus the SORDI data can be easily fed into COCO style training pipelines.","archived":false,"fork":false,"pushed_at":"2022-11-14T20:32:18.000Z","size":28,"stargazers_count":69,"open_issues_count":0,"forks_count":4,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-08T11:11:29.343Z","etag":null,"topics":["bmw","coco","computer-vision","coverter","dali","dataset","deeplearning","juypter-notebook","nocode","nvidia-gpu","object-detection","pipeline","python","sordi"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BMW-InnovationLab.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-11-02T09:14:29.000Z","updated_at":"2025-05-25T19:46:08.000Z","dependencies_parsed_at":"2023-01-15T06:25:19.828Z","dependency_job_id":null,"html_url":"https://github.com/BMW-InnovationLab/SORDI-Data-Pipeline-Reader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BMW-InnovationLab/SORDI-Data-Pipeline-Reader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BMW-InnovationLab%2FSORDI-Data-Pipeline-Reader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BMW-InnovationLab%2FSORDI-Data-Pipeline-Reader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BMW-InnovationLab%2FSORDI-Data-Pipeline-Reader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BMW-InnovationLab%2FSORDI-Data-Pipeline-Reader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BMW-InnovationLab","download_url":"https://codeload.github.com/BMW-InnovationLab/SORDI-Data-Pipeline-Reader/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BMW-InnovationLab%2FSORDI-Data-Pipeline-Reader/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263102550,"owners_count":23414125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bmw","coco","computer-vision","coverter","dali","dataset","deeplearning","juypter-notebook","nocode","nvidia-gpu","object-detection","pipeline","python","sordi"],"created_at":"2025-06-08T11:10:27.683Z","updated_at":"2025-07-02T08:31:24.578Z","avatar_url":"https://github.com/BMW-InnovationLab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Loading BMW SORDI into NVIDIA DALI Pipeline (COCO based)\nSORDI dataset has per frame annotation file in json format. Following tools create a COCO style annotation out of it. Thus the SORDI data can be easily fed into COCO style training pipelines.\n\nSample code to consume the COCO style SORDI with NVIDIA DALI Pipeline is given and can be used in PyTorch/Tensorflow etc.\n\n## Table of Contents\n* [Prerequisites](#prerequisites)\n* [Unzipping SORDI dataset](#unzipping-sordi-dataset)\n* [Building and Running the docker image](#building-and-running-the-docker-image)\n* [Run 2_traverse_unzipped_SORDI.ipynb](#run-2_traverse_unzipped_sordiipynb)\n* [Run 3_create_coco_annotation.ipynb](#run-3_create_coco_annotationipynb)\n* [Run 4_run_DALI_coco_pipeline.ipynb](#run-4_run_dali_coco_pipelineipynb)\n\n## Prerequisites\n- NVIDIA Docker 2\n- Docker CE latest stable release\n\n## Unzipping SORDI dataset\nOpen a Terminal in the SORDI folder and excute the following command to unzip. \n\n```sh\nfor i in *.zip; do unzip \"$i\"; done\n```\n\nDelete zip files using the following command.\n\n```sh\nrm -rf *zip\n```\n\nThis is how it should look like from a structure after unzipping.\n\n```sh\nls -l SORDI\n```\n![unzip](https://user-images.githubusercontent.com/69092782/201635392-7560bf91-6637-4859-a1a9-d2cd311eaf2c.png)\n\n## Building and Running the docker image\nBase image is coming from NGC cloud (ngc.nvidia.com).\n\nPlease register if not done already (3 minutes).\n\nBefore you start, map the SORDI directory into the docker:\n\nopen ``1_run.sh`` \n\nChange ``/home/me/SORDI`` to the path of the extracted SORDI dataset file.\n\n![image](https://user-images.githubusercontent.com/69092782/201643133-bd5e3410-fc6d-49f3-8861-90f1f55590c4.png)\n\n1 - Login to NVIDIA NVCR using the following command:\n```sh\ndocker login nvcr.io\n```\n2 - Login to Docker using the following command:\n```sh\ndocker login\n```\n3 - Build and run the image using the following command:\n```sh\nsource 1_run.sh\n```\nWhen done it should look like this:\n\n![Screenshot from 2022-11-14 12-11-29](https://user-images.githubusercontent.com/69092782/201646427-1b8c68dc-17a5-4b5e-b653-094c2b1f0b03.png)\n\n## Run 2_traverse_unzipped_SORDI.ipynb\nIt walks through the unzipped SORDI files. It opens a sqlite database. For each frame and annotation it creates an entry into the ``FRAMES`` table.\n\nInside the terminal run:\n```sh\njupyter notebook\n```\n![image](https://user-images.githubusercontent.com/69092782/201647438-b9f87f37-7321-4da8-b222-5749daaccaf5.png)\n\nOpen the provided URL in a browser and run 2_traverse_unzipped_SORDI.ipynb\n\n![image](https://user-images.githubusercontent.com/69092782/201648575-b5c7adc3-fd41-4938-a5e7-ecbf40eb0555.png)\n\nYou find the created sqlite database in the workspace folder. Check its entries via:\n```sh\nsqlite SORDI.sqlite\n.tables \nselect * from FRAME limit 10;\n```\nFeel free to create additional table entries like:\n* Amount of objects in frame\n* Overlap/Pixeloverlap of objects in frame\n* Uncertainty estimation\n* Single class or multiclass\n\n## Run 3_create_coco_annotation.ipynb\nRun the notebook to create the COCO annotation file.\n\nThe outcome is the file:\n\n``sordi.coco``\n\nThis is a great place to filter the training dataset in a smart manner. E.g. choose multiclass training frames with a certain object overlap only. By now, this notebook does not filter at all but exports all data found in the database.\n\n\n## Run 4_run_DALI_coco_pipeline.ipynb\nReady to run the pipeline? Lets go. NVIDIA DALI does image decompression and augmentations on the GPU. Since the annotation file can get larger, the initial loading and parsing takes a moment.\n\n![image](https://user-images.githubusercontent.com/69092782/201652778-f57ade11-f28d-4534-9f59-ca1c9c65fbdc.png)\n\n## Acknowledgments\n* Adolf Hohl\n* Ziad Saoud, BMW Group TechOffice MUNICH\n* Chafic Abou Akar, BMW Group TechOffice MUNICH\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbmw-innovationlab%2Fsordi-data-pipeline-reader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbmw-innovationlab%2Fsordi-data-pipeline-reader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbmw-innovationlab%2Fsordi-data-pipeline-reader/lists"}