{"id":15149104,"url":"https://github.com/kyflores/lumine","last_synced_at":"2026-01-21T10:35:51.898Z","repository":{"id":88317667,"uuid":"569612414","full_name":"kyflores/lumine","owner":"kyflores","description":"Prototype multi object detector and tracker suite for FRC","archived":false,"fork":false,"pushed_at":"2024-04-04T00:07:41.000Z","size":7592,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"develop","last_synced_at":"2025-04-07T05:42:20.722Z","etag":null,"topics":["apriltags","first-robotics-competition","frc","nanodet-plus","object-detection","openvino","sort","tvm","yolov8"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyflores.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-23T08:18:33.000Z","updated_at":"2024-04-04T00:07:44.000Z","dependencies_parsed_at":"2025-04-07T05:42:03.237Z","dependency_job_id":"a2fcd892-b138-45d0-83c5-6227e6fbf4a4","html_url":"https://github.com/kyflores/lumine","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kyflores/lumine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2Flumine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2Flumine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2Flumine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2Flumine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyflores","download_url":"https://codeload.github.com/kyflores/lumine/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2Flumine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28631936,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-21T04:47:28.174Z","status":"ssl_error","status_checked_at":"2026-01-21T04:47:22.943Z","response_time":86,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apriltags","first-robotics-competition","frc","nanodet-plus","object-detection","openvino","sort","tvm","yolov8"],"created_at":"2024-09-26T13:41:49.825Z","updated_at":"2026-01-21T10:35:51.868Z","avatar_url":"https://github.com/kyflores.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lumine\nPrototype multi object detector and tracker suite for FRC\n\n## Install\nProvided you are on ubuntu 22.04, just use\n```\nbash install.sh\n```\n`install.sh` installs everything it can from apt, creates a venv that\nuses system packages, then adds a few python packages that aren't in apt repos.\n\nTry `conda install -c conda-forge gcc=12.1.0` if you get an error about GLIBCXX_3.4.30\n\n## Example Launch\n```\npython lumine.py --weights /path/to/openvino/weights/folder --source 0 --table --stream 5800 --draw\n```\n\n`weights`: Path to a folder containing openvino .bin, .xml, .mapping files\n`source`: A name that can be passed to cv2.VideoCapture. Numbers refer to  /dev/videoX devices\n`table`: Draw an ASCII table of the detection results\n\n## Embedded Hardware Support (goals)\n* CPU, with pytorch and OpenVINO backends.\n* Intel IGPs with OpenVINO\n\n## Custom detectors\nAll detectors should support a `detect(img) -\u003e list(dict)` function.\n\n`detect` is passed a 3 channel RGB image as a numpy array of shape (H, W, 3).\nDetectors should not modify the image they receive, if you must modify it,\ncreate a copy and modify that.\n\nDetectors should produce a dictionary with at least the following\nfor each detection instance.\n* `type`, Which detector this originated from.\n* `id` Could be a tag family, object type. Info about what's in the bounding box.\n* `color` RGB triple for the box color. Used when drawing the feed.\n* `corners` of the bounding area of the detection\n  * numpy array, 4 rows, 2 columns, shape = (4, 2)\n  * `corners` should be scaled to the input image: If the input is rescaled inside the\n     `detect` function, corners must be scaled back to the original.\n* `confidence` A confidence value between 0 and 1 for the detection.\n  * 0 is worst, 1.0 is best. Scale the value if it's not between 0 and 1.0.\n\n`detect` should always return a list of detections. If there is only one detection, it\nshould still be in a list.\n\nSeveral detector-specific or post-processing keys may also\nexist, but are not guaranteed for every object.\n* `distance` (stereo depth estimator, apriltags)\n* `pose` (apriltags)\n* `confidence` (yolov5)\n* `sort_id` (SORT)\n\n## Camera configuration\nMost USB webcams have a few configurable parameters. To minimize\nmotion blur we want to operate at the fastest exposure time possible,\neven if that doesn't always translate to more frames delivered per second.\n\nIn this context is a device number corresponding to /dev/videoX.\nHere's an example command set for configuring the Logitech C310.\nPass `-C` instead of `-c` to read a property instead of set it.\n```\nv4l2-ctl --list-devices # Find your /dev/videoX number\nv4l2-ctl --all -d device # Show all configuration properties\nv4l2-ctl -d 3 -c exposure_auto=1\nv4l2-ctl -d 3 -c gain=200\nv4l2-ctl -d 3 -c exposure_time_absolute=250\n```\n\nOpenCV can configure these parameters with the `cv2.CAP_PROP_*` fields,\nbut just from experimenting with it, it doesn't seem reliable. Setting\nthe gain for instance changes the number reported by `v4l2-ctl` but\ndoesn't affect the image.\n\n## Networktables\nIf using `lumine` with network tables, pass `--nt TE.AM` on the command line or pass\na full ip like `127.0.0.1`.\n\nLumine creates the table `/lumine` and a subtable for each detector family. For instance,\n`lumine/yolo` and `lumine/apriltags`.\n\nWithin each detector specific table, data is stored in a \"struct of arrays\" consisting\nof an array of scalars for each property. For instance, `yolo` produces...\n```\n/lumine/yolo\n\nlen: \u003cnumber\u003e\nids: [0, 4, 62, 1....]\nconfidence: [0.5, 0.42, 0.32, 0.12....]\n```\nTo access all the properties for a one particular detection, you must request the same\nindex from each property array.\n\nDetections are always sorted in order of descending confidence.\n\n## Prepping OpenVINO models\nThis project uses OpenVINO for inference on Intel integrated graphics.\nThe easiest way to get the Ultralytics/YoloV5 model in OpenVINO format is to\nsimply request it when calling export.\n`python export.py --weights yolov5s.pt --include openvino`\nModels exported this way appear to be the standard float32 version.\n\nThe hardware may also take advantage of float16 for a performance improvement\nwith minimal effort.\n\nTo get the model into float16 format:\n```\npython export.py --weights /path/to/your/weights.pt --include onnx --opset 12\n\n# mo is installed with the openvino-dev package.\nmo --input_model /path/to/your/weights.onnx --compress_to_fp16 True\n```\n\nOpenVINO produces a `.xml, .bin, and .mapping` file from your model, which should be put\nin its own folder. Then, passing the path to the folder to `--weights` when calling lumine.\n\nYoloV8 can be easily exported to OpenVINO weights by the Ultralytics command line util.\n```\nyolo export model=yolov8n.pt format=openvino half=true\n```\n\n## Retraining Models\nLumine aims to offer some choice of object detection model, depending on the coprocessor in use.\nThe YoloV5 or YoloV8 families will be the first choice for powerful coprocessors\nthat provide acceleration hardware like NVIDIA Jetson or Intel Iris Xe. More limited systems\ncan benefit from a lighter model like NanoDet.\n\nWe recommend `CVAT` (cvat.ai) for image labeling. CVAT is a labeling server developed by\nIntel that runs a web interface that users can connect to and label images. CVAT is primarily\nself-hosted, but the setup procedure is relatively low-effort thanks to its docker-compose\nprocedure. Some cloud based solutions are Roboflow or Supervisely, but may have certain limitations\nin the free-tier.\n\n### YOLO\nRetraining YOLO is quite easy thanks to how complete the Ultralytics solution is.\nStart by exporting to the YOLO 1.1 format in CVAT.\nTODO\n\n### Autolabeling\nLumine provides a script (`utils/autolabel.py`) that uses an existing model to generate YOLO format\nlabels to be imported into CVAT. The workflow would look something like this.\n* Collect and label a small dataset. For FRC teams, this could be the field walkthrough videos that\n  FIRST releases on day one.\n* Train the largest instance of YOLO your hardware can handle on the small dataset.\n* Collect additional datasets. Have autolabel analyze them and generate labels.\n* Import labels into CVAT and manually correct errors. Add the finished labels to the cumulative\n  dataset\n* Retrain the model with the cumulative dataset and repeat the process. As the dataset improves\n  autolabel should produce better results.\n\nIdeally autolabeling can progress to a state where it's practical for teams to add data collected\nfrom practice matches on the first day of competition, and produce a model with higher\nperformance in their particular environment as quickly as possible.\n\n## Dev TODOs\n* Improve mapping SORT boxes back to detect boxes. Current method allows\ndouble assignment, and this seems bugged.\n  * We might want to run a single sort instance per detector type\n* Add blob detector module. Kind of questionable b/c it is being phased out.\n* Support streaming the augmented camera feed to the driver station.\n  * Test cscore module\n* Add OpenVINO int8 quantization flow. Should accept the same dir hierarchy\n  as the yolov5 training set since we need representative images during calibration.\n  * int8 is only for CPUs, so maybe this is not useful. Xe IGPU cannot benefit from int8\n* Support fp16 and int8 calibration for Jetson with TensorRT, and validate pytorch.\n* Figure out how to request a faster update rate from NT\n* Support multiple cameras and switching\n* Add a timestamp to networktables API\n* Support Nanodet training and inference.\n* (High effort) Implement a Lumine vendor library that provides an RPC interface for\n  querying detects rather than using network tables.\n* Replace cscore with a gstreamer solution for HW accelerated temporal codecs.\n  * Also need to provide some sort of driver station client or dashboard plugin\n\n### Style\nThis project uses `black` because it's easy.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyflores%2Flumine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyflores%2Flumine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyflores%2Flumine/lists"}