{"id":16201056,"url":"https://github.com/lukexyz/flightvision","last_synced_at":"2025-03-19T05:31:06.991Z","repository":{"id":106690527,"uuid":"298709058","full_name":"lukexyz/FlightVision","owner":"lukexyz","description":"🎥🎯 Tracking dart coordinates with fastai v2","archived":false,"fork":false,"pushed_at":"2024-01-14T01:36:16.000Z","size":27355,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-01-28T23:12:49.580Z","etag":null,"topics":["computer-vision","dart"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lukexyz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-09-26T00:50:06.000Z","updated_at":"2024-04-23T13:34:20.147Z","dependencies_parsed_at":"2024-04-23T13:44:30.480Z","dependency_job_id":null,"html_url":"https://github.com/lukexyz/FlightVision","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukexyz%2FFlightVision","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukexyz%2FFlightVision/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukexyz%2FFlightVision/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukexyz%2FFlightVision/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lukexyz","download_url":"https://codeload.github.com/lukexyz/FlightVision/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243971180,"owners_count":20376784,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","dart"],"created_at":"2024-10-10T09:35:27.571Z","updated_at":"2025-03-19T05:31:01.983Z","avatar_url":"https://github.com/lukexyz.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🎥🎯 FlightVision\n\u003e Serious bar sports, with seriously accurate XY coordinates.  \n\nA realtime solution to track dart targets with `fastai` and `unets`, using segmentations layers rather than classical regression.  \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/tracker_shot_hq.gif\"\u003e \n\u003c/p\u003e\n\n:bookmark_tabs: [FlightVision_Unet_c950.ipynb.ipynb](https://github.com/lukexyz/FlightVision/blob/master/nbs/16fastai2_FlightVision_Unet_c950.ipynb)\n\n##### 📋 `x, y` Coordinate Tracking  \n\n1. Input frame passes through `unet` model for segmentation mask inference.  \n2. `OpenCV` blob detection identifies centroid location.  \n3. `OpenCV` overlay is applied to show the results\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/method.JPG?raw=true\" width=70%\u003e \n\u003c/p\u003e\n\n\n## Background\nFlight Club is the name of a franchise in London that modernised the classic [pub sport](https://flightclubdarts.com/london/our-story). It's success comes from the [gamified](https://en.wikipedia.org/wiki/Gamification) automated scoring system. It has a 3D vision system comprised of live cameras, and also includes a well-polished interface which runs on a screen above the board. This opens the traditional game up for improved entertainment with the inclusion of minigames like team elimination, or a snakes-and-ladders adaption – where the target of your dart throw is how far forward you move on the board. \n\nAs of 2020 it has expanded into 3 locations in the UK and one in Chicago, USA. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/location.JPG?raw=true\" width=70%\u003e \n\u003c/p\u003e  \n\n#### How it works\n\nPowered by a \"[highly sophisticated 3D vision system](https://www.stemmer-imaging.com/en-gb/applications/imaging-systems-for-sports-tracking/)\", the software brute-forces the solution using calibrated cameras positioned in a frame above the dart board. \n\n\u003e A normal dart impact on the board triggers Flight Club Darts’ specially developed 3D fitting algorithms to identify, recognise and measure the precise position, pose and score of the dart to within a fraction of a millimetre. **The software manipulates three virtual darts through millions of different orientations and angles until it finds what matches where the dart landed on the board.** Using multiple cameras reduces obscuration effects.\n\nA deep learning approach was attempted with a [challenge to PhD students as part of a country-wide university competition](https://www.telegraph.co.uk/connect/small-business/flight-club-innovation-written-into-business-objectives/) around 2014. At that time no solution was found, and so more conventional computer vision techniques were used. \n\n# Bootleg Flightclub using `fastai` v2\n\nTake your own pictures, find the `x,y` coordinates of the dart point using a unet segmentation layer.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/training_centroid_01.JPG?raw=true\" width=70%\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/training_sofa_01.JPG?raw=true\" width=50%\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/centroid_handhold.JPG?raw=true\" width=50%\u003e\n\u003c/p\u003e\n\n\n# Development\n\n### 1. What Didn't Work\n\nIn [lesson 3](https://nbviewer.jupyter.org/github/fastai/course-v3/blob/master/nbs/dl1/lesson3-head-pose.ipynb) of the fastai deep learning course, it explains how regression can be used within computer vision. Unlike `classification` tasks, which are used to make a categorical predicions (i.e. *cat or dog*), `regression` is used to find a continuous numerical values. This fits for our example, because we are trying to determine the `x, y` pixel locations of the dart point. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/biwi_regression.JPG?raw=true\" width=70%\u003e\n\u003c/p\u003e\n\nI spent a fair amount of time trying to get this regression based `cnn_learner` working – but it simply failed to converge during training. I was surprised that the model couldn't accurately predict the validation data given the simplicity of some of the samples. For example, the third image down was a picture of a dart, by itself, on a plain background – and it still couldn't make a reasonable prediction.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/regression_training_problems.JPG?raw=true\" width=60%\u003e\n\u003c/p\u003e\n\nI knew something must be wrong either with the training process and architecture, or there was a limitation with the dataset I had created. In order to isolate the problem I generated a new dataset.\n\n### 2. Synthetic Dataset with `Blender`\n\nUsing blender I simulated a simplified version of the problem. With this I could determine whether it was my training database that was wrong, or whether it was my ML/Dl approach that was wrong. I downloaded a 3d model from [grabcad](https://grabcad.com/library/simple-dart-1-0-1) and created a python script to automate rendering. \n\n  :bookmark_tabs: [blender_script.py](https://github.com/lukexyz/FlightVision/blob/master/blender/blender.py)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/blender_generation.JPG?raw=true\" width=50%\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/blender_output.JPG?raw=true\" width=50%\u003e\n\u003c/p\u003e\n\nBlender allows for scripting with python. It is quick to learning the scripting API, because when you create an event with the UI the equivalent python command appears in the console, which you can then paste directly into a script. Then with python you can add a loop around it – specify the inputs/outputs, and the whole process is automated. My blender script randomised the rotation, zoom, light position, and centroid `x, y` location. With these in place I generated 15,000 renders in about half an hour. \n\n##### 📋 Results\nWhat I found was that even with this much simplified computer vision problem, the `cnn_learner` with regression head still couldn't converge (and overfit) the training dataset. With that I was convinced the problem was with the architecture, so I moved on to try something else. \n\n### 3. Learning with `Unets` \n\n`Unets` are a segmentation method, where the output of the model is a per-pixel classification mask of the original image. These are commonly used in examples of self-driving car solutions online. My intuition here was that instead of using a multi-classification class of scenery objects (like `tree`, `road`, `traffic_light` etc), the target segmentation mask could simply be `0` or `1`, where `0` is a pixel where the centroid isn't, and `1` is a pixel where the centroid is. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/training_segmentation.JPG?raw=true\" width=70%\u003e\n\u003c/p\u003e\n\n### 4. Applying learnings to synthetic dataset\n\nUpon observing that the novel architecture significantly enhanced accuracy by an order of magnitude, I fully committed to exploring this approach in my manually sourced data.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/synthetic_with_unets.png?raw=true\" width=70%\u003e\n\u003c/p\u003e\n\n\n### 5. Back to the real dataset with the new architecture\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/back_to_real_dataset.png?raw=true\" width=70%\u003e\n\u003c/p\u003e\n\nIncredibly this new technique was now giving me about `99%` accuracy, and only failing on sample images which had heavy blurs across the photos, or obstructed views – which is a common problem of data quality. It's not something I'm interesting in solving immediately – when the other option is to ignore those frames and make a prediction when you see a high fidelity shot. \n\n\n### 6. Final Performance of real-world model..\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lukexyz/FlightVision/blob/master/media/final_inference.png?raw=true\" width=70%\u003e\n\u003c/p\u003e\n\nLike the video shown at the top of the article, the algorithm can accurately predict the point of a dart down to sub-pixel scales 🎯 It does have some failure points – like when the camera is moving to fast (causing blur) or I suppose when there is some major out-of-sample examples. But I took considerable joy in in this project branching out into different fields, to explore the limits of what was possible (synethic image generation with blender) – some of which were a dead end (regression analysis for a CV coordinate). I hope this project has been of interest to you. \n\n\n## Link to the main notebook file 📒\n  📝 [FlightVision_Unet_c950.ipynb.ipynb](https://github.com/lukexyz/FlightVision/blob/master/nbs/16fastai2_FlightVision_Unet_c950.ipynb)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukexyz%2Fflightvision","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukexyz%2Fflightvision","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukexyz%2Fflightvision/lists"}