{"id":16843051,"url":"https://github.com/zanussbaum/comma-speedchallenge","last_synced_at":"2026-04-16T17:37:13.448Z","repository":{"id":89761950,"uuid":"276467821","full_name":"zanussbaum/comma-speedchallenge","owner":"zanussbaum","description":"My attempt at the comma.ai Speed Challenge","archived":false,"fork":false,"pushed_at":"2020-08-31T21:15:03.000Z","size":20359,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-24T12:27:44.621Z","etag":null,"topics":["deep-learning","machine-learning","optical-flow","self-driving","self-driving-car","tensorflow","transfer-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zanussbaum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-01T19:45:44.000Z","updated_at":"2020-08-31T21:15:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"4d2df963-caee-4a2c-a53e-24366512d71b","html_url":"https://github.com/zanussbaum/comma-speedchallenge","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zanussbaum%2Fcomma-speedchallenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zanussbaum%2Fcomma-speedchallenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zanussbaum%2Fcomma-speedchallenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zanussbaum%2Fcomma-speedchallenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zanussbaum","download_url":"https://codeload.github.com/zanussbaum/comma-speedchallenge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244166640,"owners_count":20409177,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","machine-learning","optical-flow","self-driving","self-driving-car","tensorflow","transfer-learning"],"created_at":"2024-10-13T12:49:27.369Z","updated_at":"2025-10-15T23:06:31.768Z","avatar_url":"https://github.com/zanussbaum.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# comma-speedchallenge\nThis is my attempt at the comma.ai Speed Challenge. Here is my brief overview of my approach and results.\n\n## Background\nAs illustrated by [Ryan](https://github.com/ryanchesler/comma-speed-challenge), there are many different approaches to this problem. Optical flow and LSTMs seem to be a common approach. Many of the models seem to overfit, due to lack of training data (there's only 20400) frames in total, so training a model from scratch doesn't seem very feasible. A good alternative is to utilize [transfer learning](https://cs231n.github.io/transfer-learning/).\n\nThe training video contains 20400 frames with corresponding speeds for each frame. The test video contains 10798 frames. Both videos contain a mixture of highway and residential driving.\n\n![speed_time](figs/speed_time.png)\n\nThe first half (or so) is the highway driving where the next is the residential driving (with stops and traffic). \n\n![speed_freq](figs/speed_freq.png)\n\nOne obvious thing that sticks out to me is that the frequency of speeds above 25 is very, very low. We will probably have a hard time predicting speeds above 25 if we don't have enough examples.\n\n### Optical Flow\n[Optical Flow](https://en.wikipedia.org/wiki/Optical_flow) is a computer vision technique used to delineate the apparent motion between two images by comparing the movement of points between two images. It has many applications to [self-driving](https://nanonets.com/blog/optical-flow/) and even has been used in some [reinforcement learning](https://arxiv.org/pdf/1901.03162.pdf).  \n\n![flow](figs/flow.jpg)\n\n### Transfer Learning\nTransfer learning has been shown to be a great tool for domain transfers by utilizing the generic features extracted from a CNN trained on [imagenet](https://arxiv.org/abs/1403.6382). Transfer learning is especially important in settings where we may not have enough data and we can leverage the training of other models so we don't have to do all the hard work (i.e. train a model for weeks).\n\n## Approach\nMy approach was to use optical flow from two consecutive frames to predict the mean speed between the two frames. Instead of applying the optical flow on the raw images, I first applied a Canny Edge Detector, in an attempt to filter out the noise and hopefully pick up some useful information (such as the lane markings). Given that the speed doesn't change very quickly between frames, the mean seems appropriate here. These frames would be fed into a pretrained CNN which would predict the speed of the frames.  \n\nWe also applied random augmentations (brightness and hue) as way to combat overfitting.\nWe trained two models: one to predict the log speed and one to predict the regular speed\n\n### Architecture\nThe architecture that I used is shown below, where model is MobileNet. I took the first 37 layers and used those as a fixed feature extractor. I used Adam optimizer (lr=3e-4) and L2 regularization(.01). \n![arch](figs/architecture.png)\n\n\n### Things That Make this Problem Hard!\nOne of the first thing that I noticed that I thought would be challenging was scarcity of data. Like I stated above, since we don't have tons of data, we need to be smart about augmentation and making sure that we don't totally overfit.\n\nAnother thing that jumped out at me was the distribution of speeds. It didn't appear to follow any distribution, at least one that I know of (excuse my poor probability skills). One thing that especially stood out to me about this was the lack of examples where the speed was greater than 25. There are very few examples and it seemed unlikely we would ever predict that speed well.\n\n## Results\nAfter predicting, since the variance of the speed frame to frame can't be incredibly different, we apply an EMA to smooth the predictions. This allows our predictions to have a little more error when predicting. We use a window size of 10 as that showed the greatest performance on the training set.  \n\n\u003ctable\u003e\n\u003ctr\u003e\n    \u003ctd\u003e \u003cimg src='figs/normal_predictions.png' widht='200' height='250'/\u003e\n    \u003ctd\u003e \u003cimg src='figs/predictions.png' widht='200' height='250'/\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n    \u003ctd\u003e \u003cp\u003eNormal Predictions\u003c/p\u003e\n    \u003ctd\u003e \u003cp\u003eLog Predictions\u003c/p\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nOne thing that stood out to me was that the normal predictions had a much tighter band around higher speeds, whereas the log predictions had a much tighter band of predictions around the lower speeds. Knowing this, I thought maybe averaging the predictions of the two models would help increase the performance. \n\nHere are the errors per bucketed speed on the training set.\n\n\n\u003ctable\u003e\n\u003ctr\u003e\n    \u003ctd\u003e \u003cimg src='figs/normal_rolling_mse_per_speed.png' widht='200' height='250'/\u003e\n    \u003ctd\u003e \u003cimg src='figs/smoothed_mse_bucketed.png' widht='200' height='250'/\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n    \u003ctd\u003e \u003cp\u003eNormal Predictions\u003c/p\u003e\n    \u003ctd\u003e \u003cp\u003eLog Predictions\u003c/p\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e \u003cimg src='figs/averaged_mse_bucketed.png'/\u003e\n  \u003ctd\u003e \u003cimg src='figs/averaged_preds_train.png'/\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n    \u003ctd\u003e \u003cp\u003eAveraged Predictions\u003c/p\u003e\n    \u003ctd\u003e \u003cp\u003ePredicted Speed vs Time\u003c/p\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### Things I Tried (and Failed)\n* Offline Data Augmentation\n    * I tried manually increasing the data size by performing different data augmentation techniques (brightness, hue, zoom, flip, etc.) but results never were good.\n    * Remember when doing this to not introduce data leakage! (Split train and validation before augmentation)\n* Using the full network as feature extractor\n    * Since the data is so much different than ImageNet, this wasn't going to work well\n\n## Next Steps\nMany people have talked about using LSTMs and this seems like an appropriate approach.\n\n\nOne thing I would have liked to try is how well different networks work, whether they are other networks trained on ImageNet or Optical Flow networks, such as [FlowNet](https://paperswithcode.com/paper/flownet-20-evolution-of-optical-flow).\n\nAdditionally, semantic segmentation seems to be another popular suggestion of things to try more. I would be interested to see if using a pretrained Pix2Pix implementation on Cityscapes, you could extract only the road of the lane and then calculate the flow of the masked image.\n## Conclusion\nAfter smoothing and averaging, we were able to achieve a .81 MSE on the training set. We tried to combat overfitting through augmentation and L2 weight regularization, but there is certainly a possibility we still overfit, given the unique distribution of speeds.  \n\n\nTransfer learning is an incredible tool, but learning how to [augment](https://towardsdatascience.com/when-conventional-wisdom-fails-revisiting-data-augmentation-for-self-driving-cars-4831998c5509) data to better suit your problem is important as well!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzanussbaum%2Fcomma-speedchallenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzanussbaum%2Fcomma-speedchallenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzanussbaum%2Fcomma-speedchallenge/lists"}