{"id":16360094,"url":"https://github.com/frgfm/sdcnd-p4-behavioral-cloning","last_synced_at":"2026-04-12T00:38:30.467Z","repository":{"id":110056639,"uuid":"242542294","full_name":"frgfm/sdcnd-p4-behavioral-cloning","owner":"frgfm","description":"Behavioral cloning project of Udacity Self-Driving Car Engineer","archived":false,"fork":false,"pushed_at":"2020-03-02T14:57:23.000Z","size":16569,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-29T17:41:39.324Z","etag":null,"topics":["deep-learning","flask","keras","opencv","python","self-driving-car","tensorflow","unity3d"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/frgfm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-23T15:33:13.000Z","updated_at":"2022-12-25T05:35:58.000Z","dependencies_parsed_at":"2023-05-30T04:45:35.040Z","dependency_job_id":null,"html_url":"https://github.com/frgfm/sdcnd-p4-behavioral-cloning","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frgfm%2Fsdcnd-p4-behavioral-cloning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frgfm%2Fsdcnd-p4-behavioral-cloning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frgfm%2Fsdcnd-p4-behavioral-cloning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frgfm%2Fsdcnd-p4-behavioral-cloning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/frgfm","download_url":"https://codeload.github.com/frgfm/sdcnd-p4-behavioral-cloning/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239727060,"owners_count":19687097,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","flask","keras","opencv","python","self-driving-car","tensorflow","unity3d"],"created_at":"2024-10-11T02:10:32.709Z","updated_at":"2025-12-31T20:30:21.275Z","avatar_url":"https://github.com/frgfm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Driving behavior cloning\n\n[![License](https://img.shields.io/badge/License-MIT-brightgreen.svg)](LICENSE)[![Codacy Badge](https://api.codacy.com/project/badge/Grade/8db71da97dd04c359a9f438aa3ae3da2)](https://www.codacy.com/manual/frgfm/sdcnd-p4-behavioral-cloning?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=frgfm/sdcnd-p4-behavioral-cloning\u0026amp;utm_campaign=Badge_Grade) [![CircleCI](https://circleci.com/gh/frgfm/sdcnd-p4-behavioral-cloning.svg?style=shield)](https://circleci.com/gh/frgfm/sdcnd-p4-behavioral-cloning) [![codecov](https://codecov.io/gh/frgfm/sdcnd-p4-behavioral-cloning/branch/master/graph/badge.svg)](https://codecov.io/gh/frgfm/sdcnd-p4-behavioral-cloning)\n\nThis repository is an implementation of a simulated driving behavior for the behavioral cloning project of Udacity Self-Driving Car Nanodegree (cf. [repo](\u003chttps://github.com/udacity/CarND-Behavioral-Cloning-P3\u003e)).\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"static/images/autonomous_01_short.gif\"\u003e\n\u003c/p\u003e\n\n\n\n## Table of Contents\n\n- [Getting Started](#getting-started)\n  - [Prerequisites](#prerequisites)\n  - [Installation](#installation)\n- [Usage](#usage)\n- [Approach](#Approach)\n- [Credits](#credits)\n- [License](#license)\n\n\n\n## Getting started\n\n### Prerequisites\n\n- Python 3.6 (or more recent)\n- [pip](https://pip.pypa.io/en/stable/)\n- [Unity3D](https://unity3d.com/get-unity/download)\n\n### Installation\n\n#### Python\n\nYou can install the project requirements as follows:\n\n```shell\ngit clone https://github.com/frgfm/sdcnd-p4-behavioral-cloning.git\ncd sdcnd-p4-behavioral-cloning\npip install -r requirements.txt\n```\n\nIf you do not plan on collecting data yourself, feel free to use the sample driving [data](https://d17h27t6h515a5.cloudfront.net/topher/2016/December/584f6edd_data/data.zip) provided by Udacity. Extract it in the repo folder to obtain a folder hierarchy as follows:\n\n```\n.\n├── data\n│   ├── driving_log.csv\n│   ├── IMG\n├── LICENSE\n├── README.md\n├── requirements.txt\n├── src\n│   ├── dataset.py\n│   ├── drive.py\n│   ├── models.py\n│   ├── train.py\n│   └── video.py\n├── static\n│   └── images\n└── test\n    ├── test_dataset.py\n    └── test_models.py\n```\n\n\n\nIf you wish to collect your own training data, run your environment build, select a track and click on  `Training mode`. Press the `R` key to choose an output folder (`./data` is much recommended to be able to use default script arguments). Now you can play with the environment, just press `R` again to start your recording, and `R` again to stop it.\n\nYou can download the trained [model](https://github.com/frgfm/sdcnd-p4-behavioral-cloning/releases/download/v0.1.0/model.h5) (put the checkpoint in the `./data` folder).\n\n*Note: if you wish to use CPU only, replace the `tensorflow-gpu` dependencies with `tensorflow`*\n\n\n\n#### Unity\n\nAfter installing Unity3D, you will need an environment build to run the simulation. Download the appropriate build for your OS and extract it:\n\n- [Linux](https://s3-us-west-1.amazonaws.com/udacity-selfdrivingcar/Term1-Sim/term1-simulator-linux.zip)\n- [Mac](https://s3-us-west-1.amazonaws.com/udacity-selfdrivingcar/Term1-Sim/term1-simulator-mac.zip)\n- [Windows](https://s3-us-west-1.amazonaws.com/udacity-selfdrivingcar/Term1-Sim/term1-simulator-windows.zip)\n\nIf you encounter an issue with the above builds, please refer to the \"Available Game Builds\" section of this [readme](https://github.com/udacity/self-driving-car-sim).\n\n\n\n## Usage\n\n### Training\n\nA training script is available to train the steering angle regressor on the training dataset.\n\n```\nusage: train.py [-h] [--arch ARCH] [--folder FOLDER] [--batch-size BATCH_SIZE]\n                [--lr LR] [--size SIZE] [--output OUTPUT]\n                [--correction CORRECTION] [--dropout DROPOUT]\n                [--resampling RESAMPLING]\n                epochs\n\nDriving behavior cloning training\n\npositional arguments:\n  epochs                Number of epochs to train\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --arch ARCH           Model architecture (default: babypilot)\n  --folder FOLDER       Path to data folder (default: ./data)\n  --batch-size BATCH_SIZE\n                        Batch size (default: 32)\n  --lr LR               Learning rate (default: 0.0005)\n  --size SIZE           Image resizing (default: 64)\n  --output OUTPUT       Path where trained model will be saved (default:\n                        model.h5)\n  --correction CORRECTION\n                        Default steering correction (default: 0.25)\n  --dropout DROPOUT     Dropout probability (default: 0.0)\n  --resampling RESAMPLING\n                        Probability of keeping samples equals to mode\n                        (default: 0.5)\n```\n\nThe training script includes a Tensorboard callback. You can access your live training information by running tensorboard:\n\n```bash\ntensorboard --logdir=logs --port=6007\n```\n\nand navigate in your browser to http://localhost:6007/.\n\n#### Example with provided model training\n\nRun the training script specifying the selected architecture, the output file and the training hyperparameters:\n\n```bash\npython src/train.py 20 --output ./model.h5 --arch babypilot --lr 5e-4 --resampling 0.5 --dropout 0.3\n```\n\n*Note: the above command will train for 20 epochs the `babypilot` architecture using a starting learning rate of `5e-4`, a dropout rate of `0.3` on the dataset where 50% of the 0° steering samples are removed, and save the checkpoint in `./model.h5`. For further details, please refer to the [Approach](#Approach) section.*\n\n\n\n### Test\n\nAn inference script is available for you to test out your trained model or the one from the release on your environment build.\n\n```\nusage: drive.py [-h] [--arch ARCH] [--size SIZE] [--speed SPEED]\n                [--img-folder [IMG_FOLDER]]\n                model\n\nAutonomous driving simulation\n\npositional arguments:\n  model                 Path to model h5 file. Model should be on the same\n                        path.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --arch ARCH           Architecture used for simulation. (default: babypilot)\n  --size SIZE           Image resizing (default: 64)\n  --speed SPEED         Max allowed speed. (default: 30)\n  --img-folder [IMG_FOLDER]\n                        Path to image folder. This is where the images from\n                        the run will be saved. (default: None)\n```\n\n\n\nPass the trained model checkpoint to this script, which will run a flask app with your model. Now run your environment build *(Screen resolution: 800 x 600, Graphics Quality: Fastest)*, select a track and click on `Autonomous mode`.\n\n#### Example with the provided model\n\nIn order to use the repository's owner checkpoint, you can download it manually from the release's attachment or with command-line:\n\n```bash\ncd sdcnd-p4-behavioral-cloning\nwget https://github.com/frgfm/sdcnd-p4-behavioral-cloning/releases/download/v0.1.0/model.h5\n```\n\nThen expose the model on a flask application by running the driving script:\n\n```bash\npython src/drive.py ./model.h5\n```\n\n \n\n## Approach\n\nThis project has the particularity of using simulated data for training and evaluation, which easily solves the costs of collecting and annotating the data.\n\n![simulator](https://video.udacity-data.com/topher/2017/March/58dc749a_sim-image/sim-image.png)\n\nInstead of having a perfect driving behavior in our dataset, the user controls the simulator to adopt a reasonable driving. The collected data is retrieved directly from the controls, and the virtual visual inputs from the environment.\n\n\n\n### Environment\n\nThis Unity environment offers a car driving experience on two tracks with two available modes. \n\n| Track 01                                | Track 02                                |\n| --------------------------------------- | --------------------------------------- |\n| ![track_01](static/images/track_01.gif) | ![track_02](static/images/track_02.gif) |\n\nThe training mode lets the user explore the environment and record its driving data for later training. The autonomous mode lets the developer plug in a flask app with a model that would handle the car controls.\n\nPlease refer to this [repository](https://github.com/udacity/self-driving-car-sim) for further details.\n\n\n\n### Data collection strategies\n\nEven though the data is simulated, in order to retrieve quality annotated data, a human input is required, which makes data collection time-bounded. To avoid driving hours on the simulator, the environment lets us pick three visual point of views (left, center and right cameras).\n\n\n\n![3cams](https://video.udacity-data.com/topher/2017/February/58a50a30_carnd-using-multiple-cameras/carnd-using-multiple-cameras.png)\n\n\n\nSince we only retrieve for each frame the steering angle of the car (which applies to the center camera), in order to use the two other inputs, a small steering correction has to be made as shown above.\n\nAfter several tests, good results seemed to be obtained using a 0.25° steering correction (+0.25, 0, -025).\n\n\n\n### Dataset\n\nThe provided dataset includes 8036 samples with 3 different angles of view. Using the steering correction mentioned above, the total usable images of the dataset reaches 24 108 images and steering angles.\n\n|                | Left camera                             | Center camera                               | Right camera                              |\n| -------------- | --------------------------------------- | ------------------------------------------- | ----------------------------------------- |\n| Original image | ![left_raw](static/images/left_raw.jpg) | ![center_raw](static/images/center_raw.jpg) | ![right_raw](static/images/right_raw.jpg) |\n\n\n\n#### Dataset distribution\n\n![data_distribution](/home/fg/Documents/Udacity/sdcnd-p4-behavioral-cloning/static/images/data_distribution.png)\n\nThe angle distribution is highly imbalanced as shown above, despite the steering correction trick. This might be a problem during training as the distribution shows that very small steering angles are predominant. \n\n**54% of samples have a 0° steering angle**. Thus the model will most likely learn on straight line driving, while the trickiest part would be curves.\n\nIn order to tackle this, we will simply introduce a probability $p$ of taking 0° steering samples. By experimenting with this parameter during training, the value `p=0.5` was selected.\n\n\n\n#### Preprocessing\n\n The data preprocessing will include:\n\n- cropping: consider only the relevant part in the image (sky and hood parts of the image do not help).\n- resizing: high resolution input is not necessarily here.\n- augmentation: random horizontal flip, and shadowing will be used to improve generalization.\n- normalization: feeding normalized data to the model even on new data distribution *(with the mean and standard deviation of the training set)*. \n\n\n\n|                           | Left camera                                         | Center camera                                           | Right camera                                          |\n| ------------------------- | --------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------- |\n| Original image            | ![left_raw](static/images/left_raw.jpg)             | ![center_raw](static/images/center_raw.jpg)             | ![right_raw](static/images/right_raw.jpg)             |\n| Cropped and resized image | ![left_processed](static/images/left_processed.png) | ![center_processed](static/images/center_processed.png) | ![right_processed](static/images/right_processed.png) |\n| Augmented image           | ![left_augmented](static/images/left_augmented.png) | ![center_augmented](static/images/center_augmented.png) | ![right_augmented](static/images/right_augmented.png) |\n\n\n\n### Architecture\n\nFor this project, several architectures were explored to end up with a regression model being both light and high performing. Here are the different architectures that were explored:\n\n#### LeNet5\n\n[LeNet5](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf) architecture was designed for simple image classification tasks initially, but here we repurposed it for regression adding dropout to the inputs of hidden fully connected layers. The key differences are that here images are not in grayscale (unlike [MNIST](http://yann.lecun.com/exdb/mnist/)), but the provided dataset shows low graphics / simple visual information. Resizing the input to 64x64 ended up giving encouraging results.\n\n![lenet5](https://www.researchgate.net/profile/Vladimir_Golovko3/publication/313808170/figure/fig3/AS:552880910618630@1508828489678/Architecture-of-LeNet-5.png)\n\n*Source: Gradient-based learning applied to document recognition (1998)*\n\nDuring training experiments, some issues were noticed: the low number of convolutional layers ends up providing the network with very simple representation of spatial features. Even though the data holds less spatial information than actual pictures, it remains much richer than grayscale digits. During the test run in the simulator, we end up with a non-smooth behavior resulting in the car getting out the track a bit later.\n\n#### DAVE-2 System\n\nAs suggested by project instructions, NVIDIA released in 2016 a light architecture to control steering of the self-driving car in near real-time (10 FPS).\n\n![dave-2](https://devblogs.nvidia.com/parallelforall/wp-content/uploads/2016/08/cnn-architecture-624x890.png)\n\n\n\nCompared to our problem, the input data is much richer spatially and more complex. But it is trained for the same target task. Since the spatial complexity of the input data can be adapted using resizing, this candidate was bearing a lot of hopes in this project.\n\nWithout much suspense, it performs quite well after ~15-20 epochs of training. But for the sake of experimentation, we will focus on a custom architecture leveraging the learnings of those two architectures.\n\n#### BabyPilot\n\nLet us start with what we learned from the previous architectures:\n\n- for the steering regression on simulated data, we need to adapt the input data by\n  - Cropping: down to the section of the image visually relevant to steering, \n  - Resizing: DAVE-2 was fed with 66x200 pictures, our data being much simpler spatially, it would be beneficial to consider smaller sizes than this.\n- Spatial features extraction\n  - the number of features required is most likely similar to DAVE-2 as the task is similar, meaning that we can adopt a similar channel expansion strategy in convolutional layers.\n  - the complexity of spatial features is lower than with DAVE-2 but higher than MNIST, meaning that we will most likely have a number of convolutional layers between that of DAVE-2 and LeNet5.\n- Number of parameters\n  - having too big spatial features on your last convolutional layers may cost us a lot in fully connected layers' size, so we will consider using [global average pooling](https://arxiv.org/abs/1312.4400) instead of flattening. Alternatively, we will keep the number of hidden features low to keep the model's size reasonable.\n\n\n\nBy using the previous observations and assumptions, we implement our final model:\n\n\n\n```\nModel: \"sequential\"\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\nlambda (Lambda)              (None, 64, 64, 3)         0         \n_________________________________________________________________\nconv2d (Conv2D)              (None, 62, 62, 16)        448       \n_________________________________________________________________\nmax_pooling2d (MaxPooling2D) (None, 31, 31, 16)        0         \n_________________________________________________________________\nconv2d_1 (Conv2D)            (None, 29, 29, 32)        4640      \n_________________________________________________________________\nmax_pooling2d_1 (MaxPooling2 (None, 14, 14, 32)        0         \n_________________________________________________________________\nconv2d_2 (Conv2D)            (None, 12, 12, 64)        18496     \n_________________________________________________________________\nmax_pooling2d_2 (MaxPooling2 (None, 6, 6, 64)          0         \n_________________________________________________________________\nflatten (Flatten)            (None, 2304)              0         \n_________________________________________________________________\ndropout (Dropout)            (None, 2304)              0         \n_________________________________________________________________\ndense (Dense)                (None, 128)               295040    \n_________________________________________________________________\ndropout_1 (Dropout)          (None, 128)               0         \n_________________________________________________________________\ndense_1 (Dense)              (None, 64)                8256      \n_________________________________________________________________\ndropout_2 (Dropout)          (None, 64)                0         \n_________________________________________________________________\ndense_2 (Dense)              (None, 16)                1040      \n_________________________________________________________________\ndense_3 (Dense)              (None, 1)                 17        \n=================================================================\nTotal params: 327,937\nTrainable params: 327,937\nNon-trainable params: 0\n```\n\nYou will notice that the number of parameters is much lower than the 2.9M of DAVE-2, and that global average pooling was not used here. The reason behind this comes only from a practical perspective, as keras [implementation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GlobalAveragePooling2D) does not allow other output shapes than 1x1 (it flattens the tensor right away).\n\n\n\n### Training sequence\n\nThe training sequence is handled by the `train.py` script as follows:\n\n- optimizer: [Adam](https://arxiv.org/abs/1412.6980) *(commonly used for vision tasks)*\n- loss: MSE loss *(common loss for regression)*\n- batch size: 32 *(selected based on GPU RAM capacity)*\n- epochs: 20\n- learning rate scheduler: reduce on plateau\n- starting learning rate: 5e-4 (to speed up convergence and avoid overfitting)\n- Dropout rate: 0.3 *(reduces overfitting without extending the number of epochs too much)*\n\n\n\nThe training procedure yields very positive results with a final training loss of 0.04394 and validation of 0.04402.\n\n![training_monitoring](static/images/training_monitoring.png)\n\nThe MSE loss curves show a good generalization capacity of the model, without any apparent overfitting. The model parameters' values were saved [here](https://github.com/frgfm/sdcnd-p4-behavioral-cloning/releases/download/v0.1.0/model.h5).\n\n\n\n### Results\n\nThe trained [model](https://github.com/frgfm/sdcnd-p4-behavioral-cloning/releases/download/v0.1.0/model.h5) is able to drive around the first track without any issue (Resolution: `800 x 600`, Graphics Quality: `Fastest` ) especially after removing 50% of the 0° steering angle in the training phase.\n\n![autonomous_run](static/images/autonomous_01_short.gif)\n\nFull-length lap recordings in first-person and third-person view are available for download in the release attachments:\n\n- [First Person View](https://github.com/frgfm/sdcnd-p4-behavioral-cloning/releases/download/v0.1.0/autonomous_01_fpv.mp4)\n- [Third Person View](https://github.com/frgfm/sdcnd-p4-behavioral-cloning/releases/download/v0.1.0/autonomous_01_tpv.mp4)\n\n\n\n## Limitations and improvements\n\n### Limitations\n\nDuring tests in autonomous mode, a few issues were spotted:\n\n- track overfitting: training the model on one track only makes evaluation hard on a different track. Color augmentation might help.\n- track recovery: when putting the car already outside of the track, the model is rarely able to recover even in close proximity.\n- graphics overfitting: this one is a very interesting lead for augmentation. For report render purposes, different graphics and resolution were tested with the trained model. While the resolution does not change much (because of resizing), the graphics settings influence a lot the performances.\n\n![graphics_fail](static/images/autonomous_01_graphicsfail.gif)\n\nSince the provided dataset seemed to have been recorded on `Graphics Quality: Fastest`, best performances are obtained when evaluating in the same mode. Apparent failures appear very quickly if you were to use `Graphics Quality: Fanstastic` instead as shown above. Knowing this, while driving the simulator, it would be interesting to record for each cameras the renders in different graphics settings.\n\n\n\n### Improvements\n\nA few ideas to explore for next versions:\n\n- **learned task**: the current input is a single frame, using also the past frames could bring interesting results exploring either RNN or 3D convolutions. A different approach could also be taken using reinforcement learning where the action space is the steering angle.\n- **data augmentations**: on top of the environment flexibility, domain augmentation could be applied using a GAN approach.\n- **transfer learning**: using the model trained on this environment, trying to perform the same task on higher-end graphics game would be an interesting project.\n- **normalization**: compute training dataset statistics and use them for normalization for easier convergence.\n\n\n\n## Credits\n\nThis implementation is vastly based on the following methods:\n\n- [Gradient-based Learning Applied to Document Recognition](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf)\n- [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf)\n- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Coveriate Shift](https://arxiv.org/pdf/1502.03167.pdf)\n- [End to End Learning for Self-Driving Cars](https://arxiv.org/pdf/1604.07316.pdf)\n\n\n\n## License\n\nDistributed under the MIT License. See `LICENSE` for more information.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrgfm%2Fsdcnd-p4-behavioral-cloning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrgfm%2Fsdcnd-p4-behavioral-cloning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrgfm%2Fsdcnd-p4-behavioral-cloning/lists"}