{"id":13936436,"url":"https://github.com/sagar448/Self-Driving-Car-3D-Simulator-With-CNN","last_synced_at":"2025-07-19T22:30:28.965Z","repository":{"id":39708560,"uuid":"103585741","full_name":"sagar448/Self-Driving-Car-3D-Simulator-With-CNN","owner":"sagar448","description":"Implementing a self driving car using a 3D Driving Simulator. CNN will be used for training","archived":false,"fork":false,"pushed_at":"2020-10-19T01:34:25.000Z","size":45280,"stargazers_count":163,"open_issues_count":2,"forks_count":54,"subscribers_count":19,"default_branch":"master","last_synced_at":"2024-08-08T23:23:39.811Z","etag":null,"topics":["autonomous","cnn","keras","opencv","python","self-driving-car"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sagar448.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-14T21:57:30.000Z","updated_at":"2024-07-06T00:26:23.000Z","dependencies_parsed_at":"2022-08-24T01:31:32.019Z","dependency_job_id":null,"html_url":"https://github.com/sagar448/Self-Driving-Car-3D-Simulator-With-CNN","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sagar448%2FSelf-Driving-Car-3D-Simulator-With-CNN","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sagar448%2FSelf-Driving-Car-3D-Simulator-With-CNN/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sagar448%2FSelf-Driving-Car-3D-Simulator-With-CNN/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sagar448%2FSelf-Driving-Car-3D-Simulator-With-CNN/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sagar448","download_url":"https://codeload.github.com/sagar448/Self-Driving-Car-3D-Simulator-With-CNN/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226686719,"owners_count":17666928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous","cnn","keras","opencv","python","self-driving-car"],"created_at":"2024-08-07T23:02:40.228Z","updated_at":"2024-11-27T04:31:05.746Z","avatar_url":"https://github.com/sagar448.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Self-Driving-Car-3D-Simulator-With-CNN\n\u003cp\u003e\n  \u003cimg align=\"left\" width=\"425\" height=\"400\" src=\"https://github.com/sagar448/Self-Driving-Car-3D-Simulator-With-CNN/blob/master/src/3D%20Car%20Simulator.png\"\u003e\n  \u003cimg align=\"right\" width=\"420\" height=\"400\" src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/0/0a/Python.svg/2000px-Python.svg.png\"\u003e\n\u003c/p\u003e\n\u003cp\u003e\n  \u003cimg align=\"center\" width=\"600\" height=\"7\" src=\"http://getthedrift.com/wp-content/uploads/2015/06/White-Space.png\"\u003e\n\u003c/p\u003e\n\n## Introduction\n\u003cp align=\"center\"\u003e\n  Self Driving car after 50 epochs of training\u003cbr\u003e\n  \u003cimg width=\"400\" height=\"300\" src=\"https://github.com/sagar448/Self-Driving-Car-3D-Simulator-With-CNN/blob/master/src/SelfDrivingAfter50Epochs.gif\"\u003e\n\u003c/p\u003e\n\nSome point in our life as programmers we all wonder how a self driving car is actually programmed. I went through the same phase and so here it is, a very simple DIGITAL self driving car controlled using Python with a Reinforcement Q-learning algorithm as well as a Convolutional Neural Network.\n\nYou can essentially apply this to any game, the algorithm can be adapted and the reward rules can be changed to allow for different outcomes. As we go through the code I will explain step by step what each line does and once you've mastered it you can go ahead fork the code and do as you wish.\n\n```\nNote: You need to have sufficient knowledge about Reinforcment learning before progressing, this tutorial \nonly explains the code it does not go into the theoretical details\nThe links below help explain the theoretical details as well as other details I had problems with:\n\nhttps://stats.stackexchange.com/questions/221402/understanding-the-role-of-the-discount-factor-in-reinforcement-learning\nhttp://mnemstudio.org/path-finding-q-learning-tutorial.htm\nhttps://yanpanlau.github.io/2016/07/10/FlappyBird-Keras.html\nhttps://keon.io/deep-q-learning/\nhttps://en.wikipedia.org/wiki/Q-learning\nhttps://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0\n```\n\nWe will be using Keras to make the magic happen with Tensorflow backend. Assuming you are familiar with Keras and Tensorflow and have them installed we can start!\n\n```\nNote: Check my other gits for brief explanation on Keras and other simple algorithms such as the CNN \nand RNN if you are unfamiliar with Keras/Tensorflow!\n```\n\n## My Setup\n\nIn order to detect lanes, we need to send game frames to the algorithm for processing. I used a library called [mss](https://pypi.python.org/pypi/mss/)(MultipleScreenShot), it allows the users to take quick screenshots with almost minimal effect in FPS.\nUnfortunately, it takes the screen shot of the entire screen if coordinates aren't specified, therefore in order to get game frames, the game needs to be properly positioned. \n\nThe picture below depicts my environment.\n\n\u003cp align=\"center\"\u003e\n  Layout as displayed on my screen\n  \u003cimg width=\"800\" height=\"600\" src=\"https://github.com/sagar448/Self-Driving-Car-3D-Simulator-With-CNN/blob/master/src/Environment.png\"\u003e\n\u003c/p\u003e\n\u003cp\u003e\n  \u003cimg width=\"600\" height=\"4\" src=\"http://getthedrift.com/wp-content/uploads/2015/06/White-Space.png\"\u003e\n\u003c/p\u003e\nYou can set it up anyway you want but make sure to change the coordinates of the ScreenShot module so it only covers the game area.\n\nBefore we start the implementation it's a good idea to have the code open on the side as the comments have details you wouldn't want to miss.\n\n## Implementation\n\n### Imports\n```python\n1     import cv2\n2     import mss\n3     import numpy as np\n4     from keras.models import Sequential\n5     from keras.layers import Dense, Flatten\n6     from keras.optimizers import SGD\n7     from keras.layers.convolutional import Conv2D\n8     import pyautogui as p\n9     import random\n10    import time\n```\nWe start by importing a couple libraries. \nIn order, we import OpenCV, our Mss library, Numpy for computation, Keras for our CNN, Pyautogui to control our keyboard, Random and finally Time for delay purposes.\n\n### Detecting Lanes\n```python\n1     #Function calculates the lanes\n2     def CalculateLanes(OrgImage):\n3         errors = False\n4         #Since our game has yellow lanes, we can detect a specific color\n5         #keep that color, and get rid of everything else to make it easier\n6         #to detect the yellow lanes\n7         #So we convert our image to the HSL color scheme\n8         HSLImg = cv2.cvtColor(OrgImage, cv2.COLOR_BGR2HLS)\n9         #The lower and upper arrays define boundaries of the BGR color space\n10        #BGR because OpenCV represents images in Numpy in reverse order\n11        #So for our yellow color we say that our pixels color that are yellow will be\n12        # R\u003e= 100, B \u003e= 0, G\u003e=10 (lower limit), R\u003c=255, B\u003c=255, G\u003c=40\n13        lower = np.uint8([ 10,   0, 100])\n14        upper = np.uint8([ 40, 255, 255])\n15        #inRange basically finds the color we want in the HLSImg with the lower and upper\n16        #boundaries(the ranges)\n17        yellow_mask = cv2.inRange(HSLImg, lower, upper)\n18        #We then apply this mask to our original image, and this returns an image showing\n19        #only the pixels that fall in the range of that mask\n20        YellowImg = cv2.bitwise_and(OrgImage, OrgImage, mask=yellow_mask)\n21        #Convert the original image to gray\n22        GrayImg = cv2.cvtColor(YellowImg, cv2.COLOR_BGR2GRAY)\n23        #Apply blurring\n24        #The 5x5 is the gaussianblur kernel convolved with image\n25        #The 0 is the sigmaX and SigmaY standard deviation usually taken as 0\n26        blurredImg = cv2.GaussianBlur(GrayImg, (5, 5), 0)\n27        #Detect edges in the image\n28        #700 is the max val, any edges above the intensity gradient of 700 are edges\n29        #200 is the lowest intensity gradient, anything below is not an edge\n30        imageWithEdges = cv2.Canny(blurredImg, threshold1=200, threshold2=700)\n31        #These are the points of our trapezoid/hexagon that we crop out \n32        points = np.array([[0, 310],[0, 300], [220, 210], [380, 210], [600, 300], [600, 310]])\n33        #Now we calculate the region of interest\n35        #We first create a mask (a blank black array same size as our image)\n36        mask = np.zeros_like(imageWithEdges)\n37        #Then we fill the mask underneath the points(inside the polygon) we defined\n38        #with color white (255)(This is the part of the image we want to process)\n39        cv2.fillPoly(mask, [points], 255)\n40        #this bitwise and function basically combines the two images\n41        #The coloured bit where the pixels had a value 255 is kept while the\n42        #top bit is removed (which is 0)\n43        croppedImg = cv2.bitwise_and(blurredImg, mask)\n```\n**Line 2** Our parameter being our screen shot (OrgImage)\n\n**Line 3** We initialise our variable errors to False indicating that currently we have no errors produced\n\n**Line 8** The lanes in the game are yellow and so we convert our image to the HSL color space in order to enhance our lanes.\nThey were not very clear in the RGB space, therefore HSL was used.\n\n**Line 13\u002614** We define our upper and lower limit of the color space. Although the boundaries are given in terms of RGB it is actually in HSL. The comments are in RGB to make it easier to understand. Those limits represent the region where the color yellow falls within. Therefore, we use those limits so we can seek out a similar color.\n\n**Line 17** Now we apply the limits to our HSL image. It seeks out the color yellow and sets the rest of the pixels of the image to 0. We have essentially created a mask. An area where relevant pixels keep their values and the ones not needed are set to 0. \n\n**Line 20** The bitwise_and function basically takes a look at the pixel values and if the pixel value in the mask and the pixel value in the image have the same value they are kept, if they are different then it is set to 0. We are left with a image with only yellow region visible.\n\n\u003cp align=\"center\"\u003e\n  Yellow Image\u003cbr\u003e\n  \u003cimg width=\"400\" height=\"300\" src=\"https://github.com/sagar448/Self-Driving-Car-3D-Simulator-With-CNN/blob/master/src/YellowImg.png\"\u003e\n\u003c/p\u003e\n\n**Line 22** Now we can convert our image to grayscale. We do this in order to make the edge detection more accurate. The canny edge detection function used later on essentially measures the magnitude of pixel intensity changes. Therefore if we have colors that are similar to each other there isn't a big change in pixel intensity and it might not be considered an edge. Grayscale images are also less computation heavy.\n\n**Line 26** We now apply a gaussian blur. We do this in order to get rid of rough edges. Some realistic games or even in real life there are cracks on the road that might be considered something of interest so in order to get rid of the \"noisy edges\" we apply a blur.\n\n**Line 30** Now we finally apply the edge detection function. We have thresholds that identify what can and cannot be considered an edge.\n\n**Line 32** We don't want all the edges detected in the image. We only want those that concern the lanes. So we create a region of interest, a specific set of coordinates. \n\n**Line 36** We create an empty black mask with the same space dimension as our image.\n\n**Line 39** Anything around the polygon defined by our ROI is filled with black while the inside is filled with the color white (255).\n\n**Line 43** Finally we take our blurred image and we apply our mask to it. So the white region of our mask is replaced with our image while the rest is black (not used).\n\n\u003cp align=\"center\"\u003e\n  croppedImg\u003cbr\u003e\n  \u003cimg width=\"400\" height=\"300\" src=\"https://github.com/sagar448/Self-Driving-Car-3D-Simulator-With-CNN/blob/master/src/croppedImg.png\"\u003e\n\u003c/p\u003e\n\nGreat now we've managed to narrow down our edges to the region that we are interested in. Thats most of the processing done. We now want to get the appropriate lines and combine them into lanes. The next half of this function does exactly that.\n\n```python\n1         #Basically the accumulation of the most imperfect edges with the minimum\n2         #length being defined by 180\n3         #Thickness of the lines is 5\n4         lines = cv2.HoughLinesP(croppedImg, 1, np.pi/180, 180, np.array([]), 180, 5)\n5         #Now we need to find the slope, intercept and length of each of our detected lines\n6         left_lines = []\n7         length_left = []\n8         right_lines = []\n9         length_right = []\n10        #We may not always detect a line that is why we do try/except statement\n11        try:\n12            for line in lines:\n13                #Coordinates of a single line\n14                for x1, y1, x2, y2 in line:\n15                    #We dont want a vertical line or a horizontal line\n16                    if x1==x2 or y1==y2:\n17                        continue\n18                    #Slope formula\n19                    slope = (y2-y1)/(x2-x1)\n20                    #Intercept\n21                    intercept = y1 - slope*x1\n22                    #Length\n23                    length = np.sqrt((y2-y1)**2+(x2-x1)**2)\n24                    #Y is reversed in images therefore a negative slope is a left line not right\n25                    if slope\u003c0:\n26                        left_lines.append((slope, intercept))\n27                        length_left.append((length))\n28                    else:\n29                        right_lines.append((slope, intercept))\n30                        length_right.append((length))\n31          #Now we have collected our similar lines into right and left lists\n32          #Now we can convert them into lanes by dot product all the similar lines with lengths\n33          #The longer lines are weighted more therefore affect the lanes more\n34          #Then we normalise them by dividing by sum of the lengths(sort of like averaginng)\n35          left_lane  = np.dot(length_left,  left_lines) /np.sum(length_left)  if len(length_left) \u003e0 else None\n36          right_lane = np.dot(length_right, right_lines)/np.sum(length_right) if len(length_right)\u003e0 else None\n37          #Now we have the right LANE and the left LANE through averaging and dot product\n38          #Now we need to convert them back into coordinates for pixel points\n39          #Having an equation of a line (assume infinite) we can select arbitrary points and find\n40          #the x or y value accordingly.\n41          #So we select arbitrary points for y1 = croppedImg.shape[0]\n42          #and for y2 = y1*0.6, We need this in order to draw our lines (converting to pixel coordinates)\n43          #We all need them to be int so cv2.line can use them\n44          LeftX1 = int((croppedImg.shape[0] - left_lane[1])/left_lane[0])\n45          LeftX2 = int(((croppedImg.shape[0]*0.6) - left_lane[1])/left_lane[0])\n46          RightX1 = int((croppedImg.shape[0] - right_lane[1])/right_lane[0])\n47          RightX2 = int(((croppedImg.shape[0]*0.6) - right_lane[1])/right_lane[0])\n48          left_lane = ((LeftX1, int(croppedImg.shape[0])), (LeftX2, int(croppedImg.shape[0]*0.6)))\n49          right_lane = ((RightX1, int(croppedImg.shape[0])), (RightX2, int(croppedImg.shape[0]*0.6)))\n50          #Now we can draw them on the image\n51          #We first create an empty array like our original image\n52          #Then we draw the lines on the empty image and finally combine with our original image\n53          emptImg = np.zeros_like(OrgImage)\n54          #[255, 0, 0,]is the color, 20 is the thickness\n55          #The star allows us to input a tuple (it processes as integer points)\n56          cv2.line(emptImg, *left_lane, [255, 0, 0], 20)\n57          cv2.line(emptImg, *right_lane, [255, 0, 0], 20)\n58          #Finally we combine the two images\n59          #It calculates the weighted sum of two arrays\n60          #1.0 is the weight of our original image, we don't want to amplify it\n61          #0.95 is the weight of our lines, and 0.0 is the scalar added to the sum\n62          #be very significant in the image, just enough so we can see it and not obstruct anything else\n63          finalImg = cv2.addWeighted(OrgImage, 1.0, emptImg, 0.95, 0.0)\n64      except:\n65          errors = True\n66          print(\"Nothing detected\")\n67          #If we dont detect anything or to avoid errors we simply return the original image\n68          return OrgImage, errors\n69      #If all goes well, we return the image with the detected lanes\n70      return finalImg, errors\n```\n**Line 6-Line 9** We initiate empty lists for our data. Left and right lines and their corresponding lengths. \n\n**Line 4** The function HoughLines is quite difficult to understand. In layman term it is simply detecting lines in our region and returning coordinates. We set a threshold of 180 the length, and the thickness being 5. To find out more about hough transformation, go to this [link](http://www.swarthmore.edu/NatSci/mzucker1/opencv-2.4.10-docs/doc/tutorials/imgproc/imgtrans/hough_lines/hough_lines.html)\n\n**Line 12-Line 30** Looping over all the detected lines we take one line at a time and we calculate the intercept and the slope. We omit horizontal and vertical lines as our lanes will never be straight in that perspective. Finally, depending on the slope we append our lines accordingly to the right and left lanes.\n\n**Line 35-Line 36** We want to combine all our lines into lanes. We compute the dot product of the lines and their respective lengths. Longer lines have a heavier effect and so the slopes and intercepts of those line will be more dominant. Finally divide by the lengths to essentially normalise the values(So it can be mapped to the image)\n\n**Line 44-Line 47** We have the lanes, but in order to draw them we need coordinates. To draw any line (assuming infinite) any arbitrary point can be used. Using the arbitrary y values I calculate the x values.\n\n**Line 48-Line49** Now we just group those points accordingly to the right and left lanes.\n\n**Line 53** We want to draw the line on top of our image. But in order to do that we need to have an overlay image. So here, we create an empty image with the same space dimensions as our original. \n\n**Line 56-Line 57** Then we draw our lines on our empty image. The color used is blue as the format is BGR and not RGB.\n\n**Line 63** Finally we combine the two images. This is done by calculating the weighted sum of the two arrays of images. In our empty image most of the pixels are set to 0 and so only the lane pixels will be effected.\n\n**Line 65-Line 68** If any errors or a lane wasn't detected then we simply just output our original image.\n\n**Line 70** If all goes well, we output our final processed image.\n\n\u003cp align=\"center\"\u003e\n  finalImg\u003cbr\u003e\n  \u003cimg width=\"400\" height=\"300\" src=\"https://github.com/sagar448/Self-Driving-Car-3D-Simulator-With-CNN/blob/master/src/finalImg.png\"\u003e\n\u003c/p\u003e\n\nNow we can go ahead and explore the next part of the code. The next part explains how to format our processed images so it could be accepted by our Keras CNN.\n\n```python\n1     #Processes the images and returns the required data\n2     def getFrames():\n3         #We initialise the mss screenshot library\n4         sct = mss.mss()\n5         #This essentially takes a screenshot of the square from the coordinates\n6         #You can adjust these to your liking, \n7         game = {'top': 122, 'left': 0, 'width': 512, 'height': 286}\n8         #This converts the screenshot into a numpy array\n9         gameImg = np.array(sct.grab(game))\n10        #We want to resize the array so we can easily display it\n11        gameImg = cv2.resize(gameImg, (600, 400))\n12        #We pass the array into our calculateLanes function\n13        #it returns our detected lanes image as well as if any errors were produced\n14        img, errors = CalculateLanes(gameImg)\n15        #You can show the render if you want with the lanes detections\n16        cv2.imshow('window', img)\n17        #To further process the image we convert it to a grayscale\n18        img = cv2.cvtColor(cv2.resize(img, (84, 84)), cv2.COLOR_BGR2GRAY)\n19        #In order for Keras to accept data we reshape it into the specific format\n20        #I want to use an image thats 84x84\n21        img = img.reshape(1, 84, 84)\n22        #In order to give the algorithm the feel of the \"velocity\" we stack the 4 images\n23        input_img = np.stack((img, img, img, img), axis = 3)\n24        #This is required for openCV as a failsafe for stopping render\n25        #By pressing q, you can stop render\n26        if cv2.waitKey(25) \u0026 0xFF == ord('q'):\n27            cv2.destroyAllWindows()\n28        #If all goes well we return the input_img and the errors\n29        return input_img, errors\n```\n**Line 2** This function essentially processes our screenshots using the lane detection function and then formats the image data so we can then use it with our CNN.\n\n**Line 4** We initialise our screenshot library here.\n\n**Line 7** Game stores the dimensions of our screenshots. It represents the area of the screen we took the screenshot of.\n\n**Line 9** We convert it to a numpy array for further processing.\n\n**Line 11** I resized the image so when we display it, it can fit on the screen. Note, if you change the size of the screen you will need to edit the coordinates of the ROI mask in the lane detection function in order to account for the size increase or decrease.\n\n**Line 14** We now call the CalculateLane() function passing the resized game screenshot as a paramter. It returns either the original image back to us or it returns our image with detected lanes. \n\n**Line 16** You can choose to render your detection but it will slow down the process quite a bit.\n\n**Line 18** We can now start formatting our image for our CNN. Our first step is to resize it to a suitable size for the CNN to process as well as to convert it to grayscale.\n\n**Line 21** Since Keras needs a specific dimension we reshape our image to 1x84x84. The 1 is essentially the batch number.\n\n**Line 23** The CNN needs to make logical decisions. Therefore, without any sense of velocity the CNN cannot perform. In order to provide the CNN with some sense of velocity we stack our images. Thus, our dimension of our input is now (1, 84, 84, 4).\n\n**Line 26-Line 27** If you've ever used OpenCV and decided to display your image/video then you know to always put this at the end or the image/video will not display.\n\n**Line 29** Finally, we return our input and errors (Errors for the CalculateLanes function)\n\nThat takes care of all the image processing. We can now go ahead and start taking a look at function that controls our car in game.\n\n```python\n#This function makes the car accelerate\ndef straight():\n    p.keyDown(\"up\")\n    p.keyUp(\"up\")\n\n#We can turn right with this\ndef right():\n    p.keyDown(\"right\")\n    p.keyUp(\"right\")\n\n#Turn left with this\ndef left():\n    p.keyDown(\"left\")\n    p.keyUp(\"left\")\n```\n**Function straight()** Key Down function presses the specific key on our keyboard. KeyUp is important as KeyDown holds the key, so we need to release it. This function is responsible for accelerating our car.\n\n**Function right()** Turns our car to the right.\n\n**Function left()** Turns our car to the left.\n\nWe are now ready to start building our CNN model. Our model will be quite similar to the other CNN models in the past, we will try to map our image data to the actions.\n```python\n1     #For now we make the car accelerate, turn right and turn left\n2     moves = 3\n3     #learning rate (discount rate)\n4     learningRate = 0.9\n5     #This is the exploration rate (epsilon)\n6     #Its better at first to let the model try everything\n7     epsilon = 1.0\n8     #We don't want our model to stop exploring so we set a minimum epsilon\n9     epsilon_min = 0.01\n10    #We also dont want our model to explore all the time therefore we want it\n11    #to decay\n12    epsilon_decay = 0.995\n13    #Number of times we want to train the algorithm (The number of games)\n14    epochs = 100\n15    #We want to store our data for our replay so our model can remember the past experiences\n16    memory = []\n17    #The max amount of stuff we want to remember\n18    max_memory = 500\n19\n20    #Lets start defining our model\n21    model = Sequential()\n22    #We will be using a CNN with 32 filters, 3x3 kernel and the input shape will be\n23    #84x84 with 4 grayscale images stacked on top\n24    #padding will be set as same(padding with 0) and we will use the rectified activation function\n25    model.add(Conv2D(32, (3, 3), input_shape=(84, 84, 4), padding='same',\n26                     activation='relu'))\n27    #This time we will use 64 filters with a 3x3 kernel, with the same act function \n28    #but the padding will change\n29    model.add(Conv2D(64, (3, 3), activation='relu', padding='valid'))\n30    model.add(Conv2D(64, (3, 3), activation='relu', padding='valid'))\n31    #We flatten our data in order to feed it through the dense(output) layer\n32    model.add(Flatten())\n33    model.add(Dense(512, activation='relu'))\n34    #We have 3 outputs, forward, left, right\n35    model.add(Dense(3, activation='linear'))\n36    #We will be using the mean squared error\n37    model.compile(loss='mean_squared_error',\n38                  optimizer=SGD())\n```\n**Line 2** As shown in the code above, our car can do 3 things. Accelerate, turn right and turn left. Thus we set our moves variable to 3.\n\n**Line 4** This is our discount rate. We want our immediate reward to be worth more than our future reward therefore we discount the future reward in order make the current reward stand out. This is because our model is uncertain what the next step may be. (More on this later)\n\n**Line 7** This is the exploration rate. We want our algorithm to start off by trying different actions.\n\n**Line 9** We don't want our model to ever stop trying random actions so we set our minimum exploration rate.\n\n**Line 12** This is the rate at which our exploration factor decays.\n\n**Line 14** This is the number of games we want to play in total.\n\n**Line 16** All the games ever played go in here. We want our model to learn from its mistakes.\n\n**Line 18** We don't want to store too many games as it becomes computation heavy.\n\nNow we can start building our actual CNN model.\n\n**Line 21** We initialise our machine learning algorithm.\n\n**Line 25** This is our first convolutional layer. We want to output 32 filters with a 3x3 kernel and our input shape will be 84x84x4. We set our activation function to rectified linear unit.\n\n**Line 29-Line 30** We add another two convolutional layers for better accuracy.\n\n**Line 32-Line 33** We flatten our data so we can put it through a hidden layer of a simple neural network.\n\n**Line 35** This is the final output layer with 3 nodes. It calculates the probability of our 3 actions. \n\n**Line 37** Configuration for our loss and optimisation function. \n\nFinally, we've reached the last step of the tutorial, our Q-Learning algorithm. The brain and heart of the algorithm. This algorithm decides the actions to take and essentially trains our car to be a better driver.\n```python\n1      #loop over the number of epochs (essentially the number of games)\n2      for i in range(epochs):\n3         #time.sleep(5)\n4         #We set the game_over to false as the game is just starting\n5         game_over = False\n6         #We start of by getting initial frames and errors\n7         input_img, errors = getFrames()\n8         #We set the errors to false to begin with\n9         errors = False\n10        #We set the reward to 0\n11        reward = 0\n12        #While the game is not over we loop\n13        while game_over==False:\n14            #Np.random.rand() returns a number between 0 and 1\n15            #We check if its smaller that our exploration factor\n16            if np.random.rand() \u003c= epsilon:\n17                #if the random number is smaller than our exploration factor\n18                #We select a random action from our 3 actions\n19                action = np.random.randint(0, moves, size=1)[0]\n20            else:\n21                #If it's not smaller than we predict an output by inputting our\n22                #4 stacked images\n23                #ouput is the probability of our 3 directions\n24                output = model.predict(input_img)\n25                #action is the index of the highest probability and therefore\n26                #indicates which turn to take\n27                action = np.argmax(output[0])\n28            #if our action == 0 then we go straight   \n29            if int(action) == 0:\n30                straight()\n31            #If our action == 1 then we go right\n32            elif int(action) == 1:\n33                right()\n34            #else we go left\n35            else:\n36                left()\n```\n**Line 2** We loop over the amount of games we want to play. In this case I have set the epochs to 100. \n\n**Line 3** Originally I had left the time.sleep in the program as this allowed me to prepare for the start of the algorithm but it also slows down the learning stage therefore it is commented out.\n\n**Line 5** The AI is about to start playing the game so we originally set the game_over to false. We will need it later.\n\n**Line 7** We start by getting the initial \"state\" of the algorithm. We will need this to predict our corresponding action.\n\n**Line 9** Despite the error in **Line 7** we set our errors to false as errors at the start do not matter, the algorithm will be performing a random action to begin with. \n\n**Line 11** Initialise our Rewards variable to 0.\n\n**Line 13** Looping for one game, while the game isn't false\n\n**Line 16-Line 19** We start off by checking if our exploration is bigger than a random number between 0 and 1. At the begining it will be and so we select a random action from our 3 actions. \n\n**Line 20-Line 27** Once our exploration rate is low enough, we can start predicting our actions. Output stores a numpy array of size 3 produced by the prediction from our input image. Action stores the index of the maximum probability.\n\n**Line 29-Line 36** Based on our predicted or random action we select one of the functions to run that controls our car. \n\nHalfway through! From here we can now actually start studying the bulk of the Q-learning algorithm!\n```python\n1             #Once we've performed our action we get the next frame\n2             #We also check weather to reward the algorithm or not\n3             input_next_img, errors = getFrames()\n4             #If we detect lanes and therefore no errors occur we reward the algorithm\n5             if errors == False:\n6                 reward = 1\n7             #Else if there we detect no lanes and so there is an error we \n8             #say its game over\n9             else:\n10                reward = 0\n11                game_over = True\n12            #Game over or not we want to keep record of the steps the algo took\n13            #We first check if the total memory length is bigger than the max memory\n14            if len(memory) \u003e= max_memory:\n15                #If more memory then needed we delete the first ever element we added\n16                del memory[0]\n17            #We append it to our memory list\n18            memory.append((input_img, action, reward, input_next_img, game_over))\n19            #Next we set our input_img to our latest data\n20            input_img = input_next_img\n21            if game_over:\n22                print(\"Game: {}/{}, Total Reward: {}\".format(i, epochs, reward))\n23        #Once the game is over we want to train our algo with the data we just collected\n24        #We check if our memory length is bigger than our batch size \n25        if len(memory) \u003e 32:\n26        #If so then we set the batch_size to 32\n27            batch_size = 32\n28        else:\n29        #Else we set our batch size to whatever is in the memory\n30            batch_size = len(memory)\n31        #We are taking a random sample of 32 so not to overfit our algo\n32        batch = random.sample(memory, batch_size)\n33        #We itereate over every memory we've stored in that memory batch of 32\n34        for input_img, action, reward, input_next_img, game_over in batch:\n35            #if in that memory our game was over then we set the target_reward equal to reward\n36            target_reward = reward\n37            #If our game was not over\n38            if game_over == False:\n39            #This essentially is the bellman equation\n40            #expected long-term reward for a given action is equal to the \n41            #immediate reward from the current action combined with the expected \n42            #reward from the best future action taken at the following state.\n43            #The model isn't certain that for that specific action it will get the best reward\n44            #It's based on probability of the action, if the probability of that action is in the\n45            #negatives then our future reward is going to be further decreased by our learning rate\n46            #This is just the model being cautious, as to not set an impossible reward target\n47            #If the reward is impossible then the algorithm might not converge\n48            #Converge as in a stable condition where it can play the game without messing up\n49                target_reward = reward + learningRate * \\\n50                np.amax(model.predict(input_next_img)[0])\n51            #So from above we essentially know what is going to happen(input_next_img) \n52            #assuming the game wasn't over, the algorithm did well.\n53            #So we want the algorithm to perform the same, essentially we\n54            #persuade the algorithm to do what it did to get that reward\n55            #so we make the algorithm predict from the previous frame(input_img)\n56            #but we alter its prediction according to the action that got the highest\n57            #reward and...\n58            desired_target = model.predict(input_img)\n59            #we set that as the target_reward...\n60            desired_target[0][action] = target_reward\n61            #So to make the algo perform the same, we associate the input_img with the\n62            #target we want and we fit it\n63            model.fit(input_img, desired_target, epochs=1, verbose=0)\n64        #Finally we check if our exploration factor is bigger than our minimum exploration\n65        #if so we decrease it by the decay to reduce exploration, we do this every game\n66        if epsilon \u003e epsilon_min:\n67            epsilon *= epsilon_decay\n```\n**Line 3** After the action we get our next frame, and errors if any.\n\n**Line 5-Line 6** After the action has been performed and we have the next frame with calculated lanes and it does not return any errors then we set the reward to 1.\n\n**Line 9-Line 11** If it does return errors then we say that the game is over. I have set it up like that so the algorithm can learn to drive within lanes. The error is associated with either the lanes not being detected or simply because the car was not within any lanes to detect. The latter being more probable and thus provides reason for the specific guidelines.  I also set the reward to 0 as the algorithm fails to achieve its goal\n\n**Line 14-Line 16** Regardless the status of the game_over variable, we want to record the gameplay that happened. This enables the algorithm to learn from it's mistakes. So in this piece of code, we check whether the memory is full or not, if so we delete the very first item appended.\n\n**Line 18** We append to the memory array.\n\n**Line 20** We set our next set of frames to our current set of frames. Essentially progressing our variable input_img to the next undecided action frame.\n\n**Line 21-Line 22** If game was over we print out our statistics. \n\n**Line 25-Line 30** This simply put is the setup for our replay section of the Q-algorithm. We want to select random sample of batches to train our algorithm with. Our default batch size is 32, but at the begining there wouldn't be enough to sample 32 batches. Therefore, we train the algorithm with the whole memory array.\n\n**Line 34-Line 36** Iterating over our memory, we begin by setting our target reward to our reward in the first sample memory. \n\n**Line 38-Line 50** In that memory if our game wasn't over than that means our algorithm performed well. So we want to persuade our algorithm to do the same thing in the future. Therefore we set our future reward (target reward) to the current reward from the current action combined with the expected reward from the best future action taken at the following state. We multiply by our learningRate to avoid converging problems. We are essentially increasing the probability of our desired action.\n\n**Line 58** Here we ask the algorithm again what it might predict for the previous state.\n\n**Line 60** We manipulate the prediction, we take the prediction and insert our own probability of our corresponding action. Simply telling the algorithm that for a situation like this we want this action to be performed.\n\n**Line 63** We feed the manipulations and the results into our model to train it for a single epoch.\n\n**Line 66-Line 67** Finally, after everything is done, we decrease our exploration rate by multiplying our epsilon with our epsilon decay rate.\n\n## Conclusion\n\nWell thats it for the self driving car! You can definitely make your algorithm more complex by adding different directions, making your convolutional layers deeper etc. You can even apply this to another car game, create your own guidelines and own methods of rewards! If there are any question please don't hesitate to contact me, I am happy to help. I am open to feedback and different ways in which I could improve this, maybe you have a better way of doing this. Other than any questions, if you find a mistake while reading through this please let me know! Happy coding!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsagar448%2FSelf-Driving-Car-3D-Simulator-With-CNN","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsagar448%2FSelf-Driving-Car-3D-Simulator-With-CNN","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsagar448%2FSelf-Driving-Car-3D-Simulator-With-CNN/lists"}