First commit

README.md

@@ -1,2 +1,117 @@
-# Behavioral-Cloning
-End-to-End Deep Learning for Self-Driving Cars
+# Project 3: Use Deep Learning to Clone Driving Behavior
+
+[![Udacity - Self-Driving Car NanoDegree](https://s3.amazonaws.com/udacity-sdc/github/shield-carnd.svg)](http://www.udacity.com/drive)
+
+Overview
+---
+This repository contains starting files for P3, Behavioral Cloning.
+
+In this project, you will use what you've learned about deep neural networks and convolutional neural networks to clone driving behavior. You will train, validate and test a model using Keras. The model will output a steering angle to an autonomous vehicle.
+
+We have provided a simulator where you can steer a car around a track for data collection. You'll use image data and steering angles to train a neural network and then use this model to drive the car autonomously around the track.
+
+We also want you to create a detailed writeup of the project. Check out the [writeup template](https://github.com/udacity/CarND-Behavioral-Cloning-P3/blob/master/writeup_template.md) for this project and use it as a starting point for creating your own writeup. The writeup can be either a markdown file or a pdf document.
+
+To meet specifications, the project will require submitting four files: 
+* model.py (script used to create and train the model)
+* drive.py (script to drive the car - feel free to modify this file)
+* model.h5 (a trained Keras model)
+* a report writeup file (either markdown or pdf)
+
+Optionally, a video of your vehicle's performance can also be submitted with the project although this is optional. This README file describes how to output the video in the "Details About Files In This Directory" section.
+
+Creating a Great Writeup
+---
+A great writeup should include the [rubric points](https://review.udacity.com/#!/rubrics/432/view) as well as your description of how you addressed each point.  You should include a detailed description of the code used (with line-number references and code snippets where necessary), and links to other supporting documents or external references.  You should include images in your writeup to demonstrate how your code works with examples.  
+
+All that said, please be concise!  We're not looking for you to write a book here, just a brief description of how you passed each rubric point, and references to the relevant code :). 
+
+You're not required to use markdown for your writeup.  If you use another method please just submit a pdf of your writeup.
+
+The Project
+---
+The goals / steps of this project are the following:
+* Use the simulator to collect data of good driving behavior 
+* Design, train and validate a model that predicts a steering angle from image data
+* Use the model to drive the vehicle autonomously around the first track in the simulator. The vehicle should remain on the road for an entire loop around the track.
+* Summarize the results with a written report
+
+### Dependencies
+This lab requires:
+
+* [CarND Term1 Starter Kit](https://github.com/udacity/CarND-Term1-Starter-Kit)
+
+The lab enviroment can be created with CarND Term1 Starter Kit. Click [here](https://github.com/udacity/CarND-Term1-Starter-Kit/blob/master/README.md) for the details.
+
+The following resources can be found in this github repository:
+* drive.py
+* video.py
+* writeup_template.md
+
+The simulator can be downloaded from the classroom. In the classroom, we have also provided sample data that you can optionally use to help train your model.
+
+## Details About Files In This Directory
+
+### `drive.py`
+
+Usage of `drive.py` requires you have saved the trained model as an h5 file, i.e. `model.h5`. See the [Keras documentation](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) for how to create this file using the following command:
+```sh
+model.save(filepath)
+```
+
+Once the model has been saved, it can be used with drive.py using this command:
+
+```sh
+python drive.py model.h5
+```
+
+The above command will load the trained model and use the model to make predictions on individual images in real-time and send the predicted angle back to the server via a websocket connection.
+
+Note: There is known local system's setting issue with replacing "," with "." when using drive.py. When this happens it can make predicted steering values clipped to max/min values. If this occurs, a known fix for this is to add "export LANG=en_US.utf8" to the bashrc file.
+
+#### Saving a video of the autonomous agent
+
+```sh
+python drive.py model.h5 run1
+```
+
+The fourth argument `run1` is the directory to save the images seen by the agent to. If the directory already exists it'll be overwritten.
+
+```sh
+ls run1
+
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_424.jpg
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_451.jpg
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_477.jpg
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_528.jpg
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_573.jpg
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_618.jpg
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_697.jpg
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_723.jpg
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_749.jpg
+[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_817.jpg
+...
+```
+
+The image file name is a timestamp when the image image was seen. This information is used by `video.py` to create a chronological video of the agent driving.
+
+### `video.py`
+
+```sh
+python video.py run1
+```
+
+Create a video based on images found in the `run1` directory. The name of the video will be name of the directory following by `'.mp4'`, so, in this case the video will be `run1.mp4`.
+
+Optionally one can specify the FPS (frames per second) of the video:
+
+```sh
+python video.py run1 --fps 48
+```
+
+The video will run at 48 FPS. The default FPS is 60.
+
+#### Why create a video
+
+1. It's been noted the simulator might perform differently based on the hardware. So if your model drives succesfully on your machine it might not on another machine (your reviewer). Saving a video is a solid backup in case this happens.
+2. You could slightly alter the code in `drive.py` and/or `video.py` to create a video of what your model sees after the image is processed (may be helpful for debugging).

drive.py

@@ -0,0 +1,139 @@
+import argparse
+import base64
+from datetime import datetime
+import os
+import shutil
+
+import numpy as np
+import socketio
+import eventlet
+import eventlet.wsgi
+from PIL import Image
+from flask import Flask
+from io import BytesIO
+
+from keras.models import load_model
+import h5py
+from keras import __version__ as keras_version
+
+sio = socketio.Server()
+app = Flask(__name__)
+model = None
+prev_image_array = None
+
+
+class SimplePIController:
+    def __init__(self, Kp, Ki):
+        self.Kp = Kp
+        self.Ki = Ki
+        self.set_point = 0.
+        self.error = 0.
+        self.integral = 0.
+
+    def set_desired(self, desired):
+        self.set_point = desired
+
+    def update(self, measurement):
+        # proportional error
+        self.error = self.set_point - measurement
+
+        # integral error
+        self.integral += self.error
+
+        return self.Kp * self.error + self.Ki * self.integral
+
+
+controller = SimplePIController(0.1, 0.002)
+set_speed = 9
+controller.set_desired(set_speed)
+
+
+@sio.on('telemetry')
+def telemetry(sid, data):
+    if data:
+        # The current steering angle of the car
+        steering_angle = data["steering_angle"]
+        # The current throttle of the car
+        throttle = data["throttle"]
+        # The current speed of the car
+        speed = data["speed"]
+        # The current image from the center camera of the car
+        imgString = data["image"]
+        image = Image.open(BytesIO(base64.b64decode(imgString)))
+        image_array = np.asarray(image)
+        steering_angle = float(model.predict(image_array[None, :, :, :], batch_size=1))
+
+        throttle = controller.update(float(speed))
+
+        print(steering_angle, throttle)
+        send_control(steering_angle, throttle)
+
+        # save frame
+        if args.image_folder != '':
+            timestamp = datetime.utcnow().strftime('%Y_%m_%d_%H_%M_%S_%f')[:-3]
+            image_filename = os.path.join(args.image_folder, timestamp)
+            image.save('{}.jpg'.format(image_filename))
+    else:
+        # NOTE: DON'T EDIT THIS.
+        sio.emit('manual', data={}, skip_sid=True)
+
+
+@sio.on('connect')
+def connect(sid, environ):
+    print("connect ", sid)
+    send_control(0, 0)
+
+
+def send_control(steering_angle, throttle):
+    sio.emit(
+        "steer",
+        data={
+            'steering_angle': steering_angle.__str__(),
+            'throttle': throttle.__str__()
+        },
+        skip_sid=True)
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='Remote Driving')
+    parser.add_argument(
+        'model',
+        type=str,
+        help='Path to model h5 file. Model should be on the same path.'
+    )
+    parser.add_argument(
+        'image_folder',
+        type=str,
+        nargs='?',
+        default='',
+        help='Path to image folder. This is where the images from the run will be saved.'
+    )
+    args = parser.parse_args()
+
+    # check that model Keras version is same as local Keras version
+    f = h5py.File(args.model, mode='r')
+    model_version = f.attrs.get('keras_version')
+    keras_version = str(keras_version).encode('utf8')
+
+    if model_version != keras_version:
+        print('You are using Keras version ', keras_version,
+              ', but the model was built using ', model_version)
+
+    model = load_model(args.model)
+
+    if args.image_folder != '':
+        print("Creating image folder at {}".format(args.image_folder))
+        if not os.path.exists(args.image_folder):
+            os.makedirs(args.image_folder)
+        else:
+            shutil.rmtree(args.image_folder)
+            os.makedirs(args.image_folder)
+        print("RECORDING THIS RUN ...")
+    else:
+        print("NOT RECORDING THIS RUN ...")
+
+    # wrap Flask application with engineio's middleware
+    app = socketio.Middleware(sio, app)
+
+    # deploy as an eventlet WSGI server
+    eventlet.wsgi.server(eventlet.listen(('', 4567)), app)

video.py

@@ -0,0 +1,27 @@
+from moviepy.editor import ImageSequenceClip
+import argparse
+
+
+def main():
+    parser = argparse.ArgumentParser(description='Create driving video.')
+    parser.add_argument(
+        'image_folder',
+        type=str,
+        default='',
+        help='Path to image folder. The video will be created from these images.'
+    )
+    parser.add_argument(
+        '--fps',
+        type=int,
+        default=60,
+        help='FPS (Frames per second) setting for the video.')
+    args = parser.parse_args()
+
+    video_file = args.image_folder + '.mp4'
+    print("Creating video {}, FPS={}".format(video_file, args.fps))
+    clip = ImageSequenceClip(args.image_folder, fps=args.fps)
+    clip.write_videofile(video_file)
+
+
+if __name__ == '__main__':
+    main()