Transportation has always been a corner stone of progress, whether to convey goods from the Orient to Europe or enable the population to efficiently commute between cities. Tremendous progress has been achieved in the last century, thanks to breakouts in industry we are now blessed with trains cruising at 300km/h and efficient commuting grids in most cities of the developed world. With the democratization of personal transport options, such as cars, most people have now the ability to move from point A to point B 10 times as fast as our ancestors only 2 generations ago.
But having faster, safer and efficient cars are not the sole levers of innovation. After long hours on the road driving your family on vacations, who is not feeling bored or tired at some point? What if you could combine the freedom that comes with a personal car with the comfort that a train offers? Once you set the destination the car starts and drives you peacefully to your secondary house on the Atlantic coast, offering you the freedom to chat with your wife and play with your kids while an Auto-pilot deals with the traffic jams and the speed limitations.
This is the mission that we will try to tackle through this challenge. Explore, experiment, learn and hopefully develop a real self-driving car model.
In order to develop a self driving car we have first to find a good simulation environment as we will not work on a real world case. After scanning the internet I came up with the following list :
- Gazebo CitySim
- Carla Sim
- Microsoft Airsim
- Appolo Sim
- Autoware Sim
I was interested to work with Python, in a windows environment and with access to some clear documentation and a good Python API, so after some consideration I picked Carla SIM.
What is Carla ?
So what exactly is Carla ? Carla is an Open Urban Driving Simulator. It has been built to support development, training, and validation of autonomous driving systems. The simulation platform has some interesting tools such as various sensors and the possibility to spawn pedestrians, other cars and control the weather..
The Simulator is implemented in Unreal Engine 4 which offers nice graphics and pleasant driving and is designed in a client-server architecture in order to enable multiple clients in the same or in different nodes. Here are the highlights from the Carla website :
- Scalability via a server multi-client architecture: multiple clients in the same or in different nodes can control different actors.
- Flexible API: CARLA exposes a powerful API that allows users to control all aspects related to the simulation, including traffic generation, pedestrian behaviors, weathers, sensors, and much more.
- Autonomous Driving sensor suite: users can configure diverse sensor suites including LIDARs, multiple cameras, depth sensors and GPS among others.
- Fast simulation for planning and control: this mode disables rendering to offer a fast execution of traffic simulation and road behaviors for which graphics are not required.
- Maps generation: users can easily create their own maps following the OpenDrive standard via tools like RoadRunner.
- Traffic scenarios simulation: our engine ScenarioRunner allows users to define and execute different traffic situations based on modular behaviors.
- ROS integration: CARLA is provided with integration with ROS via our ROS-bridge
- Autonomous Driving baselines: we provide Autonomous Driving baselines as runnable agents in CARLA, including an AutoWare agent and a Conditional Imitation Learning agent.
Carla offers a wide range of sensors that would come particularly handy for our challenge :
So we have :
- RGB Camera : the infamous russian dashcam, transmitting live the image of what lays ahead of our car
- Semantic segmentation camera : this camera classifies every object in sight by displaying it in a different color. This sensor would be particularly useful to avoid crossing continuous lanes and staying on the road.
- Depth Camera : this sensor provides a raw data of the scene codifying the distance of each pixel to the camera to create a depth map of the elements. This could be particularly helpful to distinguish better the distance between the car and the objects in front and take pre-emptive correction actions (trajectory, speed..)
- Lidar : Finally we could add the lidar when we chose to also include pedestrians and cars into the Carla environment in order to be able to detect those next to our agent better.
Install Carla
To Install Carla grab the version you want to use from the website and unzip it, we will be using the 0.9.9.4 version. This is the link for the windows version : link
Then unzip wherever you want.
You then have to install Python, I used python 3.7.8 : link
Then add the python path and the Carla libraries Path to your environment variables :
Finally, to develop the DQN and the Conv imitation learning agents you will need, note that this can be done at a later stage so don’t worry and do not do it right now :
SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;%PATH% SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\extras\CUPTI\libx64;%PATH% SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include;%PATH%
You will also need various python libraries, but deal with that on the fly with a pip install when needed. Pygame and Numpy would be a descent start to be able to test your environment.
To test your environment
Go to the main directory of your Carla folder (the file that you have unzipped previously) and you will find either have a CarlaUE4.sh if you’re on Linux, or a CarlaUE4.exe on Windows. To start Carla on windows double click the .exe. This will start the Carla Server :
This will start the server and you should promptly see the environment and the map, you can move around in it with your WASD keys and your mouse. The map is empty at the moment, it is just the coffin that we will populate. You should eventually see the map, which you can navigate with the WASD keys and your mouse. Of course, nothing is yet here. This is just our environment.
You can have a look at the Carla quick start guide to get a feeling about what you can do : link
Alright, let’s have some fun and spawn a car and drive around. Open your terminal or cmd and navigate to PythonAPI\examples and launch manual_control.py
There we go, This will create another window as Carla architecture is separating client from server and spawn a car that we can drive around. You can control the car with WASD keys, and Q will change into and out of reverse.
<insert video of basic driving in carla>
Now let’s have a look at what is included in the Python API :
World : Object that represents the simulation. It is an abstract layer that contains methods to spawn actors, change the weather, get the current state and much more. Docs : Link
Client : Connects to the server which rims the simulation. Run by the user to ask for information or changes in the simulation. Docs : Link
Vehicles : Actor that represents trucks, cars, vans etc… It can be manually controlled or set to auto pulot. The speed and different measurements about it can be captured. Provides an interface for controlling it (VehicleControl). Docs : Link
VehicleControl : An object that manages the basic controls of the vehicle. it will be uised on the agent to control the vehicle. Docs : Link
Sensors : Actors that retrieve data from their surrendings. Comprised of Cameras (RGB, Semantic, Depth), detectors (Collision, Obstacles, Lane Invasion), and others (GNNS, Radar, Lidar). The output changes greatly by type. Docs : Link
And finally let’s see what are the different objects that we can interact with :
Actors : An actor is anything that plays a role in the simulation. Docs = Link
- Vehicles
- Walkers (People)
- Sensors
- Spectator
- Traffic signs and lights
Blueprints : blueprints, frameworks, existing models that can be used to create actors; models with animations and attributes. You can then customize them. Docs = Link
Maps : Includes the 3D model of a town and its road definition, based on OpenDRIVE standard 1.4. Docs = Link
Navigation : There are different elements that we can retrieve and can be useful :
- Lanes
- Junctions
- Landmarks
Waypoints : They mediate between the Carla world and the OpenDrive definition of the road. It also contains its left and right lanes, and information to know if its inside a junction.
Alright, we have our environment setup and running, we can drive around an agent and we can communicate with Carla and we know what we can manipulate. Time to start to explore how to approach the learning part.
