Getting started with OpenCV
OpenCV is an image processing toolbox originally developed in C++.
In Python, an OpenCV image is a
numpy array (2D or 3D depending
on the type of image). The figure below depicts the coordination
used in OpenCV. For example, if we have a numpy array
an OpenCV image, then
img[0,0] stores the data for top left
pixel having the coordinate
(x=0,y=0). Following this system, the
bottom right corner
img[-1,-1] has the coordinate
(x=img.shape-1, y=img.shape-1) in OpenCV.
OpenCV provides various tools for us to “get our hand dirty” with images. Generally, there are two main groups: image drawing and image transformation. Drawing on images in OpenCV is quite simple and straight forward (except for the ellipse ) as most drawing function is in the form:
The main tool to detect lines in an image is a technique named Hough Transformation. It is called transformation because it transforms the representation of a line to a pair of line angle and line distance to the origin. The detail and tutorial of Hough Transformation is provided here. To reduce the computation and increase the accuracy of the line detection pipeline, we focus only on the area in front of a car. The following list describes our pipeline:
(x2,y2). For each pair of points in the returned set, we split them into left lane points and right lane points by its coefficient.
(x0, y0, vx, vy), where
(x0,y0)is a point on the line and
(vx,xy)is the line’s co-linear vector.
The first pipeline we came up with in this project simply converts the input to gray scale, detect edges by Canny algorithm, and then draw all line that Hough transformation returns. This approach is unstable since it depends heavily on the high and low threshold of the Canny algorithm. Although using a slight Gaussian blur on the gray scale image can reduce noise and improve the quality of the pipeline, imperfection on the road can potentially disturb the pipeline’s robustness.
The unused information in our first pipeline is: 1. The region of interest, and 2. The color of the lane lines. By extracting the region of interest, we eliminate unnecessary computation:
In addition to extracting the region of interest, we also filtered out
unwanted colors. By default, the image output of
cv2.imread is a GBR
image. This color representation makes it hard to filter a certain
color since all three values (G,B,R) of a pixel represents color.
Therefore, we convert the image to HSV color space
(Hue, Saturation, Value). In HSV images, a pixel contains the color
(hue), the “amount” of that color (saturation), and its brightness
(value). This color representation enable us to specify the colors we
want to extract. To exact colors, the rule of thumb is to range
±10 in the hue value as following:
To exact black or white color, the code is different since it depends on the saturation and value rather than the hue.
After selecting only the region of interest and the colors, we have the following result:
The image above is a binary image which can be used as a mask to extract the lane lines from the original image. The example of our lane lines detection on static image is shown below.
The pipeline showed in the previous session performs well on test images and videos. However, with the challenge video, it failed to detect the lane lines for some brief moments when the lighting varies. Furthermore, in all videos, the lane lines between frame doesn’t have smooth transitions. To address this problem, we have several approaches:
The videos result for each of the approach will be updated soon. TODO: Upload videos.
We have some minor bugs during the implementation of of the buffered pipeline. Firstly, when the buffer is empty, the pipeline should not draw the line. In one of our implementation, a default line is drawn when the buffer is empty, this design decision makes it hard to debug the program. Secondly, when no line is detected from the frame, the algorithm should still return a line from the buffer. However, if there are many “no line” frames, there is a chance that there isn’t any lane lines on the road. Our current buffer implementation hasn’t taken care of this situation.
For the current testing data (images and videos), extracting the region of interest and lane line colors is enough for line detection.
As the picture above has shown, only yellow (left) and white (right) color filter is enough to give us a substantially clear image of lane lines. This output here can be put directly to the Hough Line detection (without masking with the original image or Canny edge detection) to obtain the lane lines. At this stage, we don’t know if performing Canny edge detection is necessary (i.e. makes the pipeline more robust) .
Thanks for reading!