End-to-end Lane Detection Through Differentiable Least-Squares Fitting

End-to-end Lane Detection throughDifferentiable Least-Squares FittingWouter Van Gansbeke, Bert De Brabandere, DavyNeven, Marc Proesmans, Luc Van GoolarXiv:1902.00293v3 [cs.CV] 5 Sep 2019ECE 285 – Autonomous Driving SystemsPresented by – Anirudh Swaminathan – April 23, 2020

Why LaneDetection?Detecting lanes is important because: Position vehicle within the lane Plan future trajectory, lane departures2

LaneDetectionBackgroundPrevious methods before this paper: Two step pipelines First step - segment lane line markings Second step - fit a lane line model to postprocessed mask3

2-stage examples Classical SIFT[20] / SURF[2] for feature extraction RANSAC / spline / polynomial for parameters of best fitting model Deep Learning Based Instance Segmentation – LaneNet [24] Curve fitting mostly same[24] - Towards End-to-End Lane Detection: an Instance Segmentation ApproachDavy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, Luc Van Gool ESAT-PSI, KU LeuvenarXiv:1802.05591v1 [cs.CV] 15 Feb 2018[2] H. Bay, T. Tuytelaars, and L. Van Gool. Surf: Speeded up robust features. Proceedings of the EuropeanConference on Computer Vision, 2006[20] D. G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the IEEE InternationalConference on Computer Vision, 1999.4

Objective ofthe Paper End-to-end manner Directly regress lane parameters5

MotivationWhy single step? Parameters not optimized for true task True task – estimating lane curvature parameters Proxy task – Segmenting lane markings Prevents instabilities in curve fitting 2 step – outliers End-to-end - implicitly learn features to prevent instabilities6

Methodology Key Idea - Integrate curve fitting step as a differentiable in-network optimization step Deep Network for the feature extraction step Key Idea - A geometric loss function for the network7

FrameworkThe framework consists of 3 main modules: Deep network to generate weighted pixel coordinates Differential weighted least squares fitting module Geometric Loss Function8

Example Architecture – Figure 1 from the paper9

Generating Weighted Pixel Coordinates First Module of network Normalized Coordinates - x map and y map Each coordinate - weight w Feature map - same spatial dimensions as that of input image10

Feature Maps Non-negative weights Width – w, height – h; m w * h M triplets generated – (x, y, w) One feature map for each lane11

Example Architecture – Figure 1 from the paper12

Weighted Least Squares Layer M triplets (x, y, w) - weighted points in 2D space Fit curve Module output - n parameters of best-fitting curve13

Background - Least Squares Fitting 𝑋𝑋𝑋𝑋 𝑌𝑌; 𝑋𝑋 𝑅𝑅𝑚𝑚𝑚𝑚𝑚𝑚 ; 𝛽𝛽 𝑅𝑅𝑛𝑛 1 ; 𝑌𝑌 𝑅𝑅𝑚𝑚 1 X is input, 𝛽𝛽 are parameters, and Y is output Least Squares - 𝛽𝛽 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 𝑋𝑋𝑋𝑋 𝑌𝑌 2 Normal Equation - 𝛽𝛽 𝑋𝑋 𝑇𝑇 𝑋𝑋 1𝑋𝑋 𝑇𝑇 𝑌𝑌14

Background – Weighted Least Squares Least squares extended 𝑊𝑊 𝑅𝑅𝑚𝑚 m ; Diagonal matrix - weights for each pair of observations12 𝛽𝛽 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 𝑊𝑊 (𝑋𝑋𝑋𝑋 𝑌𝑌) 2 𝑋𝑋 𝑇𝑇 𝑊𝑊𝑊𝑊𝑊𝑊 𝑋𝑋 𝑇𝑇 𝑊𝑊𝑊𝑊 𝛽𝛽 𝑋𝑋 𝑇𝑇 𝑊𝑊𝑋𝑋 1𝑋𝑋 𝑇𝑇 𝑊𝑊𝑌𝑌15

Backprop through the layer Equations involve differentiable matrix operations Calculate the derivative of 𝛽𝛽 wrt W Refer to [10] to derive backpropM. B. Giles. An extended collection of matrix derivative results for forward and reverse mode automaticdifferentiation.Technical report, University of Oxford, 2008.16

Example Architecture – Figure 1 from the paper17

Geometric Loss Function - precursor Usually, L2 loss used for curve fitting Here, 𝛽𝛽𝑖𝑖 and 𝛽𝛽 𝑖𝑖 - generated and groundtruth curve parameters18

Geometric Loss Function Lane Detection - geometric interpretation Minimize squared area between predicted curve and ground truth curve19

Geometric Meaning20

Geometric Loss Function – Parabola This paper - lane curves parabolic 𝑦𝑦 𝛽𝛽0 𝛽𝛽1 𝑥𝑥 𝛽𝛽2 𝑥𝑥 2 ; Δ𝛽𝛽𝑖𝑖 𝛽𝛽𝑖𝑖 𝛽𝛽 𝑖𝑖21

Optional Transformations Weighted coordinates - another reference frame Use fixed transformation matrix H Lane line - better as parabola from top-down/ortho view(BEV)22

Experiment – Ego Lane Detection Ego lane - the current lane of the vehicle Two lane marking - one left and one right Parabola - upto fixed distance t from car Overall error average over 2 lanes, and average over images23

Dataset TuSimple Dataset [29] Manually select and clean up the annotations of 2535 images Filter out images where ego-lane cannot be detected unambiguously 20% images - validation set Not include images of single temporal sequence in both train and val sets[29] TuSimple. Tusimple benchmark, 2017.24

Annotation Ground truth curve parameters - parabola Draw curve of fixed thickness as dense label25

Baseline – Cross-entropy training Training Segmentation Per-pixel binary cross-entropy loss Testing Segmentation mask generated Parabola fitted in least squares sense26

End-to-end training ERFNet [28] - network architecture 350 epochs; 1 GPU; 256*512 resolution; batch size 8 Adam[19] with LR 10 4 PyTorch [26][28] E. Romera, J. M. Alvarez, L. M. Bergasa, and R. Arroyo. Efficient convnet for real-time semantic segmentation.In IEEE Intelligent Vehicles Symposium, pages 1789–1794, 2017.[19] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the InternationalConference on Learning Representations, 2015.[26] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. De- Vito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.Automatic differentiation in pytorch. In NIPS-W, 2017.27

Example Architecture – Figure 1 from the paper28

Detour ERFNet29

ERFNet Semantic Segmentation Typical Encoder – Decoder architecture Last layer - adapted to output 2 feature maps One for each ego lane Transform weighted coordinates using fixed H to top-down view[28] E. Romera, J. M. Alvarez, L. M. Bergasa, and R. Arroyo. Efficient convnet for real-time semantic segmentation.In IEEE Intelligent Vehicles Symposium, pages 1789–1794, 201730

Results - Quantitative31

Result – Training curves32

Qualitative Results33

Analysis Lower error than cross-entropy method Convergence slower - supervision signal weaker Generated weight maps look like segmentation maps the network eventually discovers that the most consistent way to satisfy the loss function is to focuson the visible lane markings in the image, and to map them to a segmentation-like representationin the weight maps.34

Further Experiments – Multi-lane detection 4 lanes total - ortho-view Line prediction branch; horizon prediction branch Horizon prediction branch - regression - estimate horizon Line prediction branch - whether lane is present or not35

Architecture36

Architecture details Side branches - 4 conv layers - each 3x3 Then max pool - FC layer Losses for 3 tasks - combined linearly37

Dataset 3626 images 20% validation set 2782 test set images38

Comparison ERFNet without backprop through least squares layer - baseline [25] Spatial CNN [24] - Instance Segmentation approach[24] D. Neven, B. De Brabandere, S. Georgoulis, M. Proesmans, and L. Van Gool. Towards end-to-end lanedetection: an instance segmentation approach. arXiv:1802.05591, 2018.[25] X. Pan, J. Shi, P. Luo, X. Wang, and X. Tang. Spatial as deep: Spatial cnn for traffic scene understanding. InAAAI, 2018.39

Results Quantitative40

Results Qualitative41

Analysis Improve upon baseline by 0.7% Faster than benchmarks in test time - no post-processing required 71 fps on NVIDIA 1080Ti42

ADVANTAGES Optimized for true task - prevents instabilities in curve fitting Offers degree of interpretability Generated weight maps - segmentation-like Can be inspected and visualized Geometry aware criterion is loss function Handle large variance, faded lane markings Moves complexity from post-processing to network - one-shot fitting43

DISADVANTAGES Loss function - more complicated for higher order curves Fixed transformation H to ortho-view If ground plane is different (ex. Sloping uphill), then bad lane parameters in test time Local minimum possible – author Vanishing point in horizon/left corner of image features - good curve - no improvement Multi-lane - fixed number of maps - pre-defined order Lane changes hard; Order ambiguous Instance segmentation - not subject to specific order Quantitative results - comparatively worse from slide 4044

KEY TAKEAWAYS Including differentiable in-network optimization step. Geometric Loss function relevant to the task45

Question to the class Why do you think that the loss in the Least Squares layer is only back-propagated to thecoordinate weights only, and not to the coordinates themselves?46

THANK YOU!

Questions?

Differentiable Least-Squares Fitting Wouter Van Gansbeke, Bert De Brabandere, Davy Neven, Marc Proesmans, Luc Van Gool arXiv:1902.00293v3 [cs.CV] 5 Sep 2019 . Curve fitting mostly same [24] - Towards End-to-End Lane Detection: an Instance Segmentation Approach Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, Luc Van Gool .