Automated Driving - ADAS/AD | Research
ADAS Testing: From Subjective Customer Preferences To Objective Validation At Scale
Advanced Driver Assistance Systems (ADAS) have been around for quite a while: Modern-day vehicles inevitably come with assistance features such as Adaptive Cruise Control (ACC), Lane-Keep Assist (LKA), Blind Spot Monitors, Traffic Sign Recognition and other features supposed to make driving safer and more comfortable. In contrast to Autonomous Vehicles, ADAS is a huge and profitable market today, and will likely remain so for the foreseeable future.
If you’ve driven a car with such features in the last few years, though, you might have realized that the performance of these systems can vary – sometimes by quite a lot: While some LKA systems do a good job of keeping your vehicle on track, others tend to react too late, or overcorrect and send you right across the opposite lane border rather than properly centering the car between them. Similarly, some ACCs make for a smooth ride, while others may apply the brakes when another vehicle cuts you off after overtaking – rather than just letting their higher speed widen the gap between you, as most human drivers would do.
Why ADAS Performance Varies So Much
One of the reasons for the varying performance of ADAS features is that reliable, objective testing procedures for public roads have been rare.
During real-world testing, car makers and their suppliers routinely record all onboard data from sensors, actuators and more – allowing for in-depth analysis of system failures, near-misses and similar incidents after a drive. However: These onboard systems only record their own version of events – figuratively speaking, what the car thinks happened during the drive. If you want to iron out false positives/negatives, you need to compare this questionable version of events to a more trusted data set – reference data, or “ground truth” data.
Using Lane-Keep Assist as an example: If a test vehicle missed a lane border, overcorrected, or failed completely, you need to closely examine the actual environment/lane borders in that exact position – as well as the vehicle’s relative position and pose in that specific moment.
In the past, this has been a huge task and massive undertaking, requiring lots of on-site engineering manpower and high-precision measurements that were only possible in closed-off, controlled environments – in other words, on proving grounds.
And while proving grounds are an amazing asset for automotive testing, the total scope and variability of their test routes are by definition limited – which makes it a challenge to optimize a system for use on hundreds of thousands of kilometers of open road. Similarly, standardized tests as defined by EuroNCAP or ISO don’t come close to capturing the variety of roads and scenarios a vehicle is sure to encounter during its lifecycle.
To use another metaphor: Imagine driver training only taking place on a single safety course, with drivers unable to learn better performance after passing the test and entering public roads.
To better optimize for the real world, more real world testing is required – and it needs to happen without sacrificing the precision and fidelity of known approaches: It is not feasible having to rely on subjective feedback from test drivers that “it felt maybe a little bit strange somewhere back there.”
This leads us to a second reason why ADAS features differ so much from brand to brand: The lack of an objective standard for how they should perform – and what the criteria for optimal driving pleasure might be. The following portion of this article describes a field-tested approach to solving both of these problems.
From Subjective Feelings To Objective KPIs and Measurements
In a collaboration with the performance car maker Porsche, the companies GeneSys, MdynamiX and atlatec as well as research partner Adrive Living Lab of the Kempten University of Applied Sciences have created a solution to bridge this gap. Together, the partners have introduced a testing process that allows for objective ADAS validation at scale and on open roads, as featured in ATZ magazine (Automobiltechnische Zeitschrift).
The approach consists of 4 steps – in the aforementioned project, it was applied to validate LKA performance:
- The definition of objective criteria for ADAS performance.
- The creation of ground truth environment/HD map data
- The accurate recording of vehicle position/pose reference data during test drives
- In-depth analysis of relevant driving situations and their recreation in the virtual space/simulation
Defining Objective Criteria For Driving Pleasure
The first challenge is already one of the hardest: How do you quantify an emotional quality criterion such as driving pleasure, or the feeling of safety? To solve this challenge, the Kempten University of Applied Sciences has developed a model of 3 layers: Subjective customer assessment, subjective expert assessment and finally a layer of objective vehicle signals.
A series of test drives, workshops, benchmarking campaigns and more produce insights into the subjective preference of customers for how a feature (e. g. an LKA) should perform. These insights are then refined into categories and sub-categories by automotive experts. Finally, the results are matched with related vehicle-level signals and the expected intensity to be measured for each of them, on a scale from none (0) to high (9).
Generating Ground Truth Data At Scale
Automotive OEMs and suppliers of ADAS technology already do test drives on a defined set of routes: These routes are chosen for factors like their variety, internationality, likelihood of certain events and more – and can cover hundreds or thousands of kilometers of public roads across multiple continents.
These routes are a great resource for objective ADAS testing on public roads – if you have access to high-accuracy measurements of their features and a way to generate reference data for the trajectories your test vehicles drive over them.
To this effect, the High Definition (HD) mapping capabilities of atlatec are leveraged to create 3-dimensional maps of test routes on – in this case, on public roads around Stuttgart and Kempten in Southern Germany.
The produced HD maps contain information on lane borders’ types and their position with inch-perfect accuracy. atlatec’s vision-based approach to mapping allows for consistent high accuracy, even in areas with bad GPS reception.
The finished maps are exported into a multi-layered data format allowing for localization and matching of vehicle poses in real time. For the described collaboration, a variation of OpenDRIVE was used.
Test Drive Recording At High Fidelity
Accurately recording and recreating all trajectories driven during testing is made possible by an Automotive Dynamic Motion Analyzer (ADMA) unit by GeneSys: a high-precision motion sensor which allows for differential GNSS correction and is designed specifically for vehicle dynamics testing.
Based on this technology new test methods had to be developed for the objective evaluation of driving characteristics in the ADAS/AD context. Therefore, driver input as well as road and traffic input, control intervention and the resulting vehicle reaction/movement should be evaluated in its 6 degrees of freedom. Derived from the automated lateral control, it is necessary to obtain a high level of knowledge of the road excitation (essentially road markings and surface geometry) and the driver input in order to be able to evaluate the resulting vehicle reaction accordingly. In the case of assisted longitudinal guidance, a high-level knowledge of the surrounding traffic is required.
Like all sensors, environmental sensors such as cameras, radar or lidar are faulty and not available or sufficiently accurate in all situations. This can have a significant impact on driving characteristics: For example, a camera may not be able to reproduce the curvature of a road accurately, which can cause difficulties for the lane-keeping controller. This repeatedly leads to uncertainties if the experienced driving characteristics are a result of the poor performance of sensors, trajectories, controllers, actuators or the poor response of the vehicle influenced by steering, axles, tires and chassis control systems.
In order to investigate this cause and effect chain, a much more accurate reference measurement method should be used as “Ground Truth”.
In addition, an optimized measuring steering wheel allows for precise recording of steering speed/angle and torque/gradient. In the collaboration with Porsche, an original steering wheel was used to fully preserve the brand- and model-specific haptics, control functions and other details, ensuring realistic driver/vehicle interaction.
For the test drives, a comprehensive catalog of defined maneuvers and situations is created by MdynamiX and the University of Applied Science Kempten, ensuring that all relevant scenarios are encountered and recorded.
Turning Data Into Insights And Reproducible Scenarios For Simulation
The use of suitable algorithms makes for precise calculations and the automatic generation of KPI values from the recorded data. For example, the yaw rate and lateral acceleration recorded by the reference system – based on the ground truth curvature – can be matched with the measurements from the onboard system, allowing for accurate measurement of the production system’s deviation from the actual/reference values.
To gain further insights from the data, the digitalized test routes can be imported into automotive simulation tools. This allows for additional MIL/SIL/HIL tests as well as immersive Driver-in-the-Loop simulations.
Additionally, select scenarios encountered and recorded on real-world test drives can be reproduced – allowing for variation of parameters to further narrow down performance limitations.
Supplementing the test setup with atlatec sensor equipment also allows to accurately record real-world traffic and re-create other vehicles’ trajectories in simulations: This is particularly useful when validating ADAS features that are supposed to react to dynamic agents (e. g. Adaptive Cruise Control or Emergency Brake Assist systems). “Scenario fuzzing” again allows for manipulation of the real-world situation and can aid in the hunt for edge/corner cases.
Additional Reading/References
If you’d like to explore this topic further and in more scientific detail, we recommend the following resources:
M. Höfer, F. Fuhr, B. Schick und P. E. Pfeffer, „Attribute-based development of driver assistance systems“ in 10th International Munich Chassis Symposium 2019, P. E. Pfeffer, Hg., Wiesbaden: Springer Fachmedien Wiesbaden, 2020, 293 – 306.
J. Nesensohn, S. Levéfre, D. Allgeier, B. Schick, and F. Fuhr. “An Efficient Evaluation Method for Longitudinal Driver Assistance Systems within a Consistent KPI based Development Process”.
S. Keidler, D. Schneider, J. Haselberger, K. Mayannavar und B. Schick, „Entwicklung fahrstreifengenauer Ground Truth Karten für die objektive Eigenschaftsbewertung von automatisierten Fahrfunktionen“ in 17. VDI-Fachtagung, Hannover, 2019.
B. Schick, C. Seidler, S. Aydogdu und Y.-J. Kuo, „Driving experience vs. mental stress with automated lateral guidance from the customer’s point of view“ in Proceedings, 9th International Munich Chassis Symposium 2018, P. Pfeffer, Hg., Wiesbaden: Springer Fachmedien Wiesbaden, 2019, S. 27–44, doi: 10.1007/978-3-658-22050-1_5.
S. Aydodgu, B. Schick und M. Wolf, „Claim and Reality? Lane Keeping Assistant – The Conflict Between Expectation and Customer Experience“ in 27. Aachener Kolloquium, Aachen, 2018.
D. Schneider, B. Schick, B. Huber und H. Lategahn, „Measuring Method for Function and Quality of Automated Lateral Control Based on High-precision Digital “Ground Truth” Maps“ in 34. VDI/VW-Gemeinschaftstagung Fahrerassistenzsysteme und Automatisiertes Fahren 2018: Wolfsburg, 07. und 08. November 2018, 2018.
B. Schick, F. Fuhr, M. Höfer und P. E. Pfeffer, „Eigenschaftsbasierte Entwicklung von Fahrerassistenzsystemen“, ATZ Automobiltech Z, Jg. 121, Nr. 4, S. 70–75, 2019, doi: 10.1007/s35148-019-0006-2.
To learn more from the parties involved, feel free to reach out directly:
atlatec
Contact: Dr. Henning Lategahn
Email: lategahn@atlatec.de
LinkedIn: https://www.linkedin.com/in/henning-lategahn/
GeneSys
Contact: Peter Anrold
Email: arnold@genesys-offenburg.de
LinkedIn: https://www.linkedin.com/in/peter-arnold-16584355/
Kempten University of Applied Sciences
Contact: Prof. Bernhard Schick
Email: Bernhard.Schick@hs-kempten.de
LinkedIn: https://www.linkedin.com/in/bernhard-schick-1556aa4/
MdynamiX
Contact: Matthias Niegl
Email: Matthias.Niegl@mdynamix.de
LinkedIn: https://www.linkedin.com/in/matthias-niegl-53a661161/
