Subjective Evaluation Process for Consistent, High-Quality Driving Experience Assessment and Evaluation

Unlocking Excellence: A Smart Way to Evaluate What Matters Most

In today’s fast-evolving automotive landscape, delivering a truly high-quality driving experience requires more than just technical precision—it demands a deep understanding of how drivers and customer feel behind the wheel. MdynamiX, in collaboration with the Institute for Driver Assistance and Connected Mobility (IFM), introduces a methodology that structures automotive subjective evaluations based on the SEA J1441 to enhance the overall assessment quality.

This whitepaper presents a compliant process that helps engineers, developers, and decision-makers to capture real-world driving feedback with clarity and consistency. Using the predefined scale in the innovative MXevalApp, the approach transforms raw driving impressions into actionable insights — enabling faster development cycles, better system tuning, and a driving experience that resonates with customers. The methodology described herein may be applied or referenced, provided that proper attribution is given to the original source.

 

The Whitepaper:

Executive Summary

The MdynamiX Rating Scale (MX-RS) provides a standardized, SAE J1441-compliant [1] method for the subjective evaluation of vehicle quality. The MX-RS enables consistent, high-quality subjective assessment that reflect both technical expertise and customer experience, bridging the gap between measurable performance and perceived quality and enabling data-driven decision-making from subjective assessments. It aligns the perspective of experienced engineers with customer expectations, enabling consistent, customer-oriented evaluations across disciplines. This common language supports transparent communication of quality within development and quality teams and toward end customers. Comparing expert and customer perspectives helps engineers determine appropriate rating levels more intuitively and to align technical improvements with perceived quality. Integration with digital evaluation tools can further enhance efficiency, traceability, and predictive capability for data-driven development.

 

Content

1. Introduction.
2. SAE J1441 Standard Specification.
3. MdynamiX Rating Scale (MX-RS)
4. Combined Expert & Customer Perspective.
5. Application in the Automotive Development Process.
6. Comparison with Other Systems.
7. Challenges and Limitations.
8. Conclusion & Outlook. 10
9. Glossary of Key Terms.

References.

Disclaimer.

 

1.      Introduction

Subjective evaluation of vehicles characteristic – such as steering feel, ride comfort, noise, or usability – are crucial for perceived quality but are often inconsistent and difficult to compare. The MdynamiX Rating Scale (MX-RS) addresses this challenge by combining engineering with customer-oriented perspective.

The MX-RS meets several key requirements: it is based on the recognized SAE standard SAE J1441, is intuitively understandable, it bridges the engineering view and the customer view and allows for nuanced evaluation without overemphasizing minor differences. The scale uses clear verbal categories such as “good” or “poor” and communicates ratings transparently, making them comprehensible to end customers as well. At the same time, the MX-RS ensures comparability with other systems, such as 7-point customer scales, traditional grading systems, or star ratings.

The method promotes a high level of clarity for high-expert engineers up to non-experts, supports discussions about causes and corrective actions, and allows for clear prioritization of improvements. This creates a practical, consistent, and traceable basis for effectively assessing quality both in the development process and in communication with end customers.

When subjective evaluations are carried out according to a clearly defined process – including well-specified criteria, maneuver definitions, and desired outcomes (i.e., what is considered good or poor performance) – and conducted by trained evaluators or backed by subject studies involving multiple participants, the evaluation can be considered high-quality.

Previous studies have also proposed viewing the human driver as an active measuring and evaluation instrument for analyzing vehicle dynamic reactions [4]. By monitoring driver responses – such as steering effort and reaction time – during defined test maneuvers, valuable insights into vehicle dynamics can be obtained. This approach helps link subjective perception with objective vehicle reactions and demonstrates both its potential and limitations.

Moreover, high-quality subjective assessments form the foundation for correlating key performance indicators (KPIs) and objective data with subjective evaluations. This correlation is crucial for virtual development, as it enables engineers to predict customer perception based on simulation or measurement data and to optimize vehicle attributes in the development process.

 

2.      SAE J1441 Standard Specification

The MdynamiX Rating Scale (MX-RS) is based on the recognized SAE standard SAE J1441, which precisely defines the requirements for measuring and evaluating subjective vehicle attributes. SAE J1441 describes a “Subjective Rating Scale for Vehicle Ride and Handling” [1] and serves as a methodological foundation for consistent, reproducible, and comparable assessments of driving dynamics, handling, and comfort.

Beyond its original scope, the SAE concept can be applied not only to driving dynamics, handling, and comfort, but also provides a systematic and standardized framework for the subjective evaluation of additional vehicle-specific characteristics. The standard has been updated through multiple versions, including 1985, 1992, 1998, 2007, 2016, and 2024, to address evolving methodological requirements and vehicle technologies. A core element of the standard is the rating scale, which ranges from 1 (“Not Acceptable”) to 10 (“Excellent”), enabling a nuanced classification of perceived quality.

SAE J1441 also defines the scope of application for these evaluation, covering specific driving maneuvers, road and test conditions, as well as various environmental factors that influence the subjective perception of drivers and evaluators. At the same time, the standard highlights limitations: the scale may not be equally suitable for all driver groups or market segments and should be interpreted contextually.

 

3.      MdynamiX Rating Scale (MX-RS)

The MX-RS can be used by both experts (s. Table 1) and non-experts (s. Table 2), such as customers. It is based on the concept of a Likert-type scale [5], where evaluators assign ordinal ratings to subjective attributes, quantifying perceptions while maintaining interpretability:

• Non-experts: Preferably use whole values (1–10) for clarity and simplicity (s. Table 1).
• Experts: Can also assign half values (e.g., 7.5 or 8.5) to capture finer distinctions and enable more detailed analysis. In Table 2, half-point values such as 7.5 (=Satisfactory *) or 8.5 (=Good *) indicate slightly better performance than the base integer. Here, the “*” sign serves to acknowledge incremental improvements and recognize the developer’s work, highlighting contributions that are truly appreciated.

This flexibility ensures the scale is widely applicable while allowing experienced evaluators to make precise differentiations. According to Lenzo & Naets [8], subjective ride quality is typically scored using SAE J1441 1–10 scales, where 1–5 is considered undesirable and 5–10 acceptable to excellent. Experts and developers should take the target customer into account, work under consistent conditions, and apply structured evaluation to ensure reliable and comparable results.

The following evaluation scales (Table 1 and Table 2) are conceptually based on the SAE J1441 standard [1] but have been adapted and extended by MdynamiX to support both expert and customer-oriented assessments.

Table 1: Detailed High-Quality Subjective Evaluation Scale [adapted from SAE J1441, 1] – for expert evaluators

Table 2: Normal High-Quality Subjective Evaluation Scale [adapted from SAE J1441, 1] – for non-expert evaluators

For specific questions, especially for non-experts, the scale can be limited to a relevant range. For example, it can be used only for mid-to-high values (e.g., 4–10) or for low-to-mid values (e.g., 1–7), depending on the context of the assessment.

Depending on the subjective evaluation criterion, assessments are performed using a bidirectional 10-point rating scale. Typical adjectives illustrate the extremes, such as “too light” … “too heavy” or “too quick” … “too sluggish.” A target rating of 8 typically represents the development goal, while 10 indicates excellent performance.

The SAE J1441 scale is a linear 1–10 system. To enhance interpretability, evaluators may prefix ratings with a plus (+) or minus (−) sign to indicate whether a particular characteristic is excessive or insufficient relative to the target or reference behavior:

• A “−” prefix (e.g., −6) indicates the attribute is too low – such as underdamped ride or slow steering response.
• A “+” prefix (e.g., +6) suggests the attribute is too high – such as overly stiff suspension or excessively high steering wheel torque.

This notation helps distinguish between absolute quality and tuning deviation, making it especially useful in development phases where balancing trade-offs is key. Bidirectional ratings also indicate the required direction of improvement – for example, as illustrated in Figure 1, a rating of “−7” indicates that the steering response was perceived as “too indirect,” suggesting that a more direct response would be desirable. [9]

To ensure meaningful differentiation, the rating scale is interpreted relative to the vehicle segment. The optimal steering response for a luxury vehicle may achieve a “10” (= excellent), whereas the same characteristic in a sports car might only correspond to a “7–” (= too indirect) (s. Figure 1).

 

 

Figure 1: Divergent optimum of different car classes [9]

 

4.      Combined Expert & Customer Perspective

In each category, the perceptions of both experienced evaluators and customers are taken into account. This approach ensures transparency, enhances traceability, and allows for clear prioritization of actions. Key disciplines include Steering, Comfort, Handling, Brakes, Tires, NVH, Drivability, UX, and ADAS/HAD evaluation.

The comparison between the expert and customer perspectives helps engineers determine the appropriate rating level more easily. To do so, they put themselves in the customer’s position and ask: “How would the customer perceive this?”

The MX-RS rating scale is structured to reflect customer perceptions in a detailed, graded manner. It is defined in two levels, separated by a borderline at grade 5:

• From grade 5 downward, customers complain about the functions or perceived quality. Critical customers are likely to contact the workshop.
• From grade 4 downward, customers are likely to contact the workshop or dealer to have the issue checked – or may even stop driving immediately to verify or have it verified. Such inspections inevitably result in costs.
• Grades 6 and above are considered nearly free of complaints. The higher the rating, the greater the customer satisfaction.
• From grade 8 upward, customers are no longer able to articulate any noticeable weaknesses. However, this does not necessarily mean that they do not perceive them; rather, they are unable to differentiate or express them clearly. Nevertheless, ratings above 8 remain relevant for overall customer satisfaction – they make the difference between being best in class or average level.

The ratings are never absolute but always relative to customer expectations. They are oriented toward the benchmark – what customers can buy for their money in a specific vehicle class – and are therefore vehicle-segment dependent. This means, for example, that a grade 8 today will no longer represent a grade 8 in the future. A luxury vehicle faces higher customer expectations than a compact car and must, for instance, deliver significantly e.g. higher comfort to achieve the same grade 8.

The MX-RS can also be applied within development processes such as Quality Function Deployment (QFD), where customer requirements are quantified using the scale and transferred into the House of Quality. This supports the effective prioritization of technical features and the derivation of development actions based on customer expectations [3].

In addition, objective measurement standards such as SAE J2834 (“Ride Index Structure and Development Methodology”, [2]) provide a methodological bridge between physical vehicle parameters and subjective perception. Aligning the MX-RS with such standards enables the correlation of subjective ratings with objective data, thereby supporting virtual development and the early prediction of perceived quality.

 

5.      Application in the Automotive Development Process

The MdynamiX Rating Scale (MX-RS) is used to systematically evaluate vehicle attributes such as Steering, Comfort, Handling, Brake, Tires, NVH, Drivability, UX and ADAS/HAD. It can be applied across all development phases – from prototypes to pre-series vehicles and up to series production release – and enables integration into testing and quality processes. According to SAE J2834 [2], subjective evaluations should be reported immediately after each ride specimen, ideally within a few seconds and always within one minute, to minimize memory effects and ensure accurate data collection. Digital support tools can further improve the consistency and traceability of these evaluations (see Section 8, [10]).

The motivation for applying subjective evaluations at the whole-vehicle level lies in the fact that customer requirements and acceptance tests are typically based on subjective assessments (e.g., perceived comfort or handling). Based on these assessments, objective development targets can be derived, which engineers use to ensure that the vehicle is developed appropriately (verification) and that the developed vehicle meets customer requirements (validation) [11].

Best practices include using comparable test vehicles, controlled environmental conditions, and multiple evaluators to ensure reliable and consistent results.

 

6.      Comparison with Other Systems

The MdynamiX Rating Scale (MX-RS) is comparable to commonly used rating systems, such as 1–10 scales or star ratings, but it combines detailed description which is the basis of clear communication. While many objective measurement methods provide definite numerical values (Key Performance Indicators – KPIs), certain aspects of vehicle behavior – such as comfort, haptics, or subjectively perceived stability – remain difficult to quantify.

Aigner (ATZ, 1982) and Heißing & Brandl emphasize that vehicle subjective evaluations remain important despite improvements in measurement and calculation methods [6, 7]. Subjective assessments help explain and validate measurement results from the perspective of drivers’ and passengers’ experience. It is crucial that potential sources of error are identified and quantified. The MX-RS builds on this approach by combining the expert perspective with the customer viewpoint [3], enabling robust and transparent subjective evaluations.

 

7.      Challenges and Limitations

The MdynamiX Rating Scale (MX-RS) is a subjective evaluation (see above) method that provides developers and evaluators with a common language, ensuring that subjective assessments are based not on individual preferences but on predefined management and customer requirements.

Advantages and Opportunities:

• Common Language: Engineers, testers, and decision-makers share a unified evaluation logic.
• Traceability: Ratings are standardized, documentable, and reproducible.
• Integration into Development Processes: Subjective perceptions are systematically used to verify the achievement of target objectives.

Limitations and Challenges:

• Subjective Elements Remain: Despite standardization, individual evaluator perception may vary.
• Dependence on Training and Methodology: Consistent results are improved by trained evaluators, controlled conditions, and feedback cycles.
• Application Boundaries: Different vehicle types, market segments, or novel functions may limit direct comparability.

When applied, the MX-RS enables efficient prioritization of actions, improved communication between teams, and a reliable translation of management and customer requirements into concrete development goals.

 

8.      Conclusion & Outlook

The MdynamiX Rating Scale (MX-RS) provides a structured, practical subjective evaluation methodology that combines both expert and customer perspectives. In the future, its effectiveness can be further enhanced through digital tools and automated analyses [10]. Such tools can support the consistent application of MX-RS by guiding evaluators through defined maneuvers and systematically capturing ratings and metadata, helping to make evaluations fast and reliable.

 

9.      Glossary of Key Terms

Subjective Evaluation

A qualitative assessment based on human perception, typically involving feelings, impressions, and personal judgment. In the MX-RS context, it refers to how evaluators (experts or customers) perceive vehicle attributes like steering feel, comfort, or noise.

Subjective Assessment

Often used interchangeably with subjective evaluation, but emphasizes the process of assigning ratings or judgments based on subjective impressions. It includes structured rating scales and verbal descriptors to ensure consistency.

Assessment
A broader term encompassing both subjective and objective evaluations. It refers to the act of judging or appraising a vehicle’s performance or quality, often within a defined framework or methodology.

Evaluation
Similar to assessment, but often implies a more systematic and structured process. In the whitepaper, “evaluation” is the preferred term (as noted in the SAE J1441 standard) and is used to describe both subjective and objective rating activities.

Objective Assessment

A measurement-based evaluation using quantifiable data (e.g., KPIs, sensor readings). It contrasts with subjective assessment and is often used to validate or correlate subjective impressions.

Rating Scale

A structured system for assigning numerical or verbal scores to vehicle attributes. The MX-RS uses a 1–10 scale (based on SAE J1441), with optional half-point increments and directional indicators (+/−) to reflect deviations from target behavior.

SAE J1441

An SAE standard defining a subjective rating scale for vehicle ride and handling. It provides the methodological foundation for consistent and reproducible subjective evaluations.

Likert-Type Scale

A psychometric scale commonly used in surveys and evaluations. In MX-RS, it refers to the ordinal rating system (1–10) used to quantify subjective perceptions.

Bidirectional Rating

A rating system that includes directional indicators (e.g., +6 or −6) to show whether a characteristic is excessive or insufficient relative to a target. This helps guide development decisions.

Customer Perspective

The viewpoint of non-expert evaluators, typically end users, whose perceptions are shaped by expectations and experiences. MX-RS integrates this perspective to ensure relevance and satisfaction.

Expert Perspective

The viewpoint of trained engineers or evaluators with technical knowledge. Experts may use finer rating distinctions (e.g., half-points) and interpret results in the context of vehicle development.

Perceived Quality

The subjective impression of a vehicle’s quality, influenced by sensory experiences and expectations. It may differ from measurable performance but is crucial for customer satisfaction.

 

 

References

• [1] SAE International. (2024). SAE J1441: Subjective Rating Scale for Vehicle Ride and Handling (Latest revision). Warrendale, PA: SAE International.
• [2] SAE International. (2025). SAE J2834: Ride Index Structure and Development Methodology. Warrendale, PA: SAE International.
• [3] Schick, B., Fuhr, F., Höfer, M., & Pfeffer, P. E. (2019). Attributbasierte Entwicklung von Fahrerassistenzsystemen – in Kooperation mit Porsche [Attribute-based development of driver assistance systems – in cooperation with Porsche]. Automobiltechnische Zeitschrift (ATZ), 121(4), 38–43.
• [4] Neukum, A., Krüger, H. P., & Schuller, J. (2001). The driver as a measuring instrument for vehicle dynamic reactions? VDI Berichte, No. 1613.
• [5] Koo, T.K., & Yang, M. (2025). Understanding the Likert Scale: A Review of Its Design and Application. Journal of Behavioral Research, 18(2), 45–56. https://doi.org/10.3390/jbr18020045
• [6] Aigner, J. (1982). Zur zuverlässigen Beurteilung von Fahrzeugen [On the reliable assessment of vehicles]. Automobiltechnische Zeitschrift (ATZ), Year, 84.
• [7] Heißing, B., & Brandl, H. (2010). Subjektive Beurteilung des Fahrverhaltens [Subjective assessment of vehicle handling]. Wiesbaden: Vieweg+Teubner Verlag.
• [8] Lenzo, B., & Naets, F. (Eds.) (2026). Vehicle Dynamics, Control and Design. CISM Courses and Lectures, Vol. 620. Cham: Springer. https://doi.org/10.1007/978-3-031-97270-6
• [9] Pfeffer, P., & Harrer, M. (Eds.) (2017). Steering Handbook. Wiesbaden: Springer Vieweg. https://doi.org/10.1007/978-3-319-05449-0
• [10] MdynamiX. (2025). MXevalApp – Evaluation App. Available online: https://mdynamix.de/en/software/mxevalapp-evaluation-app/ (Accessed: October 27, 2025).
• [11] VDI/VDE. (2021). VDI/VDE 2206: Entwicklung mechatronischer und cyber-physischer Systeme [Development of mechatronic and cyber-physical systems]. Düsseldorf: Verein Deutscher Ingenieure e. V. / VDE Verband der Elektrotechnik Elektronik Informationstechnik e. V.

Disclaimer

SAE J1441 and SAE J2834 are standards published by SAE International. The MdynamiX Rating Scale (MX-RS) is a more advanced methodology that builds upon the general principles described in these standards. It is not affiliated with or endorsed by SAE International.

 

Authors

Alessandro Contini¹, Prof. Dr. Peter Pfeffer², Prof. Bernhard Schick³, Matthias Niegl⁴

¹ Vienna University of Technology, Institute of Mechanics and Mechatronics
alessandro.contini@student.tuwien.ac.at
² Munich University of Applied Sciences, Department of Mechanical Engineering
peter.pfeffer@hm.edu
³ Kempten University of Applied Sciences, Institute for Driver Assistance & Connected Mobility
bernhard.schick@hs-kempten.de
⁴ MdynamiX AG, Head of Vehicle Dynamics
matthias.niegl@mdynamix.de