Voice recognition: robust, efficient, and seamless testing to solve a complex, ever-changing problem for automakers.

Challenge

As automotive manufacturers continue to focus on developing fully integrated, connected IoT vessels on wheels, they have quickly — and rightfully — placed a high emphasis on evolving as the technology does. Moreover, they are elevating the importance of quickly and efficiently problem-solving along the way, particularly in areas like voice recognition.

Indeed, voice recognition technology is one such example because it’s fully embedded into the infotainment systems of connected vehicles while simultaneously communicating with the cloud.

The reality is this – delivering high-quality, reliable solutions for complex technology such as voice recognition has become a major challenge for automakers. This is primarily because what goes on behind the scenes to capture accurate voice recognition involves an endless number of variables that users shouldn’t even have to think about. However, automakers and EagleTC must consider every one of them.

For instance, when a driver is on the road utilizing voice recognition, voice recognition tuning has to consider road conditions, weather, the sound quality of each different vehicle, as well as the engine noises of each vehicle when listening to a prompt. And that’s just focusing on the vehicle.

Additionally, the complexity gets exponentially more difficult when software development teams have to account for genders, the native language of the driver, accents, and enunciation, as well as the rate, tempo, and timbre of each individual driver.

In this case study, EagleTC worked hand-in-hand with a major automaker’s software development team to conduct robust testing. As a result, the team seamlessly and efficiently captured over 1,800 different voice commands. Consequently, EagleTC was able to provide expert consult every step of the way to ensure cost-effective testing and solutions.

Solution

EagleTC was awarded this ongoing multiyear project to perform voice recognition rate testing for different languages, locations, vehicle lines, software releases, infotainment functionalities, and voice recognition solutions.

Tester Setup and Methodology

For any given vehicle line and software release, EagleTC provides 3 males and 3 females to perform US English voice recognition rate testing over a span of 3 days. Specifically, there are 100 commands tested, repeated 3 times, for a total of 300 iterations per tester. The commands cover a wide range of functionalities (i.e., phone calls, contacts, text messages, navigation-point-of-interest, climate and comfort, etc.).

On testing days, two technicians (one male and one female) perform the commands. There are two roles for each team and each technician performs both roles. The first role involves driving at a speed of 65-70 mph and exercising the commands. Meanwhile, the second role focuses on recording results and feeding the prompts to the driver. Once every prompt has been completed three times, the technicians switch roles to ensure consistency.

Test Result Classification

As part of the validation process, results for every command can be either a pass, fail, no response from the server, or not applicable. Therefore, each result is clearly categorized to ensure clarity and consistency. Moreover, the entire testing session is video recorded for playback and accuracy confirmation. Additionally, the technicians take logs for issues found during testing and provide a detailed explanation of their observations during testing (system response, road conditions, and unusual circumstances, such as rainy days, noisy roads, etc.). Furthermore, following the final round, a summary of the test results is then provided to the customer at the completion of all the test rounds.

In addition to English, the same testing is also performed for Canadian French, Mexican Spanish, and Brazilian Portuguese, based on customer needs. As a result, this multilingual approach ensures thorough validation across a variety of regions and languages, further enhancing the reliability of the voice recognition system.

Sample Testing Results

Table (1) illustrates a sample portion of the testing results with arbitrary percentages, including repetition of the same success rate where applicable to reflect expected benchmarks.

	Success Rate %
	Calls	POI	Text
Male Avg	90%	90%	86%
Female Avg	85%	90%	80%
Overall Avg	87.5%	90%	83%
Benchmark Avg	90%	90%	90%

Table (1) – Sample test result for any given vehicle with arbitrary percentages

Table (2) describes the test result verdict meanings. All commands will be assigned one of the verdicts listed below. Each functionality tested is then assigned an average percentage based on the pass rate, per female and male tester (See Table (1) above). Table (3) summarizes the scope for the project.

Table (2) – Test result verdict description

Table (3) – Summary of the project

 

Results

Thanks to agility and thorough testing processes, EagleTC was able to ensure the client delivered the best possible product to its consumers. Moreover, since being awarded the ongoing project in 2020, EagleTC has excelled with every assignment. Currently, the project consists of approximately 50 test sessions per year, all in several regions with several languages based on the client’s needs. Furthermore, EagleTC continuously adapts its process to meet the evolving requirements of each region and language group.

Consequently, with this project, EagleTC continues to demonstrate its ability to:

• Notably, have teams on the ground, deployed and testing, with short-term notice in different regions and for different language speakers.
• Additionally, provide cost-effective solutions achieved by minimizing overhead cost. Proper training for testers and strong program management processes also minimizes overhead.
• Moreover, find and analyze voice recognition rate issues in real time, resulting in improvements for future vehicle model years, providing a better user experience.
• In parallel, provide prompt support to customers and communicate issues and concerns as they arise.
• Likewise, adapt to unpredictable circumstances due to a pandemic. EagleTC has continued to work and deliver accurate and timely results while adhering to CDC guidelines.

Ultimately, EagleTC will always place a high priority on delivering excellence with quality and agility. Furthermore, we aim to provide clients with peace of mind by conducting any assignment without delays and meeting deadlines without exception. Without a doubt, this case study is just one example of that commitment.