Voice recognition: robust, efficient, and seamless testing to solve a complex, ever-changing problem for automakers.

Challenge

As automotive manufacturers continue to focus on developing fully integrated, connected IoT vessels on wheels, they have quickly — and rightfully — placed a high emphasis on evolving as the technology does, while elevating the importance of quickly and efficiently problem-solving along the way.

Voice recognition technology is one such example because it’s fully embedded into the infotainment systems of connected vehicles while communicating with the cloud at the same time.

The reality is this – delivering high-quality, reliable solutions for complex technology such as voice recognition has become a major challenge for automakers. That’s because what goes on behind the scenes to capture accurate voice recognition has an endless amount of variables that users shouldn’t even have to think about, but automakers and EagleTC must.

For instance, when a driver is on the road utilizing voice recognition, voice recognition tuning has to consider road conditions, weather, the sound quality of each different vehicle, as well as the engine noises of each vehicle when listening to a prompt. And that’s just focusing on the vehicle.

The complexity gets exponentially more difficult when software development teams have to account for genders, the native language of the driver, accents, and enunciation, as well as the rate, tempo, and timbre of each individual driver.

In this case study, EagleTC worked hand-in-hand with a major automaker’s software development team to conduct robust testing that seamlessly and efficiently captured over 1,800 different voice commands, providing expert consult every step of the way to ensure cost-effective testing and solutions.

Solution

EagleTC was awarded this ongoing multiyear project to perform voice recognition rate testing for different languages, locations, vehicle lines, software releases, infotainment functionalities, and voice recognition solutions.

For any given vehicle line and software release, EagleTC provides 3 males and 3 females to perform US English voice recognition rate testing over a span of 3 days. There are 100 commands tested, repeated 3 times, for a total of 300 iterations per tester. The commands cover a wide range of functionalities (i.e., phone calls, contacts, text messages, navigation-point-of-interest, climate and comfort, etc.).

On testing days, two technicians (one male and one female) perform the commands. There are two roles for each team and each technician performs both roles. The first role is driving at a speed of 65-70 mph and exercising the commands. The second role is recording results and feeding the prompts to the driver. Once every prompt has been completed three times, the technicians switch.

Results for every command can be either a pass, fail, no response from the server, or not applicable. The entire testing session is video recorded for playback and accuracy confirmation. The technicians take logs for issues found during testing and provide a detailed explanation of their observations during testing (system response, road conditions, and unusual circumstances, such as rainy days, noisy roads, etc.). A summary of the test results is provided to the customer at the completion of all the test rounds. The same testing is performed for Canadian French, Mexican Spanish, and Brazilian Portuguese, based on customer needs.

Table (1) illustrates a sample portion of the testing results with arbitrary percentages. The male and female averages are calculated based on the tester’s success rate and the number of voice commands tested for each function. This example shows only a portion of the functionality that is typically tested. Further investigation takes place when the overall average is below the benchmark average.

Success Rate %

Calls

POI

Text

Male Avg

90%

90%

86%

Female Avg

85%

90%

80%

Overall Avg

87.5%

90%

83%

Benchmark Avg

90%

90%

90%

Table (1) – Sample test result for any given vehicle with arbitrary percentages

Table (2) describes the test result verdict meanings. All commands will be assigned one of the verdicts listed below. Each functionality tested is then assigned an average percentage based on the pass rate, per female and male tester (See Table (1) above). Table (3) summarizes the scope for the project.

Test Result Verdict

Indication

Pass

System recognized the voice command properly

Fail

System misrecognized the voice command

Not Applicable (N/A)

Feature isn’t supported by system

No Response from server

Cloud sever isn’t responding.

Table (2) – Test result verdict description

Languages

US English, Mexican Spanish, Canadian French, Brazilian Portuguese

Location of testing

Detroit and Montreal

Model Years

2020, 2021,2021, 2022, etc..

Vehicle Engine

Gas, Hybrid, and Electric

Speed

65-70 mph

Road Type

Highways

Voice Recognition Solutions

Embedded, and Cloud Based

Voice Commands

Phone calls, text messages, phonebook, navigation, etc…

Table (3) – Summary of the project

 

Results

Thanks to agility and thorough testing processes, EagleTC was able to ensure the client delivered the best possible product to its consumers. Since being awarded the ongoing project in 2020, EagleTC has excelled with every assignment. For now, the project consists of approximately 50 test sessions per year, all in several regions with several languages based on the client’s needs.

With this project, EagleTC continues to demonstrate its ability to:

  • Have teams on the ground, deployed and testing, with short-term notice in different regions and for different language speakers.
  • Provide cost-effective solutions achieved by minimizing overhead cost. Proper training for testers and strong program management processes also minimizes overhead.
  • Find and analyze voice recognition rate issues in real time, resulting in improvements for future vehicle model years, providing a better user experience.
  • Provide prompt support to customers and communicate issues and concerns as they arise.
  • Adapt to unpredictable circumstances due to a pandemic. EagleTC has continued to work and deliver accurate and timely results while adhering to CDC guidelines.

EagleTC will always place a high priority on delivering excellence with quality and agility, while providing clients with peace of mind by conducting any assignment without delays and meeting deadlines without exception, and this case study is just one example of that commitment.