Challenge
As automotive manufacturers continue to focus on developing fully integrated, connected IoT vessels on wheels, they have quickly — and rightfully — placed a high emphasis on evolving as the technology does, while elevating the importance of quickly and efficiently problem-solving along the way.
Voice recognition technology is one such example because it’s fully embedded into the infotainment systems of connected vehicles while communicating with the cloud at the same time.
The reality is this – delivering high-quality, reliable solutions for complex technology such as voice recognition has become a major challenge for automakers. That’s because what goes on behind the scenes to capture accurate voice recognition has an endless amount of variables that users shouldn’t even have to think about, but automakers and EagleTC must.
For instance, when a driver is on the road utilizing voice recognition, voice recognition tuning has to consider road conditions, weather, the sound quality of each different vehicle, as well as the engine noises of each vehicle when listening to a prompt. And that’s just focusing on the vehicle.
The complexity gets exponentially more difficult when software development teams have to account for genders, the native language of the driver, accents, and enunciation, as well as the rate, tempo, and timbre of each individual driver.
In this case study, EagleTC worked hand-in-hand with a major automaker’s software development team to conduct robust testing that seamlessly and efficiently captured over 1,800 different voice commands, providing expert consult every step of the way to ensure cost-effective testing and solutions.
Solution
EagleTC was awarded this ongoing multiyear project to perform voice recognition rate testing for different languages, locations, vehicle lines, software releases, infotainment functionalities, and voice recognition solutions.
For any given vehicle line and software release, EagleTC provides 3 males and 3 females to perform US English voice recognition rate testing over a span of 3 days. There are 100 commands tested, repeated 3 times, for a total of 300 iterations per tester. The commands cover a wide range of functionalities (i.e., phone calls, contacts, text messages, navigation-point-of-interest, climate and comfort, etc.).
On testing days, two technicians (one male and one female) perform the commands. There are two roles for each team and each technician performs both roles. The first role is driving at a speed of 65-70 mph and exercising the commands. The second role is recording results and feeding the prompts to the driver. Once every prompt has been completed three times, the technicians switch.
Results for every command can be either a pass, fail, no response from the server, or not applicable. The entire testing session is video recorded for playback and accuracy confirmation. The technicians take logs for issues found during testing and provide a detailed explanation of their observations during testing (system response, road conditions, and unusual circumstances, such as rainy days, noisy roads, etc.). A summary of the test results is provided to the customer at the completion of all the test rounds. The same testing is performed for Canadian French, Mexican Spanish, and Brazilian Portuguese, based on customer needs.
Table (1) illustrates a sample portion of the testing results with arbitrary percentages. The male and female averages are calculated based on the tester’s success rate and the number of voice commands tested for each function. This example shows only a portion of the functionality that is typically tested. Further investigation takes place when the overall average is below the benchmark average.
-
Success Rate %
Calls
POI
Text
Male Avg
90%
90%
86%
Female Avg
85%
90%
80%
Overall Avg
87.5%
90%
83%
Benchmark Avg
90%
90%
90%
Table (1) – Sample test result for any given vehicle with arbitrary percentages
Table (2) describes the test result verdict meanings. All commands will be assigned one of the verdicts listed below. Each functionality tested is then assigned an average percentage based on the pass rate, per female and male tester (See Table (1) above). Table (3) summarizes the scope for the project.
Test Result Verdict |
Indication |
Pass |
System recognized the voice command properly |
Fail |
System misrecognized the voice command |
Not Applicable (N/A) |
Feature isn’t supported by system |
No Response from server |
Cloud sever isn’t responding. |
Table (2) – Test result verdict description
Languages |
US English, Mexican Spanish, Canadian French, Brazilian Portuguese |
Location of testing |
Detroit and Montreal |
Model Years |
2020, 2021,2021, 2022, etc.. |
Vehicle Engine |
Gas, Hybrid, and Electric |
Speed |
65-70 mph |
Road Type |
Highways |
Voice Recognition Solutions |
Embedded, and Cloud Based |
Voice Commands |
Phone calls, text messages, phonebook, navigation, etc… |
Table (3) – Summary of the project
Results
Thanks to agility and thorough testing processes, EagleTC was able to ensure the client delivered the best possible product to its consumers. Since being awarded the ongoing project in 2020, EagleTC has excelled with every assignment. For now, the project consists of approximately 50 test sessions per year, all in several regions with several languages based on the client’s needs.
With this project, EagleTC continues to demonstrate its ability to:
- Have teams on the ground, deployed and testing, with short-term notice in different regions and for different language speakers.
- Provide cost-effective solutions achieved by minimizing overhead cost. Proper training for testers and strong program management processes also minimizes overhead.
- Find and analyze voice recognition rate issues in real time, resulting in improvements for future vehicle model years, providing a better user experience.
- Provide prompt support to customers and communicate issues and concerns as they arise.
- Adapt to unpredictable circumstances due to a pandemic. EagleTC has continued to work and deliver accurate and timely results while adhering to CDC guidelines.
EagleTC will always place a high priority on delivering excellence with quality and agility, while providing clients with peace of mind by conducting any assignment without delays and meeting deadlines without exception, and this case study is just one example of that commitment.