Index

Challenge

Infotainment interoperability testing was a critical requirement for a major Japanese Original Equipment Manufacturer (OEM) preparing to launch its infotainment products globally. The OEM aimed to enhance product quality by conducting interoperability (IOP) testing across various regions. This testing ensured compatibility with a broad range of phone manufacturers and cellular carriers, improving the user experience across multiple vehicle models. Leveraging its global reach and proven expertise, EagleTC secured a multi-year contract to perform infotainment interoperability testing in North America (including the USA, Canada, and Mexico) and the Middle East (i.e., Jordan).

Solution

EagleTC collaborated closely with the automaker to create a detailed testing plan that covered multiple vehicle platforms. Before testing began, EagleTC addressed several foundational elements to ensure a successful project launch. These included Test Case Development, Phone Selection and Procurement, Test Bench Setup, Guideline Documentation, and Test Execution.

Test Case Development

EagleTC’s engineers began by focusing on developing detailed test cases for infotainment interoperability testing. They built these cases based on technology standards, the automaker’s requirements, and EagleTC’s own engineering expertise. As the process progressed, EagleTC and the automaker collaborated through a series of structured reviews to finalize a comprehensive list of test cases for each vehicle platform. As shown in Figure (1), this workflow ensured alignment and precision. Additionally, Table (1) provides a clear summary of the technologies and features evaluated during the testing.

Phone Selection/Procurement

In parallel with test case development, EagleTC took full responsibility for securing the phones required for infotainment interoperability testing. To guide the selection process, the team conducted market research to identify a wide variety of phones and cellular carriers. Because testing every device on the market wasn’t feasible, EagleTC prioritized devices based on three factors: market research insights, customer requests, and regional availability. As a result, the team selected over 200 phones from 15 different manufacturers, covering model years from 2017 to 2024. Table (2) highlights the distribution of selected phone manufacturers and cellular carriers.

Once the phone list was finalized, EagleTC prepared each device for testing. This preparation included updating all phones and applications to their latest software versions. In addition, engineers downloaded music, created sample contacts with multiple numbers and photo formats, and configured the devices for log generation.

Figure (1) – Test Case Development Workflow.

Table (1) – List of Technologies and Features Tested.

Table (2) – Distribution of Phone Manufacturers and Cellular Carriers.

Test Bench Setup

A predefined number of test benches were shipped to each region based on the vehicle platform and testing timeline. Before starting each testing session, the engineers verified that the benches in every region were operating correctly and running the appropriate software version as specified by the OEM.

Guideline Document

Maintaining consistency across four countries and numerous testing cycles was paramount. To achieve this, EagleTC generated and followed a common set of testing guidelines for both the infotainment systems and the phones for each of the four countries. The guidelines included pre-test preparation for the infotainment system and phone, instructions for generating all logs, performing infotainment system software updates, and conducting all IOP test cases. They also outlined the format in which test results should be submitted to the OEM, ensuring clear and uniform reporting.

Test Execution

With the guidelines in place, EagleTC’s engineers conducted the IOP testing and provided videos and system logs for the phone and infotainment system, for all identified issues. These logs included a Gal Monitor log, Bug Reports, and an infotainment system log enabling the OEM to review and resolve issues as necessary. Figure (2) illustrates the complete IOP testing process.

All issues identified were tracked through EagleTC’s defect database, allowing team members in the different regions to collaborate efficiently, especially when identifying the same issues across the different regions. After reviewing internally, EagleTC entered the identified issues into the OEM’s defect database for review and resolution. In addition, EagleTC mapped the tested features and their results to coordinate with the customer’s website. Table (3) is an example of test result mapping.

Figure (2) – IOP Testing workflow.

Results

EagleTC continued to deliver excellence with quality and agility to the OEM during all the testing cycles. The following highlights EagleTC’s competence in providing this service to the customer:

• Test Cases: Over 350 test cases were executed per phone, validating interoperability between key technologies and features.
• Testing Guidelines: Guidelines were created to ensure consistency and reliability across all regions for each testing cycle.
• Phones: A total of 250 phones were tested across all regions, reflecting a wide variety of manufacturers, carriers, and model years.
• Timely Project Delivery: EagleTC delivered results ahead of schedule for all testing cycles, maintaining integrity, efficiency and cost-effectiveness.

This case study highlights EagleTC’s expertise in global infotainment interoperability testing and its ability to deliver scalable, agile, and high-quality solutions for next-generation in-vehicle experiences.

Key Takeaways from EagleTC’s Infotainment Interoperability Testing

• Deploying teams on short notice without compromising expertise or quality.
• Delivering completed projects with excellence, on time or ahead of schedule.
• Adapting seamlessly to changing conditions while maintaining consistent and accurate results.
• Maintaining open lines of communication with our customers, allowing real-time reporting and resolutions.

By combining global resources with deep automotive expertise, EagleTC ensures its customers receive world-class solutions—every single time.

Challenge

Building a safe, efficient, and user-friendly connected vehicle ecosystem requires collaboration between original equipment manufacturers (OEMs), technology providers, and regulatory bodies. To ensure reliable performance, especially in new markets, EV field testing plays a critical role. A comprehensive understanding of these technologies is essential for the efficient development, testing, and integration of advanced in-vehicle systems.

In the current global market, OEMs are determined to provide their consumers with a next-level driving experience.

A Chinese OEM plans to launch a high-end electric vehicle (EV) in North America (USA and Canada) in 2025. High-end vehicles, integrated with advanced features, have a higher likelihood of encountering issues, necessitating rigorous testing to ensure quality and customer satisfaction. Field testing ensures vehicles are optimized for real-world challenges, meeting consumer expectations and regulatory standards while enhancing the overall driving experience.

EagleTC partnered with the OEM to perform comprehensive field system level testing, targeting, but not limited to, the infotainment system, telematics, digital key, and Heads-Up Display (HUD) for the North American market.

Solution

EagleTC assigned a dedicated team of test engineers and a program manager to lead this project. The team received a comprehensive list of general test cases to validate the vehicle’s features. Figure (1) illustrates a high-level overview of these features, while Table (1) provides detailed descriptions of each. The project was structured into three key phases:

Phase One (Planning):

• Reviewed and customized the test cases based on North American standards, leveraging EagleTC’s regional expertise to enhance coverage.
• Aligned testing activities with the software release schedule.
• Planned multiple long-distance trips for mileage accumulation evaluate and to evaluate software stability.

Figure (1) – List of Features Tested.

Phase Two (Functional Testing):

• Updated infotainment system software using Over-the-Air (OTA) when new versions were released.
• Used Diagnostic Software Application (DSA) to update telematics and EV Communication Controller (EVCC) firmware as needed.
• Conducted smoke and sanity tests on each build and promptly communicated results to the OEM.
• Performed vehicle level functional testing for the features listed in Table (1) and updated results in SharePoint.
• Reported software issues in JIRA daily, attaching relevant logs and videos. For instance, QUALCOMM eXtensible Diagnostic Monitor (QXDM) and Android Debug Bridge (ADB) commands.
• Held daily review calls with the OEM to discuss testing details and new software releases.
• Retested to confirm that identified issues were resolved.

Phase Three (Field Testing):

• The team embarked on multiple long-distance trips, with Figure (2)depicting a sample planned route. The focus of these drives included:
  • Cross-functional and ad-hoc testing to assess the functionality of key features such as navigation, tuner, telephony, phone projection, voice recognition, and system settings.
  • Infotainment software stability and performance evaluation under real-world driving conditions.
  • Road environment testing to analyze performance across varying conditions, including tunnels, bridges, high-traffic areas, city centers with tall buildings, and border crossings.
  • Telematics connectivity assessment across different regions and major carriers, including AT&T, Verizon, T-Mobile, Tellus, and Bell.
  • Verification of EV charging interoperability with various Electric Vehicle Supply Equipment (EVSE) manufacturers and charging networks. (For further details, please click here).
• Continuously uploaded test results, logs, and photos to the customer portal for immediate analysis.

Figure (2) – Sample Planned Route for Testing.

Results

Since commencing this ongoing project in 2024, EagleTC has closely collaborated with the OEM to conduct field tests across the United States and Canada. Notable achievements include:

• Seamlessly transferring knowledge through comprehensive project documentation.
• Accumulating over 16,000 miles and testing more than 600 EVSE units across seventeen U.S. states and two Canadian provinces.
• Providing valuable, real-time feedback and assisting the OEM in resolving all software issues by continuously testing pre-development builds.
• Summarizing test results for the OEM after completing key milestones.

This project illustrates EagleTC’s capability to:

• Deploy on-ground teams with strategic route planning on short notice.
• Implement cost-effective testing solutions through efficient resource allocation, rigorous training, and effective program management.
• Deliver prompt customer support and communicate issues in real-time.
• Adapt to dynamic conditions while consistently achieving accurate and timely results.

Challenge

The e-mobility ecosystem has grown significantly over the past few decades. In the early stages, electric vehicles (EVs) faced challenges due to battery limitations, short driving range, and a lack of charging infrastructure. However, improvements in battery technology and charging networks have increased the practicality and appeal of EVs. As the industry advances, EV interoperability field testing has become essential to ensure seamless integration between vehicles and diverse charging infrastructure.

This growing need for compatibility has led to an expanding ecosystem involving EV manufacturers, charging network providers, and electric vehicle supply equipment (EVSE) manufacturers. Reliable communication between these systems is critical to delivering a smooth user experience.

According to recent reports, market analysts valued the global EV charging infrastructure market at USD 25.83 billion in 2023. They project it will grow at a compound annual growth rate (CAGR) of 25.4% from 2024 to 2030 [1].

Figure (1) illustrates the main components of EV charging infrastructure. The Charging Station Management System (CSMS) oversees charging sessions via the Open Charge Point Protocol (OCPP). OCPP enables real-time communication between EVSE and CSMS, ensuring accurate data exchange during each charging session.

Figure (1) – Components of EV Charging Infrastructure.

In 2025, an Original Equipment Manufacturer (OEM) plans to launch a new high-end EV in North America, covering the USA and Canada. With the rapid expansion of the EV charging infrastructure, potential compatibility challenges may emerge between EVs and EVSE from different manufacturers. Given the diverse set of players contributing to the e-mobility ecosystem, ensuring seamless interoperability across different EVs and charging equipment necessitates rigorous testing and validation efforts.

To address these challenges, EagleTC has partnered with the OEM to conduct ongoing field testing aimed at identifying and resolving interoperability issues specific to the USA and Canadian markets.

Solution

EagleTC assigned three dedicated test engineers and a program manager exclusively to this project. With the launch of the new EV in North America, the team designed a comprehensive testing plan to ensure broad coverage of various EVSEs across multiple states. This plan aimed to verify interoperability across a wide range of charging networks and equipment. Throughout the project, EagleTC collaborated closely with the customer and carried out the following key activities.

EVSE Selection and Coverage

To begin, EagleTC compiled a list of EVSEs for testing based on several key criteria, including manufacturer, charging network, charging speed, and geographic distribution, as shown in Table (1). In addition, the team conducted testing at various locations across both the USA and Canada. This approach ensured the inclusion of diverse charging networks and equipment manufacturers, effectively simulating a wide range of real-world conditions.

Table 1 – Methodology for Populating Multiple EVSE.

Route Planning

Routes, addresses, charging station availability, charging speeds, and plug types were planned using PlugShare. Figure (2) illustrates an example of a planned route.

Figure (2) – Planned Route for Testing.

App Integration and Payment Setup:

To begin, various EV charging station apps were downloaded, and payment accounts were set up for different providers, including Electrify America, Red E, ChargePoint, EVGO, and Tesla. As a result, the team ensured full readiness for real-time testing at each charging station. For reference, Figure (3) provides an example of one of the downloaded apps.

Figure (3) – Example of Downloaded App.

Daily Testing and Data Upload

The test team conducted daily drives to specific EVSEs based on the planned routes. Test results, Vector CAN logs, photos, EVSE details, and additional observations were uploaded in real time to the customer’s portal. This allowed the customer to analyze the data immediately and provide quick feedback.

Daily Review Meetings

EagleTC held daily calls with the customer to discuss testing results, review planned routes, and align on schedules for new software releases and fixes.

Software Updates and Retesting

The Electric Vehicle Communication Controller (EVCC) received both official and pre-development software updates. After each update, the team performed retesting to confirm that identified issues had been resolved and to ensure consistent data collection.

Participation in CharIN Testival 2024

EagleTC joined the OEM engineers at the CharIN Testival NORTH AMERICA 2024 in Ohio. Our team supported live interoperability testing, captured log data, and assisted with on-site troubleshooting and investigation.

Results

Since 2024, EagleTC has partnered with the customer to conduct EV charging ground testing across North America. After completing the first major milestone, EagleTC delivered valuable insights, thoroughly documenting issues and key observations. These results helped the OEM evaluate performance and implement targeted improvements.

• Extensive EVSE Testing: EagleTC tested over 500 EVSE units across 17 U.S. states and 2 Canadian provinces, as shown in Table 1. The team uploaded test data daily, allowing the customer to review findings and trends in real time.

• Effective Issue Resolution: EagleTC worked alongside the OEM to resolve all software-related issues. The team continuously tested pre-development builds to confirm fixes and maintain performance stability. EagleTC also participated in key industry events with the OEM, providing added support and contributing to successful interoperability outcomes.

• Milestone Summary: At the conclusion of the first milestone, EagleTC compiled a clear, concise report of the test results. This summary helped the OEM quickly understand the key findings and prioritize next steps.

This project demonstrates EagleTC’s ongoing ability to:

• Deploy teams on the ground with short notice, while developing strategic plans and testing routes efficiently.
• Implement cost-effective solutions by minimizing overhead costs through efficient resource allocation, comprehensive tester training, and robust program management processes.
• Provide prompt customer support, ensuring issues and concerns are communicated in real time.
• Adapt seamlessly to changing circumstances while consistently delivering accurate and timely results.

^[1] EV Charging Infrastructure Market Size & Share Report 2030, published by Grand View Research. Source: https://www.grandviewresearch.com/industry-analysis/electric-vehicle-charger-and-charging-station-market

Challenge

Electric Vehicles (EVs) are gaining increasing popularity in the United States. Unlike traditional vehicles powered by internal combustion engines (ICE), EVs utilize an electric motor and rechargeable batteries for propulsion. With the rising number of EVs on the road, the development of charging infrastructure has become a significant challenge. As a result, EV ground truth testing has become essential to ensure the accuracy, reliability, and usability of charging location data across different platforms and networks.

Currently, numerous companies offer electric charging services across the US and globally, including well-known names like Electrify America, SemaConnect, ChargePoint, and EVgo. Furthermore, public charging stations are becoming more prevalent in urban areas and along major highways. However, EV drivers planning long trips encounter substantial challenges. Specifically, they must decide where to charge their EVs, considering factors such as the lack of infrastructure in many rural areas, compatibility with charging ports like NCAS/CCS/CHAdeMO, available charging levels, and the real-time status of these stations.

To alleviate EV driver anxiety and enhance customer acceptance of EVs, Original Equipment Manufacturers (OEMs) are strategizing to furnish their customers with real-time information, regardless of their travel destination.

Unfortunately, much of the data provided by charging companies has proven to be inaccurate, primarily due to continuous software updates. This discrepancy poses significant obstacles for drivers, especially when their vehicles have a low charge. Therefore, EagleTC has forged a partnership with an OEM to conduct ground truth testing aimed at identifying inaccuracies. Ground truth testing involves validating the accuracy and reliability of data provided by charging companies to the OEM, which is then displayed on their infotainment systems.

Solution

EagleTC deployed four test engineers and a program manager exclusively to the project. Following a thorough examination of the OEM requirements, the engineering team devised a comprehensive two-phase approach for EV ground truth testing. These two phases, described below, were strategically designed to ensure accuracy and reliability throughout the testing process.

Phase One: Formulation and Planning

• EagleTC developed a suite of test cases that aligned with customer requirements. Specifically, examples of requirements validated in these test cases include:
  
  • Accuracy of address and route guidance.
  • Real-time availability of various charger types.
  • Operating hours and payment options at each station.
  • Distinguishing between public and private access.

• To support this effort, EagleTC also devised driving routes based on a predefined list of EV charging locations spanning multiple cities and states. Importantly, the list encompassed diverse charging companies offering different types of chargers, located in places such as apartment communities, parking structures, and store parking lots. Refer to Figure (1) for a sample route featuring EV charging station data.

• In preparation for Phase Two testing, EagleTC downloaded various EV charging station apps for comparison with the infotainment systems. Through this meticulous approach, the team ensured that the apps were fully set up and ready for thorough examination. See Figure (2) for a glimpse of the apps and the data they provide.

Overall, this strategic and detail-oriented methodology underscores EagleTC’s commitment to delivering precise and reliable ground truth testing results.

Figure (1) – Sample route with EV charging station data.

Figure (2) – Example of EV charging station apps data.

Phase Two: Groundwork

In this phase, the testing unfolded across two vehicles equipped with distinct infotainment platforms. The essential equipment for ground truth testing encompassed laptops, cameras, power stations, a jump starter tool set, hotspot devices, and phones.

• Refer to Table (1) for a comprehensive breakdown of the coverage of charging companies, charger types, locations, and the apps employed.

• Next, our adept test engineers embarked on a journey to each identified charging station along the designated route. At each location, they meticulously gathered all pertinent details for every test case. Subsequently, the gathered data was compared with the information displayed on both the infotainment system and the charging company’s app. A test case was deemed unsuccessful if any disparities were detected in the data. To substantiate the testing process, pictures were taken at each location, serving as evidence for the testing conducted.

Thus, this meticulous approach ensures the thoroughness and accuracy of our ground truth testing, demonstrating our unwavering commitment to delivering reliable results.

Table (1) – Coverage of Ground Truth Testing.

Results

Since being entrusted with this project in 2023, EagleTC has collaborated closely with the customer to conduct EV charging ground truth testing in numerous locations across the United States. Right from the outset, our team worked in close alignment with the customer’s objectives. Over the course of this engagement, we have consistently provided valuable feedback. In parallel, we meticulously documented inaccuracies in the data disseminated to consumers. As detailed below, the following highlights underscore EagleTC’s competence in executing this project with distinction:

• EagleTC conducted successful tests on over 150 EV charging stations spanning six states. Detailed test results were presented, accompanied by documented photos, and logs were furnished as needed.
• Navigation issues and other noteworthy observations related to EV charging data were promptly reported by EagleTC
• Leveraging their findings, EagleTC delivered a comprehensive list of lessons learned to the OEM, aiming to enhance processes, expand testing coverage, and optimize costs.
• Daily progress reports were diligently provided to the customer, ensuring transparency and constant communication.
• Upon completion of the project, EagleTC compiled a concise summary of the test results for the OEM.

This project exemplifies EagleTC’s ongoing ability to:

• Mobilize ground teams swiftly, creating and implementing test cases and processes with short notice.
• Offer cost-effective solutions by minimizing overhead costs, assigning appropriate resources for designing test strategies, providing thorough training for testers, and employing robust program management processes.
• Identify and analyze issues in real-time, resulting in continuous improvements for future projects and an enhanced user experience.
• In addition, EagleTC adapted seamlessly to variable circumstances while consistently delivering precise and timely results.

Challenge

In today’s digital age, cyber threats are rapidly evolving and becoming increasingly sophisticated. As a result, utility companies face growing pressure to improve cybersecurity for the electric grid. They must not only invest continuously but also ensure those investments are effective and financially justifiable.

However, measuring the true impact of cybersecurity spending remains a challenge. The threat landscape shifts constantly, and grid infrastructure is highly critical. Consequently, the utility sector has long struggled to quantify the value of its cybersecurity efforts.

To address this, our client launched a research project in 2015. The goal was to build a robust cybersecurity metrics framework. By 2018, they developed version two, aiming for objective, consistent metrics covering protection, detection, and response.

Despite the progress, significant challenges remained. Extensive pilot testing and usability surveys revealed two key issues: complex data collection and the urgent need for standardized metrics across the industry.

>>>>>>>>>As the cybersecurity landscape becomes increasingly perilous and complex, the challenge now is not only to refine these metrics but to transition them from a procedural burden to an operational necessity. This case study explores the critical next steps in this pivotal project—streamlining processes, achieving alignment with industry benchmarks, and transforming cybersecurity metrics from a theoretical framework to a practical, actionable tool that can reinforce the security posture of utilities on a global scale.

Solution

EagleTC delivered software development expertise to the client, playing a crucial role in the rollout and standardization of cutting-edge security metrics through a carefully designed pilot project. This effort was directed at reinforcing the utility sector’s security framework by putting into practice and benchmarking state-of-the-art security metrics.

Spanning from 2019 to 2021, this pilot initiative aimed to systematically overhaul and enhance the cybersecurity protocols employed by utility companies. Figure (2) displays the scope of the pilot project.

The scope of the project encompasses several key development areas:

• Customizing the client’s metric calculation tool to align with the specific requirements of each utility company.
• Automating the data collection process to minimize manual efforts and enhance efficiency.
• Establishing benchmark procedures to enable a comparative analysis of security measures against industry standards.
• Leveraging advanced analytics to pinpoint and address opportunities to enhance security readiness.
• Developing detailed processes and educational materials to ensure the effective implementation of the tool and to foster widespread user acceptance.

Through EagleTC’s collaboration, the pilot project is not just operationalizing and benchmarking security metrics but also empowering utility companies to adopt a more proactive and informed approach to their cybersecurity strategies.

Results

The outcome of this venture emphasizes EagleTC’s competence to:

• Swiftly assemble and deploy a specialized team of experts, ensuring immediate engagement and start time of project.
• Rapidly deploy tailored tools and processes, accommodating immediate needs with minimal lead time.
• Provide the Utility company with a quantitative measure of the effectiveness of the security measure.
• Offer cost-effective, robust software development solutions that are aligned with the nuances of organizational and industry standards.

In essence, EagleTC exemplifies dedication to quality and adaptability, providing our clients with assurance that projects will progress seamlessly, upholding the highest standards of execution and punctuality.

Challenge

Over the Air (OTA) commonly refers to the ability to download applications, services, and configurations over a mobile or cellular network. In the automotive industry, OTA updates enable OEMs to eliminate the need for software-related recalls and make software updates easy and seamless, allowing for better maintainability of the overall system. The two key categories of OTA updates for automobiles are infotainment and drive control. These updates improve the overall driving experience and are considered critical.

Why Over The Air Updates Are Becoming Essential

Over the Air is becoming a necessity in the automotive industry. Most recall campaigns are software related and the amount of software used in vehicles continues to increase. The complexity of the vehicles, drivers’ behavior, and safety regulations mandate thorough testing to capture all scenarios that could occur while performing an OTA update for the different electronic control units (ECUs).

For the OTA update to start, the backend Server has to recognize the availability of a new software package to be downloaded. Once recognized, the OTA process starts which consists of three phases:

1. Software package downloads from the server to the vehicle, in the background.
2. Software package starts unpacking in the background, internally in the infotainment system.
3. Software Package installation process starts in the foreground.

Figure (1) describes an overview of the OTA architecture. Automotive companies must ensure the system has enough CPU power and memory so that the load caused from the downloading does not affect normal use. The in-vehicle system must have a roll-back process in case of download failure. The driver should never experience issues utilizing the system, nor any decline in performance due to an OTA update. In addition, throughout the OTA process, the vehicle must be in safe state to drive. Depending on the size of the software package, the speed and coverage of the cellular network, an OTA download and system update, can take up to eight hours. Since the average commute to work is twenty-six minutes, one OTA could easily take a whole week for the average user.

In this case study, EagleTC worked together with a major automaker’s software team to conduct system level testing. We captured all phases of the OTA update for their infotainment systems, across all of their vehicle lines.

Solution

EagleTC’s engineers reviewed the specification provided by the automotive automaker for the OTA process. Test cases were then developed to test all the OTA features, this included high-level features of the infotainment system during all OTA phases, as well as different scenarios that could affect performance. Testing activities for this case study as well as the different scenarios (i.e., by changing different variables) are described in in Table (1).

Table (1) – OTA Phases, Variables to Consider and Testing Activities

*It is possible that a user loses connection to the server during an OTA. When that occurs, engineers need to verify the OTA handles the disconnection properly. When server connection is lost, the OTA should pause and resume from the same spot. It should not revert to the beginning of the OTA download. Test cases are performed through the OTA to confirm it is functioning properly.

Results

Since being awarded this project in 2022, EagleTC has worked hand in hand with the customer to perform Over the Air (OTA) testing. Engineers developed test cases and executed testing according to the guidelines outlined in Table (1). They captured logs for any issues encountered during testing and provided detailed explanations of their observations — including system responses and unusual circumstances — while also investigating potential fixes. Each day, the team delivered progress reports to the customer, outlining issues and the current status of software downloads and updates. At the completion of each vehicle, EagleTC also delivered a summary of the test results. Through this project, EagleTC continues to demonstrate its ability to:

• Have teams on the ground, deployed, developing test cases and testing, with short-term notice.
• Provide cost-effective solutions achieved by minimizing overhead cost. Assigning the appropriate resources for designing test strategy, proper training for testers and strong program management processes also minimizes overhead.
• Find and analyze OTA issues in real time, resulting in improvements for future vehicle model years, providing a better user experience.
• Provide prompt support to customers and communicate issues and concerns as they arise.
• Adapt to unpredictable circumstances due to process issues (i.e., server issues, corrupted software packages, vehicle problems, etc.). EagleTC continues to work and deliver accurate and timely results.

Challenge

As automotive manufacturers continue to focus on developing fully integrated, connected IoT vessels on wheels, they have quickly — and rightfully — placed a high emphasis on evolving as the technology does. Moreover, they are elevating the importance of quickly and efficiently problem-solving along the way, particularly in areas like voice recognition.

Indeed, voice recognition technology is one such example because it’s fully embedded into the infotainment systems of connected vehicles while simultaneously communicating with the cloud.

The reality is this – delivering high-quality, reliable solutions for complex technology such as voice recognition has become a major challenge for automakers. This is primarily because what goes on behind the scenes to capture accurate voice recognition involves an endless number of variables that users shouldn’t even have to think about. However, automakers and EagleTC must consider every one of them.

For instance, when a driver is on the road utilizing voice recognition, voice recognition tuning has to consider road conditions, weather, the sound quality of each different vehicle, as well as the engine noises of each vehicle when listening to a prompt. And that’s just focusing on the vehicle.

Additionally, the complexity gets exponentially more difficult when software development teams have to account for genders, the native language of the driver, accents, and enunciation, as well as the rate, tempo, and timbre of each individual driver.

In this case study, EagleTC worked hand-in-hand with a major automaker’s software development team to conduct robust testing. As a result, the team seamlessly and efficiently captured over 1,800 different voice commands. Consequently, EagleTC was able to provide expert consult every step of the way to ensure cost-effective testing and solutions.

Solution

EagleTC was awarded this ongoing multiyear project to perform voice recognition rate testing for different languages, locations, vehicle lines, software releases, infotainment functionalities, and voice recognition solutions.

Tester Setup and Methodology

For any given vehicle line and software release, EagleTC provides 3 males and 3 females to perform US English voice recognition rate testing over a span of 3 days. Specifically, there are 100 commands tested, repeated 3 times, for a total of 300 iterations per tester. The commands cover a wide range of functionalities (i.e., phone calls, contacts, text messages, navigation-point-of-interest, climate and comfort, etc.).

On testing days, two technicians (one male and

Solution

EagleTC was awarded this ongoing multiyear project to perform voice recognition rate testing for different languages, locations, vehicle lines, software releases, infotainment functionalities, and voice recognition solutions.

Tester Setup and Methodology

For any given vehicle line and software release, EagleTC provides 3 males and 3 females to perform US English voice recognition rate testing over a span of 3 days. Specifically, there are 100 commands tested, repeated 3 times, for a total of 300 iterations per tester. The commands cover a wide range of functionalities (i.e., phone calls, contacts, text messages, navigation-point-of-interest, climate and comfort, etc.).

On testing days, two technicians (one male and one female) perform the commands. There are two roles for each team and each technician performs both roles. The first role involves driving at a speed of 65-70 mph and exercising the commands. Meanwhile, the second role focuses on recording results and feeding the prompts to the driver. Once every prompt has been completed three times, the technicians switch roles to ensure consistency.

Test Result Classification

As part of the validation process, results for every command can be either a pass, fail, no response from the server, or not applicable. Therefore, each result is clearly categorized to ensure clarity and consistency. Moreover, the entire testing session is video recorded for playback and accuracy confirmation. Additionally, the technicians take logs for issues found during testing and provide a detailed explanation of their observations during testing (system response, road conditions, and unusual circumstances, such as rainy days, noisy roads, etc.). Furthermore, following the final round, a summary of the test results is then provided to the customer at the completion of all the test rounds.

In addition to English, the same testing is also performed for Canadian French, Mexican Spanish, and Brazilian Portuguese, based on customer needs. As a result, this multilingual approach ensures thorough validation across a variety of regions and languages, further enhancing the reliability of the voice recognition system.

Sample Testing Results

Table (1) illustrates a sample portion of the testing results with arbitrary percentages, including repetition of the same success rate where applicable to reflect expected benchmarks.

	Success Rate %
	Calls	POI	Text
Male Avg	90%	90%	86%
Female Avg	85%	90%	80%
Overall Avg	87.5%	90%	83%
Benchmark Avg	90%	90%	90%

Table (1) – Sample test result for any given vehicle with arbitrary percentages

Table (2) describes the test result verdict meanings. All commands will be assigned one of the verdicts listed below. Each functionality tested is then assigned an average percentage based on the pass rate, per female and male tester (See Table (1) above). Table (3) summarizes the scope for the project.

Table (2) – Test result verdict description

Table (3) – Summary of the project

Results

Thanks to agility and thorough testing processes, EagleTC was able to ensure the client delivered the best possible product to its consumers. Moreover, since being awarded the ongoing project in 2020, EagleTC has excelled with every assignment. Currently, the project consists of approximately 50 test sessions per year, all in several regions with several languages based on the client’s needs. Furthermore, EagleTC continuously adapts its process to meet the evolving requirements of each region and language group.

Consequently, with this project, EagleTC continues to demonstrate its ability to:

• Notably, have teams on the ground, deployed and testing, with short-term notice in different regions and for different language speakers.
• Additionally, provide cost-effective solutions achieved by minimizing overhead cost. Proper training for testers and strong program management processes also minimizes overhead.
• Moreover, find and analyze voice recognition rate issues in real time, resulting in improvements for future vehicle model years, providing a better user experience.
• In parallel, provide prompt support to customers and communicate issues and concerns as they arise.
• Likewise, adapt to unpredictable circumstances due to a pandemic. EagleTC has continued to work and deliver accurate and timely results while adhering to CDC guidelines.

Ultimately, EagleTC will always place a high priority on delivering excellence with quality and agility. Furthermore, we aim to provide clients with peace of mind by conducting any assignment without delays and meeting deadlines without exception. Without a doubt, this case study is just one example of that commitment.

Challenge

Modern automotive infotainment systems bring expanded functions and enjoyment to the driving experience, making compatibility testing of infotainment system features essential for ensuring seamless operation. With each new automotive platform and model year, as well as for major infotainment system updates, automakers and their key suppliers must ensure in-market compatibility of these systems with a wide range of consumer electronic devices and service providers including, but not limited to, cellular phones and cellular carriers in markets worldwide.

This compatibility testing of infotainment systems is conducted for the start of production launch (i.e. fourth months prior to the actual vehicle production start date) and continues on an ongoing basis for Bluetooth, WiFi, and phone projection (Android Auto; Apple CarPlay) with cellular phones. The complexity of this assignment recognizes that for any given platform and infotainment system, multiple iterations will simultaneously undergo testing, with test sample size reflecting the vehicle volume for a given region.

As it is neither possible nor desirable to test every cellular phone available, a key element of this compatibility testing is to know in any given market the popular cellular phone models that drivers are most likely to use and the prominent cellular carriers.

In this case study, a major automaker and several of their tier-one suppliers need to confirm that the infotainment systems would function properly in two key regions—Australia and the Middle East.

Solution

With its global reach and proven expertise in compatibility testing, EagleTC was assigned this multi-year project, which includes the start of production testing for each model year’s new infotainment system, as well as ongoing maintenance testing for existing systems. The start of production testing and maintenance testing is and will continue on a quarterly basis until the projected end date of 2028.

On the ground in Melbourne, Australia and Amman, Jordan, EagleTC’s engineering team quickly established the processes and acquired phones and tools to start testing. Phones and cellular carriers are selected based on popularity and regional demographics. Testing is conducted on different benches and hardware based on the model year and vehicle line. Initially, for the extensive start of the production phase of the assignment, the Middle East team tested 15 different phones for compatibility; and the Australia team tested 25 different phones. Subsequently, 15 phones in each region were assigned to maintenance testing. Maintenance testing validates system updates made during the course of a year.

A sample distribution of testing phones throughout the year for multiple model vehicle lines
are demonstrated in table (1). The number of test cases per phone used for the start of production (SOP) testing is 250, while maintenance testing uses 150. Test cases were distributed across several Bluetooth profiles (i.e., Hands-free Profile (HFP), Phone book Access Profile (PBAP), Advanced Audio Distribution Profile (A2DP), and Audio/Video Remote Control Profile (AVRCP)), phone projection, and WIFI features.

Table (1) – Sample Schedule in Australia and the Middle East in a given year (i.e., 2020). Two cellular carriers are used in each region.

As the testing progresses, EagleTC continues to discover several compatibility issues, which are promptly reported to the automaker’s tier-one suppliers. If a vehicle infotainment system is responsible for the issue, the tier one supplier would make improvements that are tested in the subsequent quarter during the maintenance testing. If the incompatibility could be isolated to a specific cellular phone model, no changes would be needed to the vehicle’s infotainment system.

Thus, an important part of compatibility testing is root cause analysis, to determine whether a compatibility issue derives from the automaker’s infotainment system or a given cellular phone.

Results

EagleTC’s thorough compatibility testing of infotainment system with cellular phones in these regions over the span of this project permitted the automaker and its tier-one suppliers to be fully confident that their customers in Australia and the Middle East would have a positive experience with their vehicle’s infotainment systems.

In doing so, EagleTC is continuing to demonstrate its ability to:

• Have teams on the ground, deployed and testing, with short-term notice.
• Have an understanding of the popular cellular phones in specific regions and the cellular carriers in which drivers subscribe.
• Provide cost-effective solutions with a full understanding of regional infrastructure and telecommunications protocols and regulations.
• Find software issues during testing resulting in infotainment software improvements for the current and future vehicle model years.
• Adapt to unpredictable circumstances due to a Pandemic. EagleTC’s team continued to work remotely to deliver accurate and timely results.

In summary, EagleTC thrives for excellence with quality, and agility. In this case study, we continue to provide our clients with the peace of mind that any assignment will be conducted without delays and that deadlines are met without exception.