Continuous HIL Testing

To achieve faster software development, we integrated the hardware-in-the-loop infrastructure into our continuous integration and delivery pipeline. Now developers get feedback on their work overnight.

Background

The more time that passes between a software change and a failure, the harder it is to find the problem. The industry standard is a segregation of development and testing, sometimes across companies. This results in more context switching and longer feedback paths.

Solution

The integration of the hardware-in-the-loop (HIL) infrastructure into the continuous integration and delivery (CI/CD) pipeline required a redesign of all HIL components, now consisting of a modular hardware design and a software infrastructure defined as code. The hardware infrastructure provides the devices necessary to intensively test the ECU, while the software infrastructure sets up the required test environment and connects it to the test hardware. Together they enable a fully-automated and quickly scalable HIL testing process.

Hardware

Test Rack

1. Host Machine


2. Managed Switch


3. Power Supply


4. Test Benches


Infrastructure

The hardware infrastructure is designed for scalability through modular design, and consists of test racks that contain up to six test benches. Installed on the test bench are the different hardware devices required to simulate vehicle functions and run different tests on the target ECU. While all test benches are built with a standard set of hardware, they are designed to be altered quickly to meet project-specific requirements.

Test Bench

1. Patch Panel


2. Debug Board


3. Device Under Test


4. Relay Card


5. Debugger


Software

Infrastructure

The software infrastructure is designed for scalability through automation. Following the principle of infrastructure as code, the automation server (Jenkins) can set up both hardware and software components required for testing by simply running a script. To enable a full automation of the embedded software testing process, we needed to develop two additional components. (Grizzly and ELITE).

jenkins
Project grizzly
TEST BENCH
ELITE
VM
Proxmox

1

Jenkins is the test automation server that controls the flow of the entire testing process.

2

Grizzly is a resource manager that manages the access to and allocation of hardware resources, along with mapping virtual machines to those resources as needed by test jobs.

3

Proxmox is an open-source server virtualization environment that provisions the virtual machine following a configuration requested by Grizzly.

4

ELITE is the in-house test framework that runs on the VM and executes the tests, reporting results back to Jenkins.

Process

Hardware in the loop

Both infrastructures are designed to allow fully automated hardware in the loop testing as part of a comprehensive continuous integration and delivery pipeline. Since the high degree of automation frees up tester resources, it is now possible to decrease the quantity and increase the quality of tickets in case of failed tests.

This is achieved through

  • Automated issue analysis
  • Manual comparison
  • Direct feedback loops
  • Personal interaction between developers
    and software testers
Test Bench Configuration
VM Provisioning
Test Execution
Test reporting
Automated comparison
issue deletion
issue review + analysis
ticket creation

Test Bench Configuration; The device-under-test is flashed with the appropriate software binary and connected to the provisioned VM.

VM Provisioning; The VM is created to suit the specific test suite and connected to the configured test bench.

Test Execution; All scheduled tests are executed.

Test Reporting; ELITE gathers test results and reports those results to Jenkins.

Automated Comparison; ELITE runs an automated comparison of the failing test results to existing tickets.

Issue deletion; If an issue is found to already have a ticket, no action is taken.

Issue analysis + review; If an issue appears to be novel, then a new review is started.

Ticket creation; For all new reviews, a new ticket is created to track the progress of issue resolution.

Mindset

Teamwork

Every project team has a member who contributes to the continuous integration and delivery pipeline. The testing team itself covers a broad spectrum of disciplines: test engineering, development operations, and framework engineering. All testing team members are specialists in one of those areas but can collaborate across a multitude of disciplines inside and outside the organization.

Future

The embedded software testing infrastructure is subject to continuous evolution. Getting to the next stage is a challenge that requires smart and impassioned individuals. Help us get there