Continuous HIL Testing
To achieve faster software development, we integrated the hardware-in-the-loop infrastructure into our continuous integration and delivery pipeline. Now developers get feedback on their work overnight.
The more time that passes between a software change and a failure, the harder it is to find the problem. The industry standard is a segregation of development and testing, sometimes across companies. This results in more context switching and longer feedback paths.
The integration of the hardware-in-the-loop (HIL) infrastructure into the continuous integration and delivery (CI/CD) pipeline required a redesign of all HIL components, now consisting of a modular hardware design and a software infrastructure defined as code. The hardware infrastructure provides the devices necessary to intensively test the ECU, while the software infrastructure sets up the required test environment and connects it to the test hardware. Together they enable a fully-automated and quickly scalable HIL testing process.
The hardware infrastructure is designed for scalability through modular design, and consists of test racks that contain up to six test benches. Installed on the test bench are the different hardware devices required to simulate vehicle functions and run different tests on the target ECU. While all test benches are built with a standard set of hardware, they are designed to be altered quickly to meet project-specific requirements.
The software infrastructure is designed for scalability through automation. Following the principle of infrastructure as code, the automation server (Jenkins) can set up both hardware and software components required for testing by simply running a script. To enable a full automation of the embedded software testing process, we needed to develop two additional components. (Grizzly and ELITE).
Jenkins is the test automation server that controls the flow of the entire testing process.
Grizzly is a resource manager that manages the access to and allocation of hardware resources, along with mapping virtual machines to those resources as needed by test jobs.
Proxmox is an open-source server virtualization environment that provisions the virtual machine following a configuration requested by Grizzly.
ELITE is the in-house test framework that runs on the VM and executes the tests, reporting results back to Jenkins.
Hardware in the loop
Both infrastructures are designed to allow fully automated hardware in the loop testing as part of a comprehensive continuous integration and delivery pipeline. Since the high degree of automation frees up tester resources, it is now possible to decrease the quantity and increase the quality of tickets in case of failed tests.
This is achieved through
Test Bench Configuration; The device-under-test is flashed with the appropriate software binary and connected to the provisioned VM.
VM Provisioning; The VM is created to suit the specific test suite and connected to the configured test bench.
Test Execution; All scheduled tests are executed.
Test Reporting; ELITE gathers test results and reports those results to Jenkins.
Automated Comparison; ELITE runs an automated comparison of the failing test results to existing tickets.
Issue deletion; If an issue is found to already have a ticket, no action is taken.
Issue analysis + review; If an issue appears to be novel, then a new review is started.
Ticket creation; For all new reviews, a new ticket is created to track the progress of issue resolution.
Every project team has a member who contributes to the continuous integration and delivery pipeline. The testing team itself covers a broad spectrum of disciplines: test engineering, development operations, and framework engineering. All testing team members are specialists in one of those areas but can collaborate across a multitude of disciplines inside and outside the organization.
The embedded software testing infrastructure is subject to continuous evolution. Getting to the next stage is a challenge that requires smart and impassioned individuals. Help us get there