Continuous software testing is incredibly important.

Not only to guarantee that we deliver a working product and catch bugs early, but also to become better software developers in the long run.

As a result of a technology strategy review, we doubled down on our testing efforts which included an in-depth analysis of how we test on a company-wide level.

Early into the analysis we realized that there was a crucial blindspot in our software testing practices that evolved over time but made it difficult to scale testing to the next level.

We identified the following practices and their implications

Developers were writing tests in their own way without any consistent structure
They were using various scripting languages and hardware debuggers
They ran many different pieces of software, none of which was shared with the rest of the company
Their custom testing setups would often only run on their machines, meaning no one could reproduce their results or benefit from their efforts
The hardware-dependent tests did not have regression runs and were not integrated into our continuous integration (CI).

All of these practices, however, pointed towards a more meta problem:

We were not fully benefiting from our testing efforts due to a lack of standardization.

Subsequently we asked the question:

“How might we standardize company-wide testing efforts?”

As an answer to this question, we analyzed the existing problems and mapped them systematically to identify four distinct areas of action:

Testing tools
Testing and test result analysis effort
Testing on target as part of a continuous integration process
Automatic test reporting

To progress in each of these areas of action, we decided we needed a company-wide testing framework that developers can use to create standardized tests.

We created a set of requirements that this framework had to be able to fulfill to be of use to us:

Fetching necessary software artifacts
Flashing the hardware with the appropriate software
Running all the tests
Making integration of third-party tools and framework easy
Collecting communication and debug traces in an organized manner and report test results

As we did not want to reinvent the wheel on this testing framework, we first started by doing an in-depth market analysis of all publicly and commercially-available testing frameworks.

However, from this analysis we did not find any that met all of our requirements. To make matters worse, each framework we evaluated had some additional issues:

Some were GUI-oriented and non-scriptable, meaning they couldn’t easily be integrated into a continuous integration and continuous delivery (CI/CD) pipeline
Some were built around testing formats that didn’t fit our testing needs
Some were dependent on a specific operating system
Most were built with the assumption that all tests would be run on only one machine, meaning we could not scale HIL testing infrastructure

This is when project ELITE (ESR Labs integrated Test Environment) was born.

#ELITE serves the purpose of providing developers with a standardized testing framework for writing hardware-in-the-loop (HIL) tests.

#Introducing ELITE

ELITE’s core functions:

Use a customizable DSL language for test development that is readable for non-technical people and make the test implementation also the test specification
Test writers to fetch necessary software artifacts
Flash their test hardware with the appropriate software
Run all hardware tests
Collect traces, annotate them
Call third-party tool interfaces
Integrate to any CI tool easily
Generate testing reports.

The framework is OS and hardware independent, capable to run even on a RaspberryPI.

Additionally, ELITE’s configuration allows users to choose exactly what hardware they require for their tests dependent on what the tester needs:

ranging from different development board
to different sets of machines and hardware testing tools

#Scaling testing operations

Our software tests are part of our automated continuous integration and delivery pipeline, which allows us to test across thousands of test cases within short periods of time, but we had a lack of automation in one specific part of the pipeline: test analysis.

When we talk about test analysis, we’re talking about the following activities:

Logging of all relevant test data
Determination why a test failed
Analysis of the software change and test execution history
Comparison of the identified issue with existing issue reports
Creation of issue tickets

Doing all of that manually was no option and could only be solved through automation. ELITE provides us two core features to help with test analysis automation: extensive automated logging and automatic test case reporting.

Automated Logging – ELITE brings in automated logging for every test case and tool that a tester uses, so that test execution results are automatically captured, and execution logs are always available in case something goes wrong. By using the captured logs, ELITE also automatically generates an ESR Labs-standardized test report to show the current state of the testing results.

Automatic Test Case Reporting – ELITE’s automatically-built test reports and testing logs are used by our CI infrastructure to do automatic detection of testing regressions and automated reporting of new software bugs.

#Further implications – FUSI and ELITE

ELITE can also be used to test the proper implementation of Functional Safety (FuSi) requirements that our software is bound to. A common validation method is to use ELITE to intentionally cause a failure in the test target, then monitor the test target to ensure that the appropriate software safety feature takes action to correct this failure.

#Future

Right now ELITE is still in development and is still being refined for easy test-case development and consistent development patterns. The goal is to mature ELITE to the point where it can be rolled out and used broadly across ESR Labs to the benefit of all ESR Labs developers who do embedded software development.