Test Strategy

Concept

The Test Strategy intends to help development team increase the reliability confidence of their software project. It is more and more difficult carry out exhaustive testing activities, as the size and complexity of the code grow. With the Test Strategy you can:

Reduce the scope of the code which needs to be tested
Define code coverage expectations for those component which need to be tested

As a result, Squore provides the Code Coverage Compliance KPI which represents the ratio of components which comply with this Test Strategy.

Settings

To Be Tested

Squore applies an algorithm to include/exclude modules which should be integrated in the Test Strategy. Parameters are:

Cyclomatic Complexity (VG)
Nesting Level (LEVL)
Number of non-cyclic paths (NPAT)
Vocabulary Frequency (VOCF)
Code Stability Index (SI)

Using these parameters, the most complex and unstable code are identified. Increasing the test on this code will increase reliability and reduce the risk of delivery. Consequences:

Modules which are not part of the "to be tested" list will be "ignored" in the Code Coverage Compliance KPI
Modules which shall be tested will be evaluated according their coverage (Statement, Branch and MCDC) with regard to their safety classification (ASIL, SIL …)

Coverage Thresholds

The code coverage thresholds can be tuned according the type of coverage and the safety classification. Here are the default settings:

Figure 1. Coverage thresholds

How to Address the Test Strategy with Squore

How to Set Up the Parameters

Parameters are available at project creation time. The "To be tested" list parameters are available in the "Test Strategy" section.

Figure 2. Test Strategy - settings

Notes:

Each of the ‘TO BE TESTED’ parameters can be disabled (by setting the value ‘-1’)
It is possible to disable the impact of test strategy on the Code Coverage Compliance KPI. "Code Coverage thresholds" are available in the "Test Coverage Thresholds" section.

Figure 3. Test Strategy - Coverage Thresholds

Note: it is possible to disable a dedicated type of coverage (Statement and/or Branch and/or MCDC). For instance if the team just wants to assess the statement coverage only, Branch and MCDC can be disabled.

How to Set Up the Safety Classification

Squore allows defining the safety classification for every component in the project.

Using the GUI

Select an artifact
Open the Form tab
Define the safety classification

The safety classification will spread to all "children artifacts" (i.e. all modules will automatically inherit the value of the file). In addition, it is possible to overload an inherited value.

Figure 4. Test Strategy - Safety Classification definition

Using Text/Csv File Import

It is possible to provide a csv file during the project creation which contains the saferty classification information. Create a csv file as follows (1=level A, 2=level B, 3=Level C, 4=level D):

Figure 5. Test Strategy - sample of "safety classification" csv file

Feed this file to the "csv tag import" Data Provider:

Figure 6. Test Strategy - Safety Classification Data Provider

The Code Coverage Compliance Treemap

The treemap highlights components according to:

Their code size (=size of the treemap zone)
Their code coverage compliance (=color of the treemap zone)

Figure 7. Test Strategy - Coverage Compliance Treemap

Interpretation:

Grey zone means the module is excluded from the Test Strategy
The color code indicates how well the component is tested with regard to expectations defined in the "Test Coverage Thresholds" form

Figure 8. Test Strategy - Coverage Compliance scale

Note: Safety classification is implicitly taken into account in this evaluation. Whatever the safety classification is, Squore highlights the distance to the coverage objectives.