Test Strategy

Concept

The Test Strategy intends to help development team increase the reliability confidence of their software project. It is more and more difficult carry out exhaustive testing activities, as the size and complexity of the code grow. With the Test Strategy you can:

  1. Reduce the scope of the code which needs to be tested

  2. Define code coverage expectations for those component which need to be tested

As a result, Squore provides the Code Coverage Compliance KPI which represents the ratio of components which comply with this Test Strategy.

Settings

To Be Tested

Squore applies an algorithm to include/exclude modules which should be integrated in the Test Strategy. Parameters are:

  • Cyclomatic Complexity (VG)

  • Nesting Level (LEVL)

  • Number of non-cyclic paths (NPAT)

  • Vocabulary Frequency (VOCF)

  • Code Stability Index (SI)

Using these parameters, the most complex and unstable code are identified. Increasing the test on this code will increase reliability and reduce the risk of delivery. Consequences:

  • Modules which are not part of the "to be tested" list will be "ignored" in the Code Coverage Compliance KPI

  • Modules which shall be tested will be evaluated according their coverage (Statement, Branch and MCDC) with regard to their safety level (ASIL, SIL …)

Coverage Thresholds

The code coverage thresholds can be tuned according the type of coverage and the safety level. Here are the default settings:

SWAN test strategy cov settings
Figure 1. Coverage thresholds

How to Address the Test Strategy with Squore

How to Set Up the Parameters

Parameters are available at project creation time. The "To be tested" list parameters are available in the "Test Strategy" section.

SWAN test strategy settings
Figure 2. Test Strategy - settings

Notes:

  • Each of the ‘TO BE TESTED’ parameters can be disabled (by setting the value ‘-1’)

  • It is possible to disable the impact of test strategy on the Code Coverage Compliance KPI. "Code Coverage thresholds" are available in the "Test Coverage Thresholds" section.

SWAN test strategy coverage form
Figure 3. Test Strategy - Coverage Thresholds

Note: it is possible to disable a dedicated type of coverage (Statement and/or Branch and/or MCDC). For instance if the team just wants to assess the statement coverage only, Branch and MCDC can be disabled.

How to Set Up the Safety Level

Squore allows defining the safety level, i.e. critical factor, for every component in the project.

Using the GUI

  • Select an artefact

  • Open the Form tab

  • Define the critical factor

The critical factor will spread to all "children artefacts" (i.e. all modules will automatically inherit the value of the file). In addition, it is possible to overload an inherited value.

SWAN test strategy asil def
Figure 4. Test Strategy - Safety Level definition

Using Text/Csv File Import

It is possible to provide a csv file during the project creation which contains the Critical Factor information. Create a csv file as follows (1=level A, 2=level B, 3=Level C, 4=level D):

SWAN test strategy asil file
Figure 5. Test Strategy - sample of "safety level" csv file

Feed this file to the "csv tag import" Data Provider:

SWAN test strategy asil dp
Figure 6. Test Strategy - Safety Level Data Provider

The Code Coverage Compliance Treemap

The treemap highlights components according to:

  • Their code size (=size of the treemap zone)

  • Their code coverage compliance (=color of the treemap zone)

SWAN test strategy treemap
Figure 7. Test Strategy - Coverage Compliance Treemap

Interpretation:

  • Grey zone means the module is excluded from the Test Strategy

  • The color code indicates how well the component is tested with regard to expectations defined in the "Test Coverage Thresholds" form

SWAN test strategy cov scale
Figure 8. Test Strategy - Coverage Compliance scale

Note: Safety level is implicitly taken into account in this evaluation. Whatever the safety level is, Squore highlights the distance to the coverage objectives.