The tracability¶
The events¶
The execution of a test is divided into events, all of these events are stored and can be viewed afterwards. An event can represent:
- an action performed by the test framework
- an action performed by the test
- a data item received by the system to be tested or checked.
- data to send to the system to test or control.
Event execution makes it possible to have robust tests thanks to the definition of observation intervals. The approach is to write the tests with the following formalism:
- I perform an action in my test.
- During a given interval, I look and compare all the events received with an expected.
- I decide on the next action
- after receiving the event I was waiting for
- or when the observation interval is over.
During the execution of a test, the framework captures all the events generated by the tested or piloted system.
The events are then converted and stored in a message called template
.
A template
is split into one or more layers
.
A layer
is defined by a set of key/value. The value of a layer
can be another layer too.
Creating a template¶
Creating a template can be done using the TestTemplates
test framework.
tpl = TestTemplates.TemplateMessage ()
layer = TestTemplates.TemplateLayer (name = 'response')
layer.addKey (name = 'code', data = '200')
layer.addKey (name = 'msg', data = 'hello world')
tpl.addLayer (layer = layer)
- This template indicates that the event should contain:
- a layer called response and containing the key` code` and msg
- the key code must be strictly equal to the value 500
- the msg key must be strictly equal to the text hello world.
Example of an expected message visible from the graphical client:
The operators¶
Operators are available to facilitate comparison of the models received.
Sample template using comparison operators:
tpl = TestTemplates.TemplateMessage()
layer = TestTemplates.TemplateLayer(name='response')
layer.addKey(name='code', data=TestOperators.LowerThan(x=500)))
layer.addKey(name='msg', data=TestOperators.Contains(x="hello"))
tpl.addLayer(layer=layer)
- This template indicates that the event should contain:
- a layer called response and containing the key` code` and msg
- the code key must be less than 500
- the msg key must contain the text hello.
Visualization¶
The client can graphically display the comparison made by the framework.
Definition of the color code:
Green | Perfect match between the value received and expected |
Red | The value received does not correspond to the expected value |
Yellow | The expected value has not been verified |
Test reports¶
After each run of a test, the framework automatically generates the associated test reports.
- There are 2 type reports:
- An advanced report
- A basic report (accessible by default from the graphical client)
The reports are accessible from the client, the web interface or from the API.
Note
Reports can be exported in html, csv, xml and pdf format.
Advanced report¶
- The advanced report displays information such as:
- the execution time of each test case
- the complete description of the test steps.
- performance statistics.
- the test parameters.
It is possible to display variables in the test report by prefixing the variables:
SUT_
Variables describing the version of the system to be tested or pilotedDATA_
Variables describing specific dataUSER_
User variables
This feature can be useful for increasing the level of traceability in reports.
Basic report¶
The basic report summarizes the result of all test cases and reports.
Color code:
Green | The test case is valid |
Red | The test case is in error |
Orange | The result of the test case is not determined |
Gray | The test case was not executed |
Tip
You must click on the test cases to display the steps.
Note
The messages displayed by the test with the Trace (self) .info ()
function are available in the
report by clicking on the [logs details]
link.
Errors are also displayed by clicking on the [errors details]
link.
The logs¶
The framework allows you to save logs while running a test and make them available quickly to the uses. All additional logs are zipped and accessible from the client or the API.
Note
For more details, read the chapter The fundamentals >> Data.