The tracability

The events

The execution of a test is divided into events, all of these events are stored and can be viewed afterwards. An event can represent:

  • an action performed by the test framework
  • an action performed by the test
  • a data item received by the system to be tested or checked.
  • data to send to the system to test or control.

Event execution makes it possible to have robust tests thanks to the definition of observation intervals. The approach is to write the tests with the following formalism:

  • I perform an action in my test.
  • During a given interval, I look and compare all the events received with an expected.
  • I decide on the next action
    • after receiving the event I was waiting for
    • or when the observation interval is over.

During the execution of a test, the framework captures all the events generated by the tested or piloted system. The events are then converted and stored in a message called template.

A template is split into one or more layers. A layer is defined by a set of key/value. The value of a layer can be another layer too.

../_images/template_message.png

Creating a template

Creating a template can be done using the TestTemplates test framework.

tpl = TestTemplates.TemplateMessage ()
layer = TestTemplates.TemplateLayer (name = 'response')
layer.addKey (name = 'code', data = '200')
layer.addKey (name = 'msg', data = 'hello world')
tpl.addLayer (layer = layer)
This template indicates that the event should contain:
  • a layer called response and containing the key` code` and msg
  • the key code must be strictly equal to the value 500
  • the msg key must be strictly equal to the text hello world.

Example of an expected message visible from the graphical client:

../_images/template_expected.png

The operators

Operators are available to facilitate comparison of the models received.

Sample template using comparison operators:

tpl = TestTemplates.TemplateMessage()
layer = TestTemplates.TemplateLayer(name='response')
layer.addKey(name='code', data=TestOperators.LowerThan(x=500)))
layer.addKey(name='msg', data=TestOperators.Contains(x="hello"))
tpl.addLayer(layer=layer)
This template indicates that the event should contain:
  • a layer called response and containing the key` code` and msg
  • the code key must be less than 500
  • the msg key must contain the text hello.

Visualization

The client can graphically display the comparison made by the framework.

../_images/client_event_mismatch.png

Definition of the color code:

Green Perfect match between the value received and expected
Red The value received does not correspond to the expected value
Yellow The expected value has not been verified

Test reports

After each run of a test, the framework automatically generates the associated test reports.

There are 2 type reports:
  • An advanced report
  • A basic report (accessible by default from the graphical client)

The reports are accessible from the client, the web interface or from the API.

Note

Reports can be exported in html, csv, xml and pdf format.

Advanced report

The advanced report displays information such as:
  • the execution time of each test case
  • the complete description of the test steps.
  • performance statistics.
  • the test parameters.
../_images/advanced_report.png

It is possible to display variables in the test report by prefixing the variables:

  • SUT_ Variables describing the version of the system to be tested or piloted
  • DATA_ Variables describing specific data
  • USER_ User variables

This feature can be useful for increasing the level of traceability in reports.

../_images/inputs_sut.png ../_images/report_inputs.png

Basic report

The basic report summarizes the result of all test cases and reports.

../_images/basic_report.png

Color code:

Green The test case is valid
Red The test case is in error
Orange The result of the test case is not determined
Gray The test case was not executed

Tip

You must click on the test cases to display the steps.

Note

The messages displayed by the test with the Trace (self) .info () function are available in the report by clicking on the [logs details] link.

Errors are also displayed by clicking on the [errors details] link.

The logs

The framework allows you to save logs while running a test and make them available quickly to the uses. All additional logs are zipped and accessible from the client or the API.

../_images/private_storage.png

Note

For more details, read the chapter The fundamentals >> Data.