Model-Based Testing

Model-Based Testing

Model-based testing is an innovative approach to perform software testing or system testing.

Models are used to represent the desired behavior of a System Under Test (SUT), or to represent testing strategies and a test environment.

In a nutshell, key advantages of Model Based Testing are:

Focus on coverage:

• • • Meaningful metrics available
    • Ensures the coverage does not drift over time
    • Control of the effective coverage

All tests are computed:

• • • Test tools have ‘only’ to implement the API provided
    • Same API for all test tools
    • Same test scripts for all test tools
      • Test scripts are the customer’s property >> full control of the effective coverage (not subject to test tool developers point of view)
        
    • All documents are fully sync’ed
      • Test scripts, readable tests…
    • Delta between versions is computed >> easy to spot updates between any 2 versions of the test plan (convenient for labs and test tools)
    • Very efficient during test updates, whatever the amount (test tools simply ‘import’ them)

Why Model-based testing: differences with legacy testing

Basically, the legacy way of designing test plan is illustrated below:

Customer writes the specifications, enumerates the requirements to cover and writes tests in natural language to cover these requirements. Once the test plan document is completed, it’s handed to a test tool vendor who takes the document as an input, interprets them and develops the tool for execution. What sequences are actually performed by the tool is unknown, at least very hard to catch.

With model-based testing, the process becomes more like this:

The customer is still in control of the readable test plan but also of the test scripts themselves and even the development API. There are clear metrics reported ensuring that coverage is met.

So the customer can focus on its expertise: its own specs and how to test them and the test tool vendor can focus on its expertise as well: test tool development.

Model-based testing process

Define the test strategy

The beginning of the story is understanding customer’s expectations in terms of testing. We first have to ensure that whatever the test plan, tests will actually be executable.

So Mellonne works with the customer to define a test strategy allowing automated execution. This can be performed in sync with the test tool vendor(s) in case it is not Mellonne.
Parties agree on a test architecture, specifying the interface, data in, data out and how to stimulate the System Under Test and get feedback

Define the test coverage required

Then the test plan work can start.

Mellonne takes the customer’s specifications as input document, extracts all the requirements and populates a spreadsheet, providing requirement name, documentation references, sub-cases to cover and more.

This output document will actually be the corner stone of the test plan as it will always be the reference against which the test coverage is computed.

Customer and Mellonne discuss & check that the coverage is matching expectations. Spreadsheet may be updated/amended any time to increase/decrease the test coverage required.

Model development

Coverage has been agreed, Mellonne now develops a model of the System Under Test. This ‘model’ is a formal high-level implementation of the specifications.

It describes all the commands, answers, behaviors and Mellonne adds inside some traceability tags, to match against the coverage spreadsheet.

Writing the tests

Tests may be either written with care or automatically generated. Either, the traceability tags ensure that the coverage is met.

The model is always present to enforce the specifications and does not allow unrelevant testing.

The model is also the asset that is actually dynamically computing each commands expectations. This is very different from a Word test plan where the expectations are mentally calculated. In this situation, the model contains a perfect representation of all internal data and mechanisms such that every bit output by the System Under Test can be checked systematically at each step of each test.

Test plan production

The last step of the test plan work is the production of the deliverables.

Mellonne relies on its publishing software that is taking model + tests as inputs and outputs development specification + tests for execution.

These assets are all generated from the same source so that all is coherent.

Test tool implementation

With the deliverables in hand, test tool(s) can start their development. Since the dev spec is provided, different test tool vendor actually implement the same methods and since they are provided with the test sequences, the customer can actually expect the same behaviors from different test tools. This is Mellonne’s experience in years of model-based testing.

Model-Based Testing Benefits