View page as slide show

Communications Software, Protocols and Architecture


Based on: Communication Protocol Engineering by Venkataram and Manvi, Prentice Hall of India, 2004

Conformance testing

  • Testing that the protocol implementations external behavior is as specified.
    • to verify the interoperability of implementation
  • Challenges
    • efficient way to create conformance test suite
    • applying test suite to running implementation
  • Test results must be repeatable and comparable

General conformance testing

  • Parts in conformance testing
    • Reference specification - Formal specification (FSMs …) where the test is derived from
    • Implementation Under Test (IUT)
    • Tester - application running test sequences to verify compatibility of reference specification and IUT
  • Black box approach with finite set of inputs and outputs
    • Input sequences are given to IUT
    • Only outputs of IUT are observed.
  • IUT passes the test only if outputs match to those prescribed by specification

Conformance testing methodology and framework

  • CTMF (Conformance testing methodology and framework) is defined in ISO/IETC standard 9646
    • describes testing procedure
  • Seven parts
  1. Protocol concepts
  2. test suite specification and test system architecture
  3. test notation
  4. test realization
  5. ,6,7 means of testing and organizational aspects

Conceptual architecture

  • IUT is part of system under test (SUT)
  • PCO, Point of control and observation
    • Standardized interface where inputs are given to system and outputs are observed
    • Modeled with two FIFO queues, one for each direction (asynchronous communication)
  • Underlying service
    • Transmits the PDUs to Lower tester that may be in remote site
  • LT, Lower tester,
    • Peer entity of IUT
    • Conducts PDU exchange with IUT
    • PDUs are encoded in the primitives of underlying service
  • UT, Upper tester
    • User of IUT.
    • Primitive exchange with IUT
  • TCP, Test Coordination Procedures
    • Used when LT and UT are run in separate processes.

Test cases

  • Abstract test suite (ATS)
    • Used for test generation or derivation
    • Consists of Abstract test cases
    • Tests are developed independently from implementation (abstract)
  • Test case always has a purpose why it is done
  • Test purpose
    • Written description of a requirement or a set of related requirements that is tested.
    • Based on requirements defined in specification of IUT
  • Executable test case
    • is compiled from abstract one
    • is adapted to the Means of Testing (MoT)
  • MoT is combination of equipment and procedures that are used to perform the test cases

Conformance test architectures

  • Local architecture
    • Everything is conducted in single device
  • Distributed
    • Lower tester is at different system e.g. test laboratory
  • Remote method

ITU Conformance Testing Standards

  • X.290 - OSI conformance testing methodology and framework for protocol recommendations
  • X.291 Abstract test suite specification
  • X.292 The Tree and Tabular Combined Notation (TTCN)
  • X.293 Test realization
  • X.294 Requirements on test laboratories and clients for the conformance assessment process
  • X.295 Protocol profile test specification
  • X.296 Implementation conformance statements

Conformance Testing

  • Verification that protocols are conforming to Standard Requirements
  • PICS - Protocol Implementation Conformance Statement
    • Information provided by protocol implementator on the extent of the implementation: what optional items are implemented, is there any restrictions …
  • PIXIT - Protocol implementation extra information
    • Provides information regarding the physical configuration of unit under test: eg. Telephone numbers, socket numbers …

Tree and Tabular Combined Notation TTCN

  • Defined as part of ISO/IEC 9646 (X.290)
  • Used as a language for formal definition of test cases
  • Each test case is an event tree in which external behavior such as: “If we send the message 'connect request,' either 'connect confirm' or 'disconnect indication' will be received” are described.
  • Messages can be defined using either the TTCN-type notation or ASN.1.

TTCN Tests Reactive Systems

A Typical Test Architecture

Parallel Test Architecture

TTCN Formats

  • TTCN is provided in two forms:
  • a graphical form (TTCN.GR) suitable for human readability
  • a machine processable form (TTCN.MP) suitable for transmission of TTCN descriptions between machines and possibly suitable for other automated processing.

Main Components of TTCN

  • Test Suite Structure
  • Data declarations (mainly PDUs)
    • TTCN data types
    • ASN.1 data types
  • Constraints
    • i.e., actual instances of PDUs
  • Dynamic behaviour trees (Test Cases)
    • Send, Receive, Timers, Expressions, Qualifiers
    • Test Steps

Test Suite Structure

  • TTCN Test suite consist of 4 major parts
    • suite overview part
    • declarations part
    • constraints part
    • dynamic part

Suite Overview Part

  • The suite overview part is a documentary feature comprised of indexes and page references.
  • This is part of the heritage of TTCN as a paper-oriented language.
  • It contains a table of contents and a description of the test suite, and its purpose is mainly to document the test suite to increase clarity and readability.
  • In this part of the test suite, a quick overview of the entire test suite is possible.

Test Suite Overview Proforma

Suite Declarations Part

  • The declarations part is used for declaring
    • types,
    • variables,
    • timers,
    • points of control and observation (PCOs),
    • test components.
  • The types can be declared using either TTCN or ASN.1 type notation.
  • Graphical table is used for declarations

TTCN Data Types

  • There is no equivalent to pointer types and, as a consequence of that, types cannot be directly or indirectly recursive.
  • Two structured types that are specific
    • protocol data unit (PDU) – data packets sent between peer entities in the protocol stack
    • abstract service primitive (ASP) - - a data type in which to embed a PDU for sending and receiving

Example of ASN.1 Type Definition

Example of TTCN Type Definition

Variable Declarations

Constraints Part

  • In this part constraints are used to for describing the values sent or received.
  • The instances used for sending must be complete, but for receiving there is the possibility to define incomplete values using wild cards, ranges and lists.

Dynamic Part

  • In this part actual tests are defined
  • Contains Test Suite with
    • test groups – a grouping of test cases. It might, for example, be convenient to group all test cases concerning connection establishment and transport into a separate test group
    • test steps – a grouping of test events, similar to a subroutine or procedure in other programming languages
    • test events – the smallest, indivisible unit of a test suite. Typically, it corresponds to sending or receiving a message and the operations for manipulating timers

TTCN Behavior Tree

  • Suppose that the following sequence of events can occur during a test whose purpose is to establish a connection, exchange some data, and close the connection.
    • a) CONNECTrequest, CONNECTconfirm, DATArequest, DATAindication, DISCONNECTrequest.
  • Possible alternatives to “valid behaviour” are
    • b) CONNECTrequest, CONNECTconfirm, DATArequest, DISCONNECTindication.
    • c) CONNECTrequest, DISCONNECTindication.

TTCN Behavior Tree

TTCN Behavior Table

  • The behavior tables build up the tree by defining the events in lines on different indention levels. The rows on the same indention are events that describe alternatives, and row on the next indention are the continuation of the previous line.
  • The line can consist of one or a number of the following:
    • send statement – ! – states that a message is being sent
    • receive statement – ? – states that a message is being received
    • assignment – ( ) – there is an assignment of values
    • timer operations – start, stop or cancel a timer
    • time-out statement
    • Boolean operations – [] – qualifying the execution
    • attachment – + – acts as a procedure call

TTCN Behavior Table

  • All the leaves in the event tree are assigned a verdict that can be:
    • pass – the test case completed without the detection of any errors
    • fail – an error was detected
    • inconclusive – there was insufficient evidence for a conclusive verdict to be assigned, but that the behavior was valid
  • A verdict can be final or preliminary – it will not terminate the active test case execution.
  • To describe what is happening in the test case, the dynamic behavior can be explained in plain language.

Dynamic Behaviour Table

TTCN , what you need to know

  • Understand Terminology
  • Understand test principles and the test environment
  • You don’t need to know how to write TTCN scripts, but you have to understand them so you can read test reports and analyse problems
Last modified: 2013/07/01 14:42