View page as slide show

Communications Software Protocols and Architecture

Verification, Validation and Testing

  • Valid functional specification
  • Bug free software
  • Correct behaviour
    • Behaves as intended
    • Meets the requirements described in use cases
    • Follows the specifications and standards.
  • Acceptable performance
    • Memory leak avoidance
    • Usability

Verification and Validation

  • Verification is a quality control process that is used to make sure that the protocol will work according to specifications
    • in normal and
    • error conditions e.g. lost package
  • Validation is process that is used to assure that the specification has no design flaws
  • If a standard is ratified by a standards body that has not been validated and contains errors the following may occur:
    • Some implementations of the protocol may exist with serious errors that will affect service operation and service assurance
    • Different implementations of the protocol , that are bug free will have proprietary solutions to overcome protocol errors and thus may not inter-work
    • Some vendors may choose to implement only a subset of standardised protocols, relying on proprietary protocols.

Possible errors

  • livelock (infinite loop)
  • deadlock
  • output with no receiver
  • output with multiple receivers
  • over maximum number of instances for a process
  • unreachable states
  • Errors related to data access, for example:
    • variable usage not compliant with its type
    • array overflow

Automated validation

  • Even with simple protocols containing a small number of functional entities, messages and states the number of possibilities will be more than most people will have time to verify by hand.
  • Communications protocol implementations are so complex, that it is likely to be impossible to test all protocol combinations on a system that has implemented a protocol.
  • In reality standards committees are design by committee,
    • some members perform automatic validation,
    • most don’t → protocols are modified to cope with bugs as they appear
  • Even with simple protocols containing a small number of functional entities, messages and states the number of possibilities will be more than most people will have time to verify by hand.
  • Some problems have more permutations than a computer can go through, due to limitations on processing speed and memory.
  • Automated tools will not be able to detect undesirable behaviour due to poor specification

Traditional design

  • Traditional design involves the following design cycle
    • High Level Design
      • Requirements are written usually in English (or equivalent)
      • Protocol is then defined using a formal techniques such as SDL, MSC and ASN.1
    • Low Level Design
    • Coding and testing
      • Most the effort occurs here. Coding and debugging thus becomes the focus of the protocol design

Formal Design

  • The later an error is detected the more expensive it is to fix.
  • Formal design techniques (FDT) are used to verify the correctness of the protocol on the high level design phase.
  • This involves testing the protocol description as defined in a specification languages such as SDL.
  • This requires
    • An unambiguous notation
    • Effective validation tools

Formal Protocol Specification Languages and Protocol Validation

  • Formal protocol definition languages such as SDL can be used as input to protocol validation tools.
  • Three common description languages used for this approach are SDL, Lotos and Estelle.
  • FDTs are also intended to satisfy objectives such as:
    • a basis for analyzing specifications for correctness, efficiency, etc.;
    • a basis for determining completeness of specifications;
    • a basis for verification of specifications against the requirement of the Recommendation;
    • a basis for determining conformance of implementations to Recommendations;
    • a basis for determining consistency of specifications between Recommendations;
    • a basis for implementation support.

Other Formal Specification Languages

  • CCS (the Calculus of Communicating Systems)
  • Lotos (Language for Temporal Ordering Systems): mathematically defined language
  • Z Specification Language: a language based upon set theory and first-order predicate calculus.
  • VDM (Vienna Development Method)
  • These other languages are discussed in the literature, but a are not used by ITU-T

ITU Programming Languages

  • ITU standardises the following formal description techniques
    • Z.100 CCITT Specification and description language (SDL)
    • Z.105 SDL combined with ASN.1 (SDL/ASN.1)
    • Z.120 Message sequence chart (MSC)
  • ITU also define the following telecommunications languages
    • Z.200 CCITT high level language (CHILL) Common text with ISO/IEC
    • Z.30x Introduction to the CCITT man-machine language

Testing in communication software

  • Traditional tests
    • Unit testing
    • Integration testing
    • Acceptance testing
      • Conformance testing
      • Load testing
      • In-field testing
  • Cleanroom engineering (not in scope of course)
    • Formal verification
    • Statistical usage testing

Unit testing

  • testing that individual unit of source code works properly.
    • unit = smallest testable part
      • function, method, procedure, class
  • test cases should be independent from each other
  • Unit testing for communication software is similar than to any other software
    • test cases for individual FSM state transitions and complex FSm transactions
  • testing frameworks like in traditional Software engineering
    • JUnit,CPPUinit …

Integration testing

  • Testing that the independently working units are working together correctly
  • Units that are not available can be replaced with imitators (simulators)
    • drivers: generate input messages for the real units under test
    • stubs: passive imitators that accept outputs from units under test. May send also replies to outputs.
  • Communication software fits well for integration testing
    • Well defined hierarchy: Layered architecture
    • Message based communications
    • Well defined interfaces
    • → Creating simulation is easy

Conformance testing

  • Testing that the implementation conforms to the original requirements
    • Conforms to the standards and specifications
  • Important to guarantee that different implementations will work together.
    • Implementers may interpret standards differently and take own short cuts
    • Testing will point out the flaws
  • Interest in the external behaviour of the implementation
    • Check functional correctness of implementation under test without checking it inner works
    • Black box testing
  • Behaviour is specified with set of scenarios
    • Testing scenarios described in TTCN (Tree and Tabular Combined Notation)
    • TTCN testing suite provided by the protocol designer /standardisation body / testing facility.

Load and performance testing

  • Goal is to test the performance of the implementation
  • Reliability under load
    • how overload protection mechanisms are working.
    • Requests coming from several independent sources
    • Requests from the sources interleaved
  • Sustainability of connection
    • How the implementation handles really slow and long lasting connections
  • load generators used to create simulated environment rather than using real independent sources
    • Programmable device that offers selection of predefined scenarious that can be run.
    • not just generate load, but also analyses the responses. (e.g. conformance in heavy load)

In-field testing

  • Testing of the system in work.
    • detect, locate and eliminate bugs that are found in real work
  • Log files of events happening in system
    • e.g. logs of messages coming that do not fit for current state.
    • Logs can be analysed remotely and bug fixes then created.

Tools for testing

  • Protocol Analysers
    • Monitors of the ‘live’ communication between equipment and interprets messages in human readable form.
    • Same H/W may support monitoring of different protocol types
    • It may provide some statistical analysis capability
    • It may have advanced search/filtering capabilities.
    • You may define event on which logging would start or stop.
  • Protocol Simulators
    • Allows Users to write a scripts (similar to TTCN) which will send a messages and react on received messages
    • It can be used for testing various protocols
    • It may be quite big effort to write required amount of scripts
    • Suppliers often provide a set of test scripts targeting particular protocol
  • Protocol Emulators
    • Emulates operation of equipment (not full implementation)
    • No script writing required
    • It provides some control of behavior and configuration
    • Not flexible as simulator
  • Tools can be hardware
    • One box can contain Protocol Analyser, Simulator and Emulator
  • or software for standard computer
    • wireshark
  • Specific testing software suites
    • TTCN tools (see conformance testing)
Last modified: 2013/07/01 14:42