Testing Requirements and Indicators of Compliance

A condition of Product Registration is that a product has not only been designed to be conformant to a Product Standard but also that conformance has been tested in practice. This applies regardless of whether or not a designated test suite (Indicator of Compliance) is identified in the Product Standard. In all cases the precise hardware/software environment in which conformance has been established must be identified in the Conformance Statement, and it must be sufficiently detailed to enable conformance to be re-demonstrated and test results reproduced.

Many Product Standards identify, in the Indicator of Compliance section, that a specific test suite must be used during conformance testing. In these cases a test report which shows that there were no unresolved issues or failures (except for those covered by appropriate Interpretations, Test Suite Deficiencies, or Temporary Waivers) must be submitted along with, or prior to, the Product Registration application.

Designated test suites are given in generic terms (for example, a "VSX4 test report"), rather than by identifying a specific version of each suite and the detailed testing requirements, to avoid the need to re-issue a Product Standard every time the situation changes. In practice, specific versions of test suites are mandated at any given point in time. The Conformance Administrator can advise, although details of the current test suites and other criteria are available on The Open Group web site at http://www.opengroup.org/testing.

New versions of test suites replace current versions with a 6-month overlap period, during which time either will be accepted as the valid Indicator of Compliance.

New test suites will normally become mandatory for Product Registration after a 9-month notice period, and thereafter a test suite report will be required at the first annual renewal of all relevant Product Registrations.

Conformance test suites typically test for the presence or absence of functionality, and the behavior of functionality when present. The Conformance Statement is the means by which an applicant declares which optional functionality is supported, and therefore a test report for a product should match the functionality claimed for it in the Conformance Statement. For example, if the Conformance Statement states that the implementation provides POSIX-2 C-language binding, then the conformance tests should have verified that the implementation provides these facilities according to the CAE Specifications.

In performing test report audits on Product Registrations, The Open Group carries out extensive checks to determine the consistency of what is claimed for a product, comparing the information in the Conformance Statement against the test suite reports.

Some test suites are made available under license from The Open Group. In other cases, for instance in the programming language Product Standards where the specifications conform to formal standards, the Product Standard references the test suites that have been developed to support the existing formal certification programs for these languages. Full details of the test suites and related matters can be found on The Open Group web site at http://www.opengroup.org/testing.

In support of a Product Registration application, The Open Group requires, among other things, a formal test report in respect of Product Standards for which an Indicator of Compliance is specified.

There are two alternative sources of test report that may be used for Product Registration:

1. The Open Group recognized laboratories, which have been through a formal quality assessment procedure

2. Other test facilities; that is, those which have not been through a formal assessment procedure

Recognized laboratories are required either to have been assessed directly by The Open Group for the conformance of their procedures to ISO guide 25, or to have been accredited by a national or regional accreditation body for the conformance of their procedures to ISO guide 25.

ISO guide 25 is concerned with the ability to repeat and reproduce formal test procedures, and therefore provides the Open Brand Program with a firm foundation of dependable test reports.

The alternative approach using other test facilities is dealt with by means of quality control rather than quality assurance. Registration applications through such test facilities are subject to a high percentage of technical audits (between 80% and 100% depending on the particular technology in question). This approach contrasts with the case of recognized laboratories in which a low percentage of test reports are subject to a technical audit (typically 5%). Through careful design of test suites and the use of detailed technical audit procedures to ISO 9000 it can be assured that the applications from other test facilities are to the same standard as those from recognized laboratories. However, this approach is less certain and often requires multiple re-submissions of the applications.