Testing Requirements and Indicators of Compliance

A condition of Product Certification is that a product has not only been designed to be conformant to a Product Standard, but also that conformance has been tested in practice. This applies regardless of whether or not a designated test suite (Indicator of Compliance) is identified in the Product Standard. In all cases the precise hardware/software environment in which conformance has been established must be identified in the Conformance Statement, and it must be sufficiently detailed to enable conformance to be re-demonstrated and test results reproduced.

Many Product Standards identify, in the Indicator of Compliance section, that a specific test suite must be used during conformance testing. In these cases a test report which shows that there were no unresolved issues or failures (except for those covered by appropriate Interpretations, Test Suite Deficiencies, or Temporary Waivers) must be submitted along with, or prior to, the Product Certification application.

Designated test suites are given in generic terms (for example, a "VSX4 test report"), rather than by identifying a specific version of each suite and the detailed testing requirements, to avoid the need to re-issue a Product Standard every time the situation changes. In practice, specific versions of test suites are mandated at any given point in time. Conformance Administration can advise, although details of the current test suites and other criteria are available at www.opengroup.org/testing.

New versions of test suites replace current versions with a six-month overlap period, during which time either will be accepted as the valid Indicator of Compliance.

New test suites will normally become mandatory for Product Certification after a nine-month notice period, and thereafter a test suite report will be required at the first annual renewal of all relevant Product Certifications.

Conformance test suites typically test for the presence or absence of functionality, and the behavior of functionality when present. The Conformance Statement is the means by which an applicant declares which optional functionality is supported, and therefore a test report for a product should match the functionality claimed for it in the Conformance Statement. For example, if the Conformance Statement states that the implementation provides POSIX-2 C-language binding, then the conformance tests should have verified that the implementation provides these facilities according to the Technical Standards.

In performing test report audits on Product Certifications, The Open Group carries out extensive checks to determine the consistency of what is claimed for a product, comparing the information in the Conformance Statement against the test suite reports.

Some test suites are made available under license from The Open Group. In other cases-for instance, in the programming language Product Standards where the specifications conform to formal standards-the Product Standard references the test suites that have been developed to support the existing formal certification programs for these languages. Full details of the test suites and related matters can be found at www.opengroup.org/testing.

In support of a Product Certification application, The Open Group requires, among other things, a formal test report from the supplier in respect of Product Standards for which an Indicator of Compliance is specified.