Round Robin Test Plan for 2012 Televisions Awards
Round robin testing, also known as inter-laboratory testing, is a process whereby individual product samples are tested at two or more different laboratories, and results are compared to ensure consistency in test results, address potential issues prior to the Television Awards verification testing, and identify any improvements needed in testing capabilities.
Round Robin Testing Scheme
Test Laboratory Selection
Procurement of Test Samples
Transportation of Samples
I. Objectives of Round Robin Testing
Just as processes that produce a product may vary, the process of performing a test and obtaining measurements is subject to variation. RRT between laboratories is a way of identifying many of the issues that cause variation in measurement. RRT is concerned primarily with identifying issues with:
Calibration and traceability of equipment;
Improving measurement standards;
Repeatability and reproducibility of the test method; and
Test laboratory capability.
II. Round Robin Testing Scheme
SEAD will utilize a sequential testing scheme for television round robin testing. Each group of test samples will be transported from one laboratory to the next. Tests will be performed with samples in “out of box” condition to minimize opportunity for variation in test results. The first laboratory to test the product samples will record the “out of box” settings and screen luminance levels and report the information to subsequent laboratories. Each following laboratory will check the product settings and luminance levels before measurement and report any discrepancies. The international expert will advise for resolution of any discrepancies.
III. Guidelines for Round Robin Testing
An impartial technical expert (also referred to as the “expert witness”) will observe and oversee testing at each test laboratory. Testing will be conducted with suitable, properly calibrated instrumentation and equipment at each laboratory. All test samples will be tested as a group, starting with a sample validation process.
Test laboratories will perform strict sample performance homogeneity testing with identical test methods and procedures, and will conduct additional examination and verification of testing conditions (installation, testing environment, etc.). Six reproducibility tests are required at each test laboratory to facilitate detection of variations not due to laboratory conditions or techniques (e.g. sample stability), and to provide better information with which to assess laboratory capability. Product samples procured for round robin testing will require a power supply of 230 VAC / 50 Hz. The North American test laboratory will utilize a 230 VAC / 50 Hz power supply to ensure homogeneity in testing conditions across all participating regions.
An instruction manual will be distributed to participating laboratories prior to the RRT commencement. The manual, which will be developed by the technical expert, will detail specific instructions on the following aspects of the round robin testing:
Test laboratory and expert witness responsibilities;
Product sample receipt and validation observation and measurement;
Test procedure requirements;
Reporting uncertainty and uncertainty calculation details;
Reporting test results and observations;
Criteria for assessment of results, including a statistical analysis;
Requirements for packaging and shipment of product samples.
A. Test laboratory selection
A total of four laboratories (one in each region participating in the 2012 SEAD Television Awards – Australia, Europe, India, and North America) will be selected for round robin testing and potentially for verification testing for candidate award-winning television models.
Prospective test laboratories with ISO/IEC 17025 accreditation with a scope that covers the IEC 62087 Ed. 3.0 test method will be preferred. Test laboratories will be required to complete questionnaires about their technical qualifications and expertise, and to provide examples of prior test reports. Test laboratories will be required to specify the number of tests performed according to test procedures outlined by IEC 62087. A Request for Proposal (RFP) will be issued to laboratories that meet the screening criteria. Final laboratory selections may require in-person evaluations by the technical expert.
Selection of a laboratory for round robin testing does not guarantee that the test laboratory will also be contracted for verification testing. Laboratory selection for verification testing will be based on results of the RRT process, which will be summarized in a SEAD-internal report.
B. Procurement of test samples
Approximately six sample products will be tested. Test samples shall be procured from different retailers, and, if more than one sample of a single model is used, from different manufacturing regions and batches. All sample products will be capable of handling high definition digital broadcast signals and will meet other pertinent eligibility criteria for the SEAD TV awards.
Each test laboratory must complete a test report to ensure consistency in testing conditions, calibration, test procedures, etc. Results will be reviewed by the expert witness to detect any anomalies or errors prior to shipment of the test samples to the next test laboratory. The reports are to be submitted electronically to CLASP for consolidation and analysis. The expert witness will investigate possible causes of any inconsistencies in test results. Once round robin testing is completed, the expert witness will produce a final RRT report to summarize test results from each test laboratory. The report will include a robust statistical analysis of the test results along with a summary of product sample model details and settings, test equipment details including calibration dates, and environmental conditions for each test laboratory.
D. Transportation of test samples
Prior to testing, each laboratory must measure and record the “out of box” settings and luminance measurements. In addition, the settings and luminance measurements of any other modes used for testing will be recorded and reported. After testing, each test laboratory will return samples to original factory settings. The expert witness will review each test laboratory’s results to ensure that any necessary re-testing is conducted prior to packaging and shipment of test samples to the next laboratory.
To avoid damage to the test samples, each test laboratory will package the sample products using wooden crates that will store two or more televisions each. The expert witness will specify the shipping crate design and packing procedures for transporting the test samples. Custom packaging will add extra protection for the televisions and will keep the samples together to reduce the possibility of losing any test samples during transportation. The first test laboratory will provide the shipping crates and packaging materials specified by the expert witness.
Test laboratories will adopt necessary measures to ensure test sample packaging is sound enough to withstand typical international shipping hazards. Test samples will be shipped via air freight to expedite the RRT process and ensure reliability in delivery. Upon receipt of the test samples, test laboratories are to record any damage that may have occurred during shipment.
The laboratory selection process will occur in November. Round robin testing will commence in early December, once contracts are in place and sample products are procured. Although actual testing of sample products is expected to take no longer than one week per laboratory, four (4) weeks of testing is allotted per laboratory to allow for transportation, customs, and results reporting. Approximately 4 months has been scheduled for RRT with an anticipated completion date of 30 March 2012.
Sample products will be procured and custom shipment packaging will be procured by the technical expert. Contracted test laboratories will be responsible for shipping and customs costs to transport the product samples to the next test laboratory.
Summary of Timeline and Deliverables
|Procure product samples in Australia
||28 Nov. 2011
||9 Dec. 2011|
Round Robin Testing
|9 Dec. 2011
||30 Mar. 2012|
- North America
1 Dec. 2011
16 Jan. 2012
13 Feb. 2012
20 Mar. 2012
13 Dec. 2011
24 Jan. 2012
24 Feb. 2012
30 Mar. 2012