Types of Proficiency Testing Scheme

 

TYPES OF PROFICIENCY TESTING SCHEME
A.1	GENERAL
	Proficiency testing has become an essential aspect of laboratory practice in all areas of testing, calibration, and inspection. Proficiency testing schemes vary according to the needs of the sector in which they are used, the nature of the proficiency test items, the methods in use and the number of participants. However, in their simplest form, most proficiency testing schemes possess the common feature of comparison of results obtained by one laboratory with those obtained by one or more different laboratories. The nature of the test or measurement performed in Proficiency Testing Schemes governs the method of comparing performance. There are three basic types of laboratory examinations: quantitative, qualitative, and interpretive.
	â The results of a quantitative measurement are numerical and are reported on an interval or a ratio scale. Test for quantitative measurement may vary in their precision, trueness, analytical sensitivity, and specificity. In quantitative proficiency testing schemes, numerical results are usually analysed statistically.
	â The results of qualitative tests are descriptive and reported on a categorical or ordinal scale, e.g. identity of microorganisms, or by identification of the presence of a specific measurement (such as a drug or a grading of a characteristic). Assessment of performance by statistical analysis may not be appropriate for qualitative examinations.
	â In interpretive tests, the “proficiency testing items” is a test result (e.g. descriptive morphology statement), a set of data (e.g. to determine a calibration line) or other set of information (e.g. a case study), concerning an interpretative feature of the participant’s competence.
	Other Proficiency Testing scheme have additional features depending on their objective, as outlines in definition 3.7, Note 1, (a) to (h). Some common applications of those types of Proficiency Testing are discussed below and illustrated in Figure A.1. These schemes may be “single occasion” and performed once, or “continuous” and performed at regular intervals.
A.2	SEQUENTIAL PARTICIPATION SCHEMES
	Sequential Participation Schemes (sometimes known as measurement comparison schemes) involve the Proficiency Test item being circulated successively from one participant to the next (i.e., sequential participation), or occasionally circulated back to the Proficiency Testing provider for rechecking. Model 1 in Figure A.1 provides a brief summary of this type of design, and the key features are typically those described below.
	a.     A reference laboratory that is capable of providing a metrologically traceable assigned value with sufficiently small measurement uncertainty and reliability for the Proficiency Test item is used. For categorical or ordinal properties, the assigned value should be determined by consensus of experts or other authoritative source. It may be necessary for the Proficiency Test item to be checked at specific stages during the conduct of the Proficiency Testing scheme, in order to ensure that there is no significant change in the assigned value.
	b.     The individual measurement results are compared with the assigned value established by the reference laboratory. The coordinator should take into account the claimed measurement uncertainty of each participant, or the claimed level of expertise. It may be difficult to compare results on a group basis as there may be relatively few participants having measurement capabilities that closely match each other.
	c.      Schemes involving sequential participation take time (in some cases, years) to complete. This causes a number of difficulties, such as
	â Ensuring the stability of the item;
	â The strict monitoring of the circulation among participants and the time allowed for measurement by individual participants, and
	â The need to supply feedback on individual performance during the scheme’s implementation, rather than waiting until it finishes.
	d.     Proficiency test items (measurement artifacts) used in this type of Proficiency test can include, for example, measurement reference standards (e.g. resistors, micrometers, and frequency counters) or, in medical programmes, histology slides with confirmed diagnosis.
	e.     Schemes that follow this design but that are limited to situations where a single participant is tested independently are often called “measurement audits.”
	f.      In some situations, the assigned value for a Proficiency Test item may be determined by consensus, after all participants (or in some situations, a subset of participants) have completed the measurement comparison
A.3	SIMULTANEOUS PARTICIPATION SCHEMES
	A.3.1	General
		Simultaneous participation Proficiency Testing Scheme usually involve randomly selected sub–samples from a source of material being distributed simultaneously to participants for concurrent testing. In some schemes, participants are required to take samples, which are then considered to be the Proficiency Test items for analysis. After completion of the testing, the results are returned to the Proficiency Testing Provider and compared with the assigned value(s) to give an indication of the performance of the individual participants and the group as a whole. Examples of Proficiency Test items used in this type of scheme include food, body fluids, agricultural products, water, soils, minerals and other environmental materials. In some cases, separate portions of previously established reference materials are circulated. Advice or educational comments are typically part of the report returned to participants by the Proficiency Testing Provider with the aim of promoting improvement in performance. Model 2 in Figure A.1 represents typical Proficiency Testing schemes of this type, usually for testing laboratories. Model 2 in Figure A.1 represents typical Proficiency Testing schemes of this type, usually for testing laboratories. Model 3 presents a type of scheme that is frequently used in conjunction with simultaneous Proficiency Testing Schemes, for oversight or educational purposes.
		As discussed in Annex B, assigned values for these Proficiency Testing schemes may be determined in a variety of ways. However, either evaluations of performance are based on consensus values form participants (all participants, or a subset of “experts”) or evaluations can be on the basis of independently determined assigned values.
		Known value schemes use assigned values that are determined independently of the participants and involve preparations of Proficiency test items with a number of known measurements or characteristics. Certified reference materials can also be used in these schemes, as their certified value and measurement uncertainty can used directly. A direct comparison can also be made between a proficiency test item and a certified reference material under repeatability conditions. However, care should be taken to ensure that the certified reference material is closely comparable with a Proficiency test item. Proficiency test items from previous proficiency testing rounds may be used in this type of scheme if the item has demonstrated stability. One special application of Proficiency Testing, often called “blind” Proficiency Testing is where the Proficiency test item is indistinguishable from normal customer items or samples received by the laboratory. This type of Proficiency Testing can be difficult since it requires coordination with a normal laboratory customer. In addition because, of unique packaging and shipping needs, bulk processing is usually not feasible and homogeneity testing is difficult.
	A.3.2	Split–level Designs
		A common design for Proficiency Testing is the “split–level” design, where similar (but not identical) levels of measurement are included in two separate Proficiency Test items. This design is used to estimate the participant’s precision at a specific level of a measurement. It avoids problems associated with replicate measurements on the same Proficiency Test item, or with the inclusion of two identical Proficiency test items in the same Proficiency Testing round.
	A.3.3	Split–sample Testing Schemes
		One special type of Proficiency Testing design that is often used by participants customers and some regulatory bodies is the “split–sample design”
		Typically, split–sample proficiency testing involves comparisons of the data produced by small groups of participants (often only two). In these Proficiency Testing schemes, samples of a product of a material are divided into two or more parts, with each participant testing one part of the sample (see Figure A.1, model 5). Uses for this type of scheme include identifying poor accuracy, describing consistent bias and verifying the effectiveness of corrective actions. This design may be used to evaluate one or both participants as suppliers of testing services, or in cases where there are too few participants for appropriate evaluation of results. Under such schemes, one of the participants may be considered to operate at a higher metrological level (i.e. lower measurement uncertainty), due to the use of reference methodology and more advance equipment, etc. or through confirmation of its own performance through satisfactory participation in a recognized interlaboratory comparison scheme. Its results are considered to be the assigned values in such comparisons and it may act as an advisory or mentor laboratory to the other participants comparing split–image data with it.
	A.3.4	Partial–process schemes
		Special types of Proficiency Testing involve the evaluation of participants abilities to perform parts of the overall testing or measurement process. For example, some existing Proficiency Testing Schemes evaluate participants abilities to transform and report a given set of data (rather than conduct the actual test or measurement), to make interpretations based on a given set of data or Proficiency Testing items such as stained blood films for diagnosis, or to take and prepare samples or specimens in accordance with a specification.
A.4	EXTERNAL QUALITY ASSESSMENT PROGRAMMES (EQA)
	EQA Programmes (such as those provided for laboratory medicine examinations) offer a variety of interlaboratory comparison schemes based on this traditional Proficiency Testing model, but with broader application to the schemes described in A.2 and A.3 and illustrated in Figure A.1. Many EQA Programmes are designed to provide insight into the complete path of workflow of the laboratory, and not just the testing processes. Most EQA programmes are continuous schemes that include long term follow up of laboratory performance. A typical feature of EQA Programmes is to provide education to participants and promote quality improvement. Advisory and educational comments comprise part of the report returned to participants to achieve this aim.
	Some EQA Programmes assess performance of pre–analytical and post–analytical phases of testing, as well as the analytical phase. In such EQA Programmes, the nature of the Proficiency Test item may differ significantly from that used in traditional Proficiency Testing schemes. The “proficiency test item, requiring the participant to select an appropriate approach to testing or interpretation of results, and not just to perform the test. In “sample view” schemes, participants may be required to provide the “’proficiency test items” to the EQA provider (see Figure A.1, model 4). This may take the form of a processed specimen or sample (e.g. stained slide or fixed tissue), laboratory data (e.g. test results, laboratory reports or quality assurance/control records) or documentation (e.g. procedures or method verification criteria).
Model 1 Sequential			Model 2 Simultaneous	Model 3 Interpretive	Model 4 Sample Review	Model 5 Split Sample
Produce/procure test items artefacts			Produce/procure test items	Produce test items, develop questionnaire, or case study	Determine test items to be received from participants	Participants agree on analytes and sample types for comparison
Determine assigned value and its uncertainty			Determine assigned value and acceptable range of results	Distribute questionnaire, case sturdy or test items to participants	Distribute specifications to participants	Participant(s) split appropriate samples and send to others
Distribute to first participant			Distribute test items to participants	Receive results and interpretations from participants	Receive test items from participants	Participants share results or send to a coordinator
Have participants return item or send to next participant			Receive results and method information from participants	Determine acceptable criteria for responses and interpretations	Determine acceptable criteria for responses	Graph or otherwise compare results in this and previous studies
Review participant results and uncertainty for acceptability			Compare participants results and method information with acceptable range	Compare participant results and interpretation with criteria	Compare participants test items with criteria	Compare with pre– established criteria or discuss needs for action
Produce reports and issue advisory /educational comments			Produce reports and issue advisory/educational comments	Produce reports and issue advisory/educational comments	Produce reports and issue advisory/educational comments	Produce reports and records with any agreed conclusions, including data and graphs