Computer Validation. Risk analysis  and system classification

Here you will find answers to the following questions:

Which criteria can be used to classify computer systems into risk classes?

Which are the GAMP classifications?

Which measures should be taken depending risk class?

Chapter 10 of the GMP Guide (see chapter 10 Risk Management) provides an excellent description of risk management in general. This chapter is therefore only intended to cover the practice-related, specific aspects of risk management for computerised systems.

9.D.1 GAMP classification

In the annexes of the GAMP4 Guide, software is divided into five classes and hardware into two classes, depending on their complexity. The classes are divided according to the validation requirement and deliverables.

The complexity and validation requirements increase from class 1 to class 5.

9.D.1.1 Class 1 - Operating system

In this class, only the name and the version are documented. The operating system is implicitly tested and hence covers the validation of the application, since all higher functions rely on this functioning flawlessly.

9.D.1.2 Class 2 - Firmware

Firmware is typically used in controls and instruments. Firmware can require that certain parameters are entered and configured, such as the date format, current date, time, etc. Firmware may also belong class 5, if it has been specifically programmed for an installation.

GAMP4, Annex M4 (Ed. ISPE) proposes recording of the name and version for firmware. This name and the version may not be visible from externally. In this case, it is pragmatic to record the equipment using design and serial number, since only externally accessible information is easy to verify and to test. Examples in this class include washing machines, drying ovens, balances, barcode scanners, etc.

The correct functioning of the software is tested and documented as a part of the installation qualification (IQ) and operational qualification (OQ). The configuration parameters must be verified and documented during qualification. It is important that all changes in the firmware, as well as any changes to the configuration parameters, are subject to change control. A detailed procedure for qualification is provided in chapter 6 Qualification.

Before start-up, it must be ensured that the necessary operating instructions (SOPs - Standard Operating Procedures) are available and that users are trained in these.

9.D.1.3 Class 3 - Standard software packages

Commercially available software normally falls into class 3. This typically includes analysis software in spectrophotometers, HPLC systems, gas chromatographs (GC), near infrared spectrophotometers (NIRA) and other laboratory instruments, as well as statistical evaluation packages. This class also includes all off-the-shelf software: COTS (commercial off the shelf), such as office programs.

The procedure for class 3 is the same as class 2. In addition, all calculation methods, algorithms, alarm messages and warning messages used must be checked and documented during qualification, normally operational qualification. For standard software packages used in highly-sensitive GMP areas with a high risk for product quality, it may necessary to execute a supplier assessment or even a supplier audit.

9.D.1.4 Class 4 - Configurable standard software packages

These are usually more complex applications that are available with a wide variety of standard interfaces. Typical members of this class are: MES (Manufacturing Execution Systems), ERP (Enterprise Resource Planning), control systems (SCADA - Supervisory Control And Data Acquisition) and DCS (Distributed Control Systems) such as room temperature controllers.

The complete functionality can be changed by configuration. Specific functions may often be realised using programs or macros; these should then be treated as class 5. Validation should be executed as described in chapter 9.E Validation of computerised systems and all details must be recorded as described above for class 1-3. In validation, particular attention should be paid to the configuration.

9.D.1.5 Class 5 - Customer-specific software

Whenever an application is programed for an individual application, it belongs to this class. Depending on the risk classification, in this case the whole set of validation activities must be executed. The details of documentation requirements are described in chapter 15 Documentation.

9.D.1.6 Validation tasks depending on classification

Listing the requirements that are detailed in the GAMP Guide results in a table similar to the following, which contains a list of the validation tasks depending on system complexity (figure 9.D-1).

Figure 9.D-1 Table of planned validation activities depending on the system class  

   

Firmware

Standard
systems

Configurable systems

Configurable
systems with customer-specific additions

Customer-
specific systems

Inventory

Classification

X

X

X

X

X

Risk evaluation

X

X

X

X

X

Inventory

X

X

X

X

X

Validation

Validation plan

 

X

X

X

X

Validation report

 

X

X

X

X

Validation registry

 

X

X

X

X

Qualification plan

X

       

Qualification report

X

       

Supplier management

Supplier assessment

 

X

X

X

X

Specify & Design

User requirement specifications

X

X

X

X

X

System specifications

   

X

X

X

Development standards

     

X

X

Project change control SOP

   

X

X

X

Functional risk evaluation

   

X

X

X

Technical design

   

X

X

X

Build

Source code, configuration, adjustment

     

X

X

Technical documents for IT

 

X

X

X

X

Test

Test protocols

 

X

X

X

X

Unit/integration test specifications

     

X

X

Unit/integration test report

     

X

X

Installation test documentation

X

X

X

X

X

System test documentation

   

X

X

X

Acceptance test documentation

X

X

X

X

X

Traceability matrix

X

X

X

X

X

System configuration basis

X

X

X

X

X

Implement & Use

Implementation plan

   

X

X

X

User SOP

X

X

X

X

X

Change control SOP

X

X

X

X

X

Security administration SOP

 

X

X

X

X

SOP for backup and restore

 

X

X

X

X

SOP for archiving

 

X

X

X

X

SOP for user training

X

X

X

X

X

Service level agreement

   

X

X

X

SOP for periodic review

X

X

X

X

X

Plan for system shutdown

 

X

X

X

X

System shutdown report

 

X

X

X

X

9.D.1.7 Risk classification in accordance with GAMP

The GAMP Guide contains an annex for risk classification that describs a simple and practically executable double classification of software.

Firstly, a risk is determined by evaluating the extent of damage and the frequency of an event. In the second stage, the probability of detection of an error is also considered, in order to determine the risk index (figure 9.D-2).

Figure 9.D-2 Determination of the risk index in accordance with GAMP

9.D.2 Risk indexes

Scientifically, the risk is the product of the probability of occurrence and the degree of severity of an unwanted effect.

Risk = Probability of occurrence x Degree of damage x Probability of detection

Since it is difficult to work with abstract probabilities, in everyday life risk indexes are used, which are a combination of the probability of occurrence, the degree of damage, and the probability of detection, . It is common to determine the risk indexes empirically. An empirical method of this type is presented here. It is assumed that the risk index can be determined as a weighted sum of the different individual risks (see figure 9.D-3). For more details on risk analysis, see chapter 10.F Failure Mode Effects Analysis (FMEA).

9.D.2.1 Determining the risk index

The following diagram (figure 9.D-3) shows a possible example for determining the risk key figure.

Figure 9.D-3 Table for determining the risk index RI  

Label

Category

Question

Points

G

Good Practice

Can GCP, GLP, GMP or other Good Practices (SOX - Sarbanes Oxley) be applied?

yes

1

no

0

A

Regulatory requirements at the implementation location

Research

1

Development - GLP (Good Laboratory Practice)

3

Development - GMP (Good Manufacturing Practice)

5

Development - GCP (Good Clinical Practice)

5

API manufacturing

3

Drug product manufacturing - GMP

5

Distribution and storage

3

Planning and logistics

1

Support processes - human resources

1

Support processes- information technology

3

Support processes - finances

1

N

No. of users

Although the probability of detection theoretically increases, the probability of occurrence and extent of damage also rapidly increase.

1

1

2 to 20

3

21 to 100

7

More than 100

10

F

Frequency of use

Annually

1

Monthly

3

Daily

5

V

Geographical distribution

At one workplace

1

In one department

3

At one manufacturing site

5

Global

10

M

Manual alternative solution

Without great effort

1

With extra effort

3

Only with considerable extra effort

5

Not feasible

7

P

Importance of the data in the system

Is the data only available in the affected system?

yes

1

no

0

I

Inspection risk

 

yes

1

no

0

P

Potential risk to patients

No influence on patients

1

Insignificant

2

Injury

5

Death

10

n

Subsequent processes

Has a correctable system failure been discovered in subsequent processes?

yes

1

no

10

S

System category

a - Firmware

1

b - Standard software package

3

c - Configurable software package

5

d- c With custom-developed supplements

8

e - custom-developed solution

10

M

Maturity or degree of maturity of the technology

New technology

10

Mature technology without internal company experience

5

Mature technology with internal company experience

2

Obsolete technology

5

o

Openness, Internet

Is part or all of the whole communication processed via the public Internet?

yes

10

no

1

RI = G * (A+N+H+V+M+W+I+P+N+S+M+O)

Example:

A spreadsheet for the calculation of content uniformity determination in accordance with the pharmacopoeia in release analytics, which does not contain any macros, would have an RI of 42:

G

Regulations applicable

yes

1

A

Regulatory requirements at the implementation location

GMP

5

N

No. of users

2 to 20

3

F

Frequency of use

Daily

5

V

Geographical distribution

In one department

3

M

Manual alternative solution

With extra effort

3

P

Importance of the data in the system

no

0

I

Inspection risk

yes

1

P

Potential risk to patients

Injury

5

N

Subsequent processes

no

10

S

System category

c - Configurable software package

5

M

Degree of maturity of the technology

Internal company experience

1

O

Openness, Internet

no

1

RI = 1 * (5+3+5+3+3+0+1+5+10+5+1+1) = 42

The RI can now be used to determine the validation priority:

Low

RI < 25

Medium

25 Ј RI < 55

High

55 Ј RI < 99

Equipment with the RI "0" result when no GxP requirements are placed on a system. Systems with no regulatory requirements are often found in administrative areas, e.g. a program for the administration of company parking spaces.

It is a good idea to perform a reality check using the empirical formula before it is globally implemented. Ultimately, the results of the RI are only a guide. Particularly in borderline cases, equipment or a computerised system can be assigned a different validation priority to that which would be determined by the analysis.

9.D.2.2 Measures that depend on the risk index

At every stage, the risk index enables efforts to be concentrated on the decisive points in the project. The following five tables contain the relevant guidelines depending on the system class and risk. In contrast to the risk index, the risk is only specified as low, medium or high.

Firmware

Figure 9.D-4 Measures for risks of the class "firmware"  

Deliverable

Risk

Level of detail / scalability of content

User SOP
technical instructions

Low

Manufacturer's instructions

Medium

Manufacturer' instructions: SOP, if use is complicated or error-prone

High

Change management

Independent of risk

SOPs for the different points are necessary and are not scalable in accordance with risk.

This may be a general SOP that covers a wide range of equipment.

User training

Periodic review

Standard systems

Figure 9.D-5 Measures for risks of the class "standard systems" 

Deliverable

Risk

Level of detail / scalability of content

User SOP
technical instructions

Low

Manufacturer' instructions: SOP, if use is complicated or error-prone

Medium

System-specific SOP

High

Change management

Independent of risk

SOPs for the different points are necessary and are not scalable in accordance with risk.

This may be a general SOP that covers a wide range of equipment.

SOP for administration of physical and logical security

SOP for backup and restore

SOP for archiving

User training

Service level agreement

Low

System can be included with various other devices in a global service level agreement

Medium

High

Service level agreement should include specific information on rapid reactions within hours.

SOP for periodic review

Low

Medium

SOPs for periodic testing are required

This may be a general SOP that covers a wide range of equipment.

High

High-risk systems should be assessed more frequently.

Retirement plan and report

Low

None

Medium

Simple plan

High

For a certain time period it should be possible to access the legacy system or reactivate it in an emergency.

Configurable standard systems

Figure 9.D-6 Measures for risks of the class "configurable standard systems"

Deliverable

Risk

Level of detail / scalability of content

Implementation plan

Low

Checklist with installation parameters for the individual workplace

Medium

System-specific implementation plan

High

User SOP
technical instructions

Low

Manufacturer's instructions
SOP, if use is complicated or error-prone

Medium

System-specific SOP

High

Change management

Independent of risk

SOPs for the different points are necessary and are not scalable in accordance with risk.

This may be a general SOP that covers a wide range of equipment.

SOP for administration of physical and logical security

SOP for backup and restore

SOP for archiving

SOP for user training

Service level agreement

Low

System can be included with various other devices in a global service level agreement

Medium

High

Service level agreement should include specific information on rapid reactions within hours.

SOP for periodic review

Low

Medium

SOPs for periodic testing are required.

This may be a general SOP that covers a wide range of equipment.

High

High-risk systems should be assessed more frequently.

Retirement plan and report

Low

None

Medium

Simple plan

High

For a certain time period it should be possible to access the legacy system or reactivate it in an emergency.

Standard systems with custom-developed supplements

Figure 9.D-7 Measures for risks for the class "standard systems with custom-developed supplements" 

Deliverable

Risk

Level of detail / scalability of content

Implementation plan

Low

Checklist with workplace settings

Medium

System-specific implementation plan

High

User SOP
technical instructions

Low

Manufacturer's instructions
SOP, if use is complicated or error-prone

Medium

System-specific SOP

High

Change management

Independent of risk

SOPs for the different points are necessary and are not scalable in accordance with risk.

This may be a general SOP that covers a wide range of equipment.

SOP for administration of physical and logical security

SOP for backup and restore

SOP for archiving

SOP for user training

Service level agreement

Low

System can be included with various other devices in a global service level agreement

Medium

High

Service level agreement should include specific information on rapid reactions within hours.

SOP for periodic review

Low

Medium

SOPs for periodic testing are required.

This may be a general SOP that covers a wide range of equipment.

High

High-risk systems should be assessed more frequently.

Shutdown plan and report

Low

None

Medium

Simple plan

High

For a certain time period it should be possible to access the legacy system or reactivate it in an emergency.

Custom-developed system

In terms of scalability and level of detail, the requirements for a custom-developed system are the same as the requirements for standard systems with custom-developed supplements. The difference is that these requirements apply to the whole system and not only to the supplements.

9.D.3 Risk management at the level of user requirements

For systems with a high risk, it is recommended to continue the risk analysis and also include the user requirements in the risk management process. Risk minimisation is particularly important in this context.

Table 4.2 in Annex M3 of the GAMP Guide presents the following examples of strategies for minimising risk:

1. Changes to process or system design elements for reducing risk:

  • Changes to the process design: One or more independent controls are integrated in the computerised process (e.g. additional data verification checks, plausibility checks, checksums as the last character when checking numeric entries).
  • Introduction of all measures: Process changes external to the system may also make a contribution. "Double check" is the most widespread method.
  • Changes to the product or system design are mainly made together with infrastructure changes. These include redundancies, error-tolerant work processes and uninterruptible power supplies, etc.

2. Changes to the project strategy for minimising risk:

  • Persons who work on the project must have the appropriate training and qualifications in order to detect risks. This can be supported by the development of specific training and coaching program.
  • The integration of testable deliverables and milestones can be used more frequently in higher-risk projects.

3. Risk elimination

  • If the risk is particularly high for certain user requirements, in some circumstances these should not be implemented at all in order to completely exclude the risk.

    Summary:

    The simple classification of the GAMP Guide only considers system complexity for the necessary validation activities, and differentiates between the five classes "operating systems, firmware, standard systems, configurable systems, and custom-developed systems". Certain deliverables such as the implementation plan, user SOPs, service level agreements, SOP for periodic review, and shutdown documentation, can be scaled according to the risk, in order to focus the validation activities on the points that represent the greatest risk.

Schьtz M. (Ed.): Risiko und Wagnis. Die Herausforderung der industriellen Welt. Bd. 1 und 2, Pfullingen 1990.

ISO/IEC Guide 73:2002 - Risk Management - Vocabulary - Guidelines for use in Standards.

ISO 14971:2000 - Application of Risk Management to Medical Devices.