SLAS

Risk management for laboratory automation projects:Part 1

From LabAutopedia

Jump to: navigation, search
JALACover small.png  A Tutorial from the Journal of the Association for Laboratory Automation

Originally appearing in JALA 2004 9 72-86 (view)


Risk management for laboratory automation projects:Part 1
Authored by: Robert D. McDowall, McDowall Consulting


Goto Part 2; Part 3

Contents

Introduction

There are many Laboratory Information Management System (LIMS) and laboratory automation projects that have collapsed or failed to deliver the expected benefits. Furthermore, surveys of information technology (IT) projects frequently show that many have run over budget, and nearly all projects end up with a changed specification from that originally described. When organizations used IT within laboratory areas in the 1980s and early 1990s, any failures were either covered up or written off as “one of those things” that should be put down to experience. Since then, organizations are far more cost conscious and sensitive of failed projects. Although failure is a powerful learning experience, it is usually never incorporated into a corporate knowledge base for use by similar projects in the future.[1]

A project is a single activity with a well-defined set of end results such as the successful implementation of a LIMS or another automation project. It follows a systems development life cycle (SDLC) from inception to completion.[2] A project does not exist in isolation and must often be coordinated or interfaced with other projects within the parent organization. Projects involve high levels of interdisciplinary communication and coordination with groups of specialists who are not usually used to such interaction. To aid the delivery of successful projects, project management provides an organization with the tools to plan, organize, implement, and control the activities necessary to achieve this.[3] This tutorial is intended to be complementary to existing project management techniques and methodologies.

The complexities and multidisciplinary nature of projects require that the many tasks and deliverable parts of each be put together so that the prime objectives of performance, timescales, and cost are achieved when delivering the defined project end-point. There is a relationship between these three factors that has to be traded off by the project manager. Some of these tradeoffs can involve risk management in varying degrees. This tutorial aims to discuss some general risks and the management of them to ensure a successful outcome of an automation project and is a revision and update of an earlier paper on risk management by the author.[4]

Risk management

To overcome possible poor implementation or failure of a LIMS or laboratory automation project, risk management should be carried out at most stages of the system development life cycle. Risk management should be used in conjunction with project management techniques to manage the overall project. Therefore, identification of the risk factors should allow better management of a project and identify specific areas where additional expertise or care should be taken.

Figure 1. Risk Management Process Flow (Adapted from Ref. 5).

Definitions

Risk is defined for the purposes of this article as the chance or probability of an event occurring that may alter the progress or outcome of a LIMS or laboratory automation project.[4]

The following definitions are taken from ISO 14971:[5]

  • Risk management is the systematic application of management policies, procedures, and practices to the tasks of analysing, evaluating, and controlling risk. From Figure 1, this is the overall process that is the subject of this tutorial.
  • Risk assessment is the overall process of a risk analysis and risk evaluation. This is the major subprocess and comprises analysis and evaluation of risk as shown in Figure 1.
  • Risk analysis is the systematic use of available information to identify hazards and estimate the risk.
  • Risk evaluation is judgment, based on risk analysis, of whether a risk that is acceptable has been achieved in a given context.
  • Risk control is the process through which decisions are reached and protective measures are implemented for reducing risks to, or maintaining risks within, acceptable levels.

Risk management as a process

There is little written in the scientific literature on risk management. Most risk analysis and management is intuitive and undertaken informally by project managers or project teams as a result of their experience or common sense. However, inexperienced individuals or project teams may have problems that could be mitigated or eliminated by the advance knowledge or experience of the common risks associated with LIMS and automation projects. The overall approach is shown in Figure 1.

Risk assessment and management is not a one-step operation but should be carried out at key stages of the SDLC of any project, and it is iterative. A project starts with a high degree of uncertainty and hence high risk. As it progresses, uncertainty in some areas is reduced but in others it can increase, hence the need for repeating the risk analysis and plan approaches to counter any newly identified risks.

Risk analysis

At the top of Figure 1, the input should be at key stages of the project such as when a project definition document is written, system selection, or before implementation and rollout. From the project plan, the individual tasks can be identified for this portion of the work and analysed for potential risk factors. Using the knowledge and experience of the project team members, risk analysis can be carried out and potential risks identified; alternatively, some of the risk management tables in this tutorial can be used (see Tables 1 - 6).

For any stage of the project, the risk analysis process is to:

  • Identify the known or foreseeable factors or hazards that could pose risk to the project, i.e., risk factors that can impact a project can be organizational, financial, or technological.
  • Estimate the risks associated with each factor. For example, what could happen if the factor occurred and what would be the impact on the project?
  • Estimate the probability of the risk occurring.



Table 1. Risk factors associated during project definition stages
Risk factor Low risk High risk
Project scope Well defined Undefined
Project deliverables Well defined Undefined
Benefits of the system Quantified business impact Undefined
System requirements Straightforward, using standard components and technologies Complex, using custom components and new technologies
Personnel providing application knowledge Knowledge of both IT and user areas Lack user area knowledge
Project members who have experience of business area All project members None
Status of documented procedures in user area Complete and current Nonexistent or outdated documentation
Number of computer applications that will interface with the system None Two or more computer applications
Status of these applications Operational Under development
Resource required <5 resource years >15 resource years
Time to develop the system <12 months >3 years



Table 2. Risk factors associated with system sponsorship and user commitment
Risk factor Low risk High risk
Project sponsor Identified and has a strong user influence Unknown
Attitude of user management Fully support the project Resistant and skeptical
Attitude of the users Understand and support the project Resistant and skeptical 
Organizational maturity Able to use the system effectively Unable to understand rationale for the system or use it
Relationship of the project to the strategic IT plan Included in the plan Not included in the plan



Table 3. Risk factors associated with the impact of the system on the organization
Risk factor Low risk High risk
The new system N/A Replaces an existing one 
    A completely new system
Effect of the system on the IT department of the organization Little change Much change
Procedural changes required by new system Little change Much change
Organizational structures None required or no changes to existing ones Not considered
Policy changes required to support the new system None Extensive



Table 4. Risk factors associated with project management
Risk factor Low risk High risk
Experience of project manager 3 or more projects No prior experience
Managing the project Full time Part time
Project team assigned Full time Part time
Experience of team members as a team All worked together before All are strangers
Number of times application/system implemented More than once No experience of this application
Team location In one place In several locations



Table 5. Risk factors associated with system selection
Risk factor Low risk High risk
New or non standard hardware or system software required No Yes
Team has experience of this application/system Expertise available Used for the first time
New language(s) required by the project None Used for the first time
Database used in the application Well established in the organization New in the organization
The system processing Batch processing Distributed system
On-line response required >7 seconds 90% of time 2 seconds or less 90% of time
System availability 95% >99%
Technology mix (database, network, etc) Existing or simple architecture New or complex architecture
Team's knowledge of the package Previous experience No knowledge
Organization has worked with the vendor Three or more times before Never
The package matches the system requirements Well (little customization required) Poor (major customization required)
Computer department involvement in package selection High involvement Not involved
User department involvement in package selection High involvement Not involved
Vendor reputation Good Poor or unknown



Table 6. Risk factors associated with system development and implementation
Risk factor Low risk High risk
System scope fixed and prioritized Yes No
Changes in organization or policy implemented Yes No
Change control procedures in place Yes No
Scope matches existing or proposed working practices Yes No
Documented roll-out plan available, detailing who, when and where Yes No
Sympathetic users identified for prototyping and testing Yes No
Quality Assurance involved for pre-operation audit and GMP advice Yes No
Development documentation to be produced identified Yes No
Performance of development system matches expectation Yes No
Contingency plan available if performance not good enough Yes No
Focused training planned and agreed with the users Yes No
Documentation available for users, backup and support staff Yes No
Assistance arranged for new users in period after training Yes No
Work schedules altered to cope with post training productivity loss Yes No
Plan for management of user expectations Yes No

Risk evaluation

The evaluation process is very simple—it asks the question, Does the risk need to be mitigated or not?

If the answer is “no,” then the risk is accepted and nothing further is required. However, if there is a requirement for mitigation, then the risk moves into the next stage of the process: risk control. Typically, only high-risk factors will pass to the next stage; however, this decision depends on the criticality of the project in question.

Risk control

Once the high-risk tasks have been highlighted, then it is possible to prepare plans and countermeasures to overcome the risk. “Risk Factors during Project Definition and Start-up,” “Evaluation and Selection of the System,” and “Risks Associated during Development and Rollout” discuss the risk and some of the possible approaches to mitigation. Note that it is not always possible to eliminate a risk, as this may be impossible or require too much effort; however, sufficient work needs to be done to ensure that the impact of the risk is managed and is acceptable.

The project manager then implements these approaches within the updated project plan. Milestones of the project can be identified and progress of the project reviewed against these; the same is true of risk factors. Once the progress of the project has been evaluated, this can be fed back into the system for an updated risk analysis. As can be seen, risk analysis is linked very closely with project management, and the two approaches should operate intimately.

Project risk information

As the project progresses, a body of information is collected. It describes how the project risk has been managed and how effective the approaches have been. It is important that this information is not forgotten or ignored. Reuse, cross-reference, and update the information; it can be used to feed back into the risk cycles as shown in Figure 1. Also, experience from failed projects, discussed briefly in “Learning from Failure,” should also be incorporated in the organization's experience of automation projects as lessons to be learnt for the future.

Areas of risk in the system development life cycle

As laboratories depend so much on automation, and LIMS in particular, it is essential that management, users, and the project team should do as much as possible to minimize the risks to ensure a successful implementation. If an LIMS is not functioning effectively, then work within the laboratories will be disrupted and productivity will suffer. However, when an LIMS is operating badly or not working at all, then the customer does not obtain the information and the reputation of the laboratory suffers.

The three main areas of risk within the SDLC are:

  • Project definition and start-up (see “Risk Factors during Project Definition and Start-up”)
  • Evaluation and selection of the proposed system (see “Evaluation and Selection of the System”)
  • Implementation of the system (see “Risks Associated During Development and Rollout”)

Risk factor tables

The factors highlighted in the accompanying tables will allow continuing assessments of risk to be made of individual projects at various stages of the SDLC. As there is usually an underestimate made of the technical complexity of systems development, risk can never be eliminated totally from a project. However, the more that significant factors can be contained or contingency plans made to manage them at the start and throughout a project, then, the greater the chances of success. Over optimism, especially in the planning stage, is a chronic problem, resulting in projects being over time and over budget. Therefore, contingency time and money should be included in all plans.

Risk can be assessed as probable (high risk), possible (low risk), and improbable (negligible). Another approach is to assign to each factor a numerical value, say 10, for the highest risk, and 1, for negligible risk. Risk can now be evaluated on a continuum, which can be useful for an assessment of risk for a number of projects such as a prioritization exercise. However, in the tables presented in this tutorial, only high and low risks have been evaluated, as it is preferable, in the author's view, to keep the scheme as simple as possible.


Background reading

Although not directly referenced in this tutorial, the following books are useful background reading for computerized system failures (and the occasional success):

  • Crash: Learning from the World's Worst Computer Disasters. Tony Collins with David Bicknell (1998), Simon and Schuster, London. ISBN 0-684-81687-3. The 10 deadly sins of computer failure are worth reading along with the case studies of many failed computer system projects—read and learn. However, as the authors note in the preface, the book has gone through two reprints and a second edition, but the book has not had the slightest beneficial effect.
  • Assessment and Control of Software Risks. Capers Jones (1994), Yourden Press, Upper Saddle River, NJ. ISBN 0-13-741406-4. This books goes into more detail about project failure, but it is a more academic approach to the subject than Crash.
  • Patterns of Software Systems Failure and Success. Capers Jones (1996), International Thompson Press, Boston, MA. ISBN 1-850-32804-8. Following the themes of his 1994 book, Jones looks at the reasons for successes and failures of software projects.
  • Managing Risk: Methods for Software Systems Development. Elaine Hall (1998), Addison-Wesley, Reading, MA. ISBN 0-201-25592-8. A detailed approach for managing software development that can also be applied easily to automation and robotic projects.

Although the emphasis of these books is mainly on software, the principles outlined in them can be applied to other automation projects with little effort.  Project management is also important and plays a major role in determining if a project will be successful or not. Two key books that should be read are:

  • Software Project Management, A Unified Framework. Walker Royce (1998), Addison-Wesley, Reading MA. ISBN 0-201-30958-0. Excellent book on managing software projects. Read and follow the principles and advice in this book and you won't need the next one.
  • Troubled IT Projects: Prevention and Turnaround. John Smith (2001),Institute of Electrical Engineers, London. ISBN 0-85296-104-9. A good practical approach to project salvage and resurrection.


Goto Part 2; Part 3

References

  1. McDowall RD. Strategic approaches to laboratory automation, chemometrics and intelligent laboratory systems. Laboratory Information Management. 1992;17:265–282.
  2. McDowall RD. The system development life cycle, chemometrics and intelligent laboratory systems. Laboratory Information Management. 1991;13:121–133.
  3. Royce W. Software Project Management - A Unified Framework. Reading, MA: Addison-Wesley; 1998;.
  4. 4.0 4.1 McDowall RD. The evaluation and management of risk during a laboratory information management system or laboratory automation project, chemometrics and intelligent laboratory systems. Laboratory Information Management. 1993;21:1–19.
  5. ISO 14971: 2000. Medical Devices: Application of Risk Management to Medical Devices. Geneva: Internal Standards Organisation; 2000
Click [+] for other articles on 
Project management(1 C, 9 P)
The Market Place for Lab Automation & Screening  The Market Place