Risk management for laboratory automation projects:Part 1
| A Tutorial from the Journal of the Association for Laboratory Automation|
Originally appearing in JALA 2004 9 72-86 (view)
There are many Laboratory Information Management System (LIMS) and laboratory automation projects that have collapsed or failed to deliver the expected benefits. Furthermore, surveys of information technology (IT) projects frequently show that many have run over budget, and nearly all projects end up with a changed specification from that originally described. When organizations used IT within laboratory areas in the 1980s and early 1990s, any failures were either covered up or written off as “one of those things” that should be put down to experience. Since then, organizations are far more cost conscious and sensitive of failed projects. Although failure is a powerful learning experience, it is usually never incorporated into a corporate knowledge base for use by similar projects in the future.
A project is a single activity with a well-defined set of end results such as the successful implementation of a LIMS or another automation project. It follows a systems development life cycle (SDLC) from inception to completion. A project does not exist in isolation and must often be coordinated or interfaced with other projects within the parent organization. Projects involve high levels of interdisciplinary communication and coordination with groups of specialists who are not usually used to such interaction. To aid the delivery of successful projects, project management provides an organization with the tools to plan, organize, implement, and control the activities necessary to achieve this. This tutorial is intended to be complementary to existing project management techniques and methodologies.
The complexities and multidisciplinary nature of projects require that the many tasks and deliverable parts of each be put together so that the prime objectives of performance, timescales, and cost are achieved when delivering the defined project end-point. There is a relationship between these three factors that has to be traded off by the project manager. Some of these tradeoffs can involve risk management in varying degrees. This tutorial aims to discuss some general risks and the management of them to ensure a successful outcome of an automation project and is a revision and update of an earlier paper on risk management by the author.
To overcome possible poor implementation or failure of a LIMS or laboratory automation project, risk management should be carried out at most stages of the system development life cycle. Risk management should be used in conjunction with project management techniques to manage the overall project. Therefore, identification of the risk factors should allow better management of a project and identify specific areas where additional expertise or care should be taken.
Risk is defined for the purposes of this article as the chance or probability of an event occurring that may alter the progress or outcome of a LIMS or laboratory automation project.
The following definitions are taken from ISO 14971:
- Risk management is the systematic application of management policies, procedures, and practices to the tasks of analysing, evaluating, and controlling risk. From Figure 1, this is the overall process that is the subject of this tutorial.
- Risk assessment is the overall process of a risk analysis and risk evaluation. This is the major subprocess and comprises analysis and evaluation of risk as shown in Figure 1.
- Risk analysis is the systematic use of available information to identify hazards and estimate the risk.
- Risk evaluation is judgment, based on risk analysis, of whether a risk that is acceptable has been achieved in a given context.
- Risk control is the process through which decisions are reached and protective measures are implemented for reducing risks to, or maintaining risks within, acceptable levels.
Risk management as a process
There is little written in the scientific literature on risk management. Most risk analysis and management is intuitive and undertaken informally by project managers or project teams as a result of their experience or common sense. However, inexperienced individuals or project teams may have problems that could be mitigated or eliminated by the advance knowledge or experience of the common risks associated with LIMS and automation projects. The overall approach is shown in Figure 1.
Risk assessment and management is not a one-step operation but should be carried out at key stages of the SDLC of any project, and it is iterative. A project starts with a high degree of uncertainty and hence high risk. As it progresses, uncertainty in some areas is reduced but in others it can increase, hence the need for repeating the risk analysis and plan approaches to counter any newly identified risks.
At the top of Figure 1, the input should be at key stages of the project such as when a project definition document is written, system selection, or before implementation and rollout. From the project plan, the individual tasks can be identified for this portion of the work and analysed for potential risk factors. Using the knowledge and experience of the project team members, risk analysis can be carried out and potential risks identified; alternatively, some of the risk management tables in this tutorial can be used (see Tables 1 - 6).
For any stage of the project, the risk analysis process is to:
- Identify the known or foreseeable factors or hazards that could pose risk to the project, i.e., risk factors that can impact a project can be organizational, financial, or technological.
- Estimate the risks associated with each factor. For example, what could happen if the factor occurred and what would be the impact on the project?
- Estimate the probability of the risk occurring.
|Risk factor||Low risk||High risk|
|Project scope||Well defined||Undefined|
|Project deliverables||Well defined||Undefined|
|Benefits of the system||Quantified business impact||Undefined|
|System requirements||Straightforward, using standard components and technologies||Complex, using custom components and new technologies|
|Personnel providing application knowledge||Knowledge of both IT and user areas||Lack user area knowledge|
|Project members who have experience of business area||All project members||None|
|Status of documented procedures in user area||Complete and current||Nonexistent or outdated documentation|
|Number of computer applications that will interface with the system||None||Two or more computer applications|
|Status of these applications||Operational||Under development|
|Resource required||<5 resource years||>15 resource years|
|Time to develop the system||<12 months||>3 years|
|Risk factor||Low risk||High risk|
|Project sponsor||Identified and has a strong user influence||Unknown|
|Attitude of user management||Fully support the project||Resistant and skeptical|
|Attitude of the users||Understand and support the project||Resistant and skeptical|
|Organizational maturity||Able to use the system effectively||Unable to understand rationale for the system or use it|
|Relationship of the project to the strategic IT plan||Included in the plan||Not included in the plan|
|Risk factor||Low risk||High risk|
|The new system||N/A||Replaces an existing one|
|A completely new system|
|Effect of the system on the IT department of the organization||Little change||Much change|
|Procedural changes required by new system||Little change||Much change|
|Organizational structures||None required or no changes to existing ones||Not considered|
|Policy changes required to support the new system||None||Extensive|
|Risk factor||Low risk||High risk|
|Experience of project manager||3 or more projects||No prior experience|
|Managing the project||Full time||Part time|
|Project team assigned||Full time||Part time|
|Experience of team members as a team||All worked together before||All are strangers|
|Number of times application/system implemented||More than once||No experience of this application|
|Team location||In one place||In several locations|
|Risk factor||Low risk||High risk|
|New or non standard hardware or system software required||No||Yes|
|Team has experience of this application/system||Expertise available||Used for the first time|
|New language(s) required by the project||None||Used for the first time|
|Database used in the application||Well established in the organization||New in the organization|
|The system processing||Batch processing||Distributed system|
|On-line response required||>7 seconds 90% of time||2 seconds or less 90% of time|
|Technology mix (database, network, etc)||Existing or simple architecture||New or complex architecture|
|Team's knowledge of the package||Previous experience||No knowledge|
|Organization has worked with the vendor||Three or more times before||Never|
|The package matches the system requirements||Well (little customization required)||Poor (major customization required)|
|Computer department involvement in package selection||High involvement||Not involved|
|User department involvement in package selection||High involvement||Not involved|
|Vendor reputation||Good||Poor or unknown|
|Risk factor||Low risk||High risk|
|System scope fixed and prioritized||Yes||No|
|Changes in organization or policy implemented||Yes||No|
|Change control procedures in place||Yes||No|
|Scope matches existing or proposed working practices||Yes||No|
|Documented roll-out plan available, detailing who, when and where||Yes||No|
|Sympathetic users identified for prototyping and testing||Yes||No|
|Quality Assurance involved for pre-operation audit and GMP advice||Yes||No|
|Development documentation to be produced identified||Yes||No|
|Performance of development system matches expectation||Yes||No|
|Contingency plan available if performance not good enough||Yes||No|
|Focused training planned and agreed with the users||Yes||No|
|Documentation available for users, backup and support staff||Yes||No|
|Assistance arranged for new users in period after training||Yes||No|
|Work schedules altered to cope with post training productivity loss||Yes||No|
|Plan for management of user expectations||Yes||No|
The evaluation process is very simple—it asks the question, Does the risk need to be mitigated or not?
If the answer is “no,” then the risk is accepted and nothing further is required. However, if there is a requirement for mitigation, then the risk moves into the next stage of the process: risk control. Typically, only high-risk factors will pass to the next stage; however, this decision depends on the criticality of the project in question.
Once the high-risk tasks have been highlighted, then it is possible to prepare plans and countermeasures to overcome the risk. “Risk Factors during Project Definition and Start-up,” “Evaluation and Selection of the System,” and “Risks Associated during Development and Rollout” discuss the risk and some of the possible approaches to mitigation. Note that it is not always possible to eliminate a risk, as this may be impossible or require too much effort; however, sufficient work needs to be done to ensure that the impact of the risk is managed and is acceptable.
The project manager then implements these approaches within the updated project plan. Milestones of the project can be identified and progress of the project reviewed against these; the same is true of risk factors. Once the progress of the project has been evaluated, this can be fed back into the system for an updated risk analysis. As can be seen, risk analysis is linked very closely with project management, and the two approaches should operate intimately.
Project risk information
As the project progresses, a body of information is collected. It describes how the project risk has been managed and how effective the approaches have been. It is important that this information is not forgotten or ignored. Reuse, cross-reference, and update the information; it can be used to feed back into the risk cycles as shown in Figure 1. Also, experience from failed projects, discussed briefly in “Learning from Failure,” should also be incorporated in the organization's experience of automation projects as lessons to be learnt for the future.
Areas of risk in the system development life cycle
As laboratories depend so much on automation, and LIMS in particular, it is essential that management, users, and the project team should do as much as possible to minimize the risks to ensure a successful implementation. If an LIMS is not functioning effectively, then work within the laboratories will be disrupted and productivity will suffer. However, when an LIMS is operating badly or not working at all, then the customer does not obtain the information and the reputation of the laboratory suffers.
The three main areas of risk within the SDLC are:
- Project definition and start-up (see “Risk Factors during Project Definition and Start-up”)
- Evaluation and selection of the proposed system (see “Evaluation and Selection of the System”)
- Implementation of the system (see “Risks Associated During Development and Rollout”)
Risk factor tables
The factors highlighted in the accompanying tables will allow continuing assessments of risk to be made of individual projects at various stages of the SDLC. As there is usually an underestimate made of the technical complexity of systems development, risk can never be eliminated totally from a project. However, the more that significant factors can be contained or contingency plans made to manage them at the start and throughout a project, then, the greater the chances of success. Over optimism, especially in the planning stage, is a chronic problem, resulting in projects being over time and over budget. Therefore, contingency time and money should be included in all plans.
Risk can be assessed as probable (high risk), possible (low risk), and improbable (negligible). Another approach is to assign to each factor a numerical value, say 10, for the highest risk, and 1, for negligible risk. Risk can now be evaluated on a continuum, which can be useful for an assessment of risk for a number of projects such as a prioritization exercise. However, in the tables presented in this tutorial, only high and low risks have been evaluated, as it is preferable, in the author's view, to keep the scheme as simple as possible.
Although not directly referenced in this tutorial, the following books are useful background reading for computerized system failures (and the occasional success):
- Crash: Learning from the World's Worst Computer Disasters. Tony Collins with David Bicknell (1998), Simon and Schuster, London. ISBN 0-684-81687-3. The 10 deadly sins of computer failure are worth reading along with the case studies of many failed computer system projects—read and learn. However, as the authors note in the preface, the book has gone through two reprints and a second edition, but the book has not had the slightest beneficial effect.
- Assessment and Control of Software Risks. Capers Jones (1994), Yourden Press, Upper Saddle River, NJ. ISBN 0-13-741406-4. This books goes into more detail about project failure, but it is a more academic approach to the subject than Crash.
- Patterns of Software Systems Failure and Success. Capers Jones (1996), International Thompson Press, Boston, MA. ISBN 1-850-32804-8. Following the themes of his 1994 book, Jones looks at the reasons for successes and failures of software projects.
- Managing Risk: Methods for Software Systems Development. Elaine Hall (1998), Addison-Wesley, Reading, MA. ISBN 0-201-25592-8. A detailed approach for managing software development that can also be applied easily to automation and robotic projects.
Although the emphasis of these books is mainly on software, the principles outlined in them can be applied to other automation projects with little effort. Project management is also important and plays a major role in determining if a project will be successful or not. Two key books that should be read are:
- Software Project Management, A Unified Framework. Walker Royce (1998), Addison-Wesley, Reading MA. ISBN 0-201-30958-0. Excellent book on managing software projects. Read and follow the principles and advice in this book and you won't need the next one.
- Troubled IT Projects: Prevention and Turnaround. John Smith (2001),Institute of Electrical Engineers, London. ISBN 0-85296-104-9. A good practical approach to project salvage and resurrection.
- ↑ McDowall RD. Strategic approaches to laboratory automation, chemometrics and intelligent laboratory systems. Laboratory Information Management. 1992;17:265–282.
- ↑ McDowall RD. The system development life cycle, chemometrics and intelligent laboratory systems. Laboratory Information Management. 1991;13:121–133.
- ↑ Royce W. Software Project Management - A Unified Framework. Reading, MA: Addison-Wesley; 1998;.
- ↑ 4.0 4.1 McDowall RD. The evaluation and management of risk during a laboratory information management system or laboratory automation project, chemometrics and intelligent laboratory systems. Laboratory Information Management. 1993;21:1–19.
- ↑ ISO 14971: 2000. Medical Devices: Application of Risk Management to Medical Devices. Geneva: Internal Standards Organisation; 2000
|Click [+] for other articles on||The Market Place for Lab Automation & Screening||The Market Place|