Making instruments automation friendly
The business case for including automation capabilities is to increase an instrument’s perceived value, demonstrate a better return on investment (ROI), and broaden customer interest. This goes beyond simply adding remote operations to considering all automation issues.
Good design takes into consideration the whole product life cycle from acquisition through to final decommissioning. The solutions built into a design make the instrument easier to integrate, use and support. When customers seek out one instrument over other competitors, more distribution channels can start to open just by word of mouth.
What does it take to design an instrument for automation? This is where integration best practices can help. The Thermo Scientific Laboratory Automation group has identified three key areas that can determine success or failure; instrument design, consumables use and the application programming interface.
Even though humans and robots have different requirements, every instrument needs to be designed for access by both types of agent. Human requirements for status information, ease-of-access, and serviceability don’t change. What does change is how information is coordinated between agents and how the safety of the operator is protected.
Too often, instrument bulkheads force a single point of access. Either the human or the robot has control, forcing the use of expensive safety barriers and interlocks to protect the operator. A turn table can provide alternating access, or the instrument can be turned sideways for simultaneous access. Better yet, use removable panels instead of bulkheads to create separate workspaces for each human and robot in front and back. Just ensure that the operator’s view of the display panel is not impaired.
As automation extends to the desktop, instruments need to be easy to setup and configure—even for non-technical users. Always provide access to the address and port number of the instrument so that users can resolve network conflicts. TCP-IP protocol is ideal for both hardware and software. If the instrument cannot support TCP-IP then RS-232 serial protocol is also well supported. USB is less stable even though it is becoming more common and serial port expanders are a good alternative.
Customers often select multiple combinations of compounds that must be processed in a specific order, specific time period or combination of both. This makes the plate orientation (landscape or portrait) and A1 well position critical for system operations. Instruments with multiple nests need sequence information to determine this order—especially when container height blocks access to rear nest positions. Customers need this information too. Simply declaring any design assumptions, and what issues are handled automatically or need to be explicitly planned for, will go a long way in helping customers plan their protocols.
Online – offline use
Automation systems take control when the system is online but still depend on human intervention offline to address anomalies that occur beyond their programming. Online, the instrument must support the ability to run unattended. If it cannot complete a step without intervention, the instrument should at least permit the protocol it is running to be aborted so that the rest of the system can continue. Offline, the instrument still needs to support manual operation to recover a plate, clean up a spill or correct a collision.
Automation systems depend on the synchronicity between these processes for performance. Instead of typing on the display console, the interface needs to provide an easy way of down loading protocols from the instrument or an automation host computer. During automatic operation the operator needs to know there is a lockout. When taken offline the automation software needs to know the instrument’s status. Changes between online and offline status need to occur rapidly without needing to re-position, re-teach or re-setup the instrument.
Instrument nests can be arrayed in rows, columns, grids and even circles. Some instruments hide these nests behind solid walls while others expose the deck but retain internal control. A few instruments even allow access to every nest on the deck. Whatever the schema, movers need to know how the nests are arrayed and which ones are legal to access. Let cus¬tomers and integrators know if only one nest can be used as a loading point, how to move internal gripper heads to a safe position programmatically and if the API allows the automation software to control the internal mover.
Figure 1: Nest presented externally
When it comes to automation, nest design isn’t just about location. Customers will judge an instrument on how hard the nest is to access, how complicated it is to teach, how long it takes their mover to access the nest and how long the instrument takes to present a plate. Watch out for these traps:
- Blocking: Instrument bulkheads or enclosures can restrict the path of a mover. Tight maneuvers often require more teaching, odd paths and awkward grip positions that reduce throughput. Use an open design for ease of access and so that human and robot can work from different approaches.
- Countersinking: Nests countersunk below the deck surface need a higher grip position and more clearance when lifting out. Counter sunk positions can add access time, extra positions (or offsets), (re)teach time and error handling to the nest. If countersinking is required, provide lots of clearance.
- Cradling: Straight walls and enclosures that wrap around the nest or ill-placed monuments (locators) can prevent fingers from gripping the plate. Most grippers need a minimum of 0.25 inches (6.35 mm) to open and close the fingers. Be¬tween nests the distance needs to be 15 mm. Try to avoid monuments. Bevel or taper the nest walls for freer access and remove as much enclosure as possible.
- Locking: Some instruments clamp test plates into the nest with snaps or springs. This requires more force than some movers or fingers can physically exert. Avoid physical locks unless the mechanism can be controlled programmatically.
- Presenting: If using an external fixed position, can the mover reach the nest? Some movers have a thick wrist that hits the instrument wall before the nest is reached. If the nest is movable, does it always return to the same position? How long does it take to retrieve the plate? Ensure external nests extend far enough and fast enough. Finally, can the nest remain extended to teach the location? Bias (locking) pins create extra steps and add issues to teaching locations.
For a mover, determining the location of a nest in 3-dimensional space can get complicated very fast. A quick way to overcome this issue is to position instruments around the mover using a template. Then quickly position the instrument using corner locators or registration pins. The design needs to allow for the position of the instrument to be registered relative to the mover and the position of the nest relative to the instrument. As long as this difference is made known, nest locations can be determined in as little as 1/32 of an inch (0.75 mm) tolerance before the mover is taught.
When it comes to ordering plate kits, many customers treat the SBS standard like the Good Housekeeping seal of approval. They concern themselves with the science and assume that any SBS plate works under all conditions. Moreover, there may be several asymmetric (non-SBS footprint) consumables still on the market that are not automation friendly.
While SBS standard has provided some much needed consistency—it is not a panacea. Test the instrument with as many plate types and sizes as possible then expect the improbable. Four common issues can be expected:
- Bevels: Angled plate walls are difficult to grip and let the plate fall from between the fingers. This can be overcome with special fingers provided that the plate has dimples. The plate can be grabbed at these specific locations but the ability to teach locations anywhere else on the plate is lost.
- Flanges: Flanges change the thickness of the plate at different points of elevation requiring multiple grip positions (or offsets). Moreover ANSI/SBS 3-2004 specifically allows for interruptions in the flange height, which can make setting the grip position for a mover more challenging.
- Flexing: Many plastics are semi-rigid, allowing the plate wall to flex and bow without reinforcement. This is allowed under the SBS standard, which “does not imply any preferred or required construction” (ANSI/SBS 1-2004). Even a small variance however, can cause plates to slip or pop out.
- Skirts: Some plates have indents or push-outs on the skirt that cannot be gripped. While most plates sit on the skirt rather than the well bottoms, many thermo cycler or PCR plates are semi-skirted. The skirt seats into the raised edges of the nest and cannot position itself properly if there is not enough skirt to align. When stacked, the bottom most plates can jam together with the raised-rim around the surface of one plate, wedged into the bottom of the plate above.
In an automated laboratory, multiple instruments and vendors must work together—not just co-exist. Instruments that perform poorly can be swapped out for other vendors to eliminate a bottleneck even when the application is specialized. There is no better insurance in a networked environment than a robust, well documented application programming interface (API) that interacts well with the automation software. It should be forward and backward compatible for trouble free migration between versions. Most importantly though, there are three main areas of design that require special attention: communications, error handling and recovery, and event logging.
The simplest automation routine is easily sabotaged by communication issues. At a minimum, the code that runs the instrument should support the following capabilities:
Support remote operation
Automation clients need the ability to put the instrument or its control software into a “remote control” mode. Always support the ability to verify that the instrument is on, working, and in a remote state.
Allow GUI suppression
When operating in a remote state, pass errors and events through the automation software and suppress any error or message boxes in the graphical user interface (GUI). Otherwise, the automation stops pending user intervention. Users don’t always know where to look in a multi-instrument setting.
Always propagate all errors and recovery messages through the API. If an instrument must present its own error message (sometimes this is needed for complex instruments requiring complex error recoveries) the automation software also needs to be notified. Otherwise, the user may never be aware of the problem and the automation software cannot issue a notice.
If a nest moves, a door opens, or a dispenser dispenses, provide a command to invoke the function. Customers tend to expect automation for anything they can do physically and it often reduces the need for manual access. Equally important; if there is no query to open a door that closes after a timeout, a collision can result and damage the instrument!
Return the appropriate status
Instrument control software may have a single application but automation software has to run multiple concurrent operations. Whether a notification event is triggered or uses polling; the status of an instrument must be visible when an operation is complete. Automation processes cannot run without it.
Write non-blocking commands
Synchronous commands that don’t return immediately block other operations and drag out the cycle time of multi-instrument protocols. As multi-instrument, multi-vendor environments grow, developing a “bottleneck” reputation could generate resistance to using the instrument.
Provide a simulation mode
A simulation mode helps customers test their automation protocols as well as software developers create automation services. Ideally, this should include a way to trigger errors so that developers can validate their driver designs without the hardware. The more third-party applications that support the instrument the more demand is created.
Return the firmware version
Help customers avoid unnecessary frustration by providing a function that returns the firmware version. Developers can trigger events and run sanity checks while end users can check for the correct installation.
Figure 2: Gripper and fingers holding a landscape plate
Error handling and recovery
When processes run unattended, the successful completion of a run is determined by how well an instrument handles errors. While it would be nice if every instrument were self-correcting—it is more important that the system be able to skip one plate and continue the protocol without requiring human intervention. Error handling should support these capabilities:
When error messages are not passed to the automation software, the state of an instrument cannot be determined and automatic operation is blocked. Status is required before a run can even start.
The automation client needs to be able to query the instrument state. At a minimum, the instrument control software should be able to return a ready, busy, or error status.
Use a common strategy
Most error dialogs use a common protocol that escalates in severity. Three common recovery strategies are: Ignore, Retry, Abort; Retry and Abort (Ignore is masked out); and Yes, No or Abort.
Write useful descriptions
Error messages should include a human readable description and not just a number. Numbers do not provide the user with useful information. Even better, provide error messages that explain how to recover from the error.
Encapsulate complex strategies
Complex error recovery strategies are better dealt with inside the instrument control software. The vendor manufacturing an instrument and its control software are the true experts and should keep the interface as minimal as possible.
Enable firmware resets
Don’t force customers to reset an instrument by turning the power off then back on to restart the device. This isn’t just inconvenient—it literally prevents the software from recovering after some types of errors. In the wrong situation, one error can bring down the whole automation routine.
Test for days not hours
Automation changes the context of an instrument’s operating environment. Automation systems are expected to run at least 24 to 48 hours unattended without failing. Extending test targets makes issues visible that might otherwise have been missed and avoids expensive service calls later on.
When a failure occurs, many instruments say something about the procedural errors in a method as well as basic com-munication problems. However, that information has no context when shared with other agents outside of the instrument itself. The automation software needs more functionality:
Provide a trace function
Even if the default is off to avoid 21 CRF Part 11 issues, allow customers to log information so that system failures can be tracked and analyzed. By convention, customers should be able to select from general, verbose and debug levels of logging depending on their need.
Avoid proprietary codes
Proprietary codes may keep can a support desk busy but they also prevent customers from trouble shooting problems. This adds cost to the customer’s operations and ownership of the instrument.
When automation routines must run for hours unattended, anything that outputs data can potentially help trouble shoot errors. Generate metadata for all output data to help trace processing issues and use an open format like XML to simplify its collection and logging.
Contextualize the entries
Explicitly tagging entries for basic communications, operations, and error events makes them easier to categorize and sort. While a separate URI for every message would be nice, some devices trigger the same transaction at multiple places in the code—making entries useless without one. Any information exported from the instrument should also contain the type and device instance name in every entry.
Most customers are keenly interested in tracing errors with their consumable media. Instruments with an onboard reader should log barcode numbers for test plates and other consumables when it makes sense.