Verifying and Validating Systems for Trustworthiness
Dev Raheja, MS, CSP, author of the books Design for Reliability”, Preventing Medical Device Recalls, and Safer Hospital Care, is international risk management, reliability, durability, and system safety consultant for the government, commercial, and aerospace industry for over 30 years. His clients include Army, Navy, Air Force, NASA, Siemens, Eaton, Boeing, Lockheed, Northrup Grumman, General Motors, Prior to becoming a consultant in 1982 he worked at GE as Supervisor of Quality Assurance/Manager of Manufacturing Engineering, at Cooper Industries as Chief Engineer, and at Booz-Allen & Hamilton as Risk Management consultant for a variety of industries.
He teaches Design for Reliability courses at the University of Maryland for degree programs in Mechanical Engineering and Reliability Engineering. He is a Fellow of the American Society for Quality and recipient of its Austin Bonis Award for Reliability Education Advancement, and former chair of the Reliability Division. He is a Senior Member of IEEE. He is a former National Malcolm Baldrige Quality Award Examiner in the first batch of examiners. He served as Vice president of the International System Safety Society where he received the Scientific Achievement Award and the Educator-of-the-Year Award. He served on the Board of Directors for the Annual Reliability and Maintainability Symposium for more than 10 years.
The purpose of verification and validation is to hunt for oversights and omissions in design, manufacturing, and servicing. Who is verifying and who is validating is a very important consideration. An independent, unbiased, and one with a full understanding of risk analysis would be an ideal participant. This is not always viable. The next best thing to then is to have a small team (5 to 7 people) to get a refresher training in risk management, to define and agree upon the verification and validation goals.
There is a difference between verification and validation. Verification is about confirming that high-quality work was done right, at right time, by qualified verifiers. If those who write specifications are not trained in writing specifications based on risk assessment, they are likely to leave out many requirements. One safeguard against too many oversights and omissions is to develop an internal checklist and let the independent reviewer add more to it from his/her personal experience. Below are some questions that should be on the checklist.
- Were the inputs from users addressed?
- Was the search made on adverse events on similar devices on the Maude database?
- Was the right intent of PHA, FMEA, and FTA was evident in these documents?
- Are the results of the PHA implemented in the specification?
- Are the results of the Functional FMEA implemented in the specification?
- Are the software risks analyzed and mitigated?
- Is reliability specified including the time duration and worst-case environment?
- Is durability specified in terms of minimum life?
- Are fail-safe requirements addressed?
- Is the device designed to isolate faults and inform users what is the problem?
- Has redundancy been considered to avoid critical failures and malfunctions?
- Is reliability centered maintenance specified for complex systems (MRI, nuclear medicine)?
- Design Validation Testing for Reliability
- Design Validation Testing for Durability
- Design Validation for Safety
- Using Field Validation to Identify New Risks
- Overview of the plan
- Immediate recall coordination
- Review of the discovered risks
- Review of data management
- Verification of activities for effectiveness
- Closing the recall
Course Level - All levels including senior management
Who Should Attend
- All Technical Managers
- All Engineers
- Quality Assurance Staff
- Safety Staff
- Marketing Staff
Why Should You Attend
Many companies have so-called Final Design Review. Some call it the Critical Design Review. This is the last chance to review if anyone is concerned about anything in the design. A very important heuristic to remember is “Learn to Say No to Yes Men.” The context here is that if the final design is presented and everyone votes “yes” to the approval, then your answer should be “no.” Why? Because there are almost always new problems lingering in the minds of the team members but they don’t speak thinking it is too late to interfere. If no one is challenging the design, the device is bound to crash in use, unless the device is very simple. No matter how good the design is, an independent facilitator can find many issues with it. Ford Motor Company hired a new Vice President during the design of the 1995 model of the Lincoln Continental car. The company has been making this car for years and everyone on the team had at least ten years of experience. The design was already approved but he insisted on questioning every detail of the design with a cross-functional team made up of engineers from each subsystem. He included the marketing manager and service engineers. They made over 700 changes in the design! Since they made these improvements while the design was still not in manufacturing, they saved 60 million dollars (the potential cost of making engineering changes in production, rework, and reducing inspectors). The new product was released four months ahead of schedule!
In software engineering, software trustworthiness delivers a complete assurance that it will execute its required functions under all probable situations, will do so on time, and will never implement any activities that have the significance of security risk in software for a specific time duration.