How to Make MS Access Programs Part 11 Compliant

Write Better!
Best Practices in Writing
Requirements and Test Cases
Tyson M. Mew, President
Ofni Systems Inc.
About the Speaker
Ty Mew is President of Ofni Systems Inc., a
provider of Part 11 software solutions and
computer validation services. He has 19 years of
experience in software development and computer
system validation, and is the creator of both the
ExcelSafe and FastVal platforms.
In his spare time he trains companies on how to
properly implement Part 11 compliant systems, how
to test software, and how to automate tasks that
should be done by software.
April 27, 2016
2
Outline
Make better use of Validation Master Plans
Understand when more or less documentation
and testing may be needed
Identify and correct bottlenecks in your process
Speeding up the process by switching to
paperless validations
Using style guidelines
April 27, 2016
4
Project & Process Definition
April 27, 2016
5
Getting Started
Formalize your Documentation
Practices
Preparing templates for documents
and certain types of systems
Determine and execute your testing
strategy
Formalizing Validation Strategy
Create a General Strategy for All Projects:
• Identify Validation Tactics/Testing Approaches for
Common Elements
– Data Inputs
– Formulas, Calculations
– Reports, Charts
• Define Strategies for Specific Software Types
Validation Strategy Documented in Validation
Master Plan
Spreadsheet Validation Master Plan
April 27, 2016
8
Benefits of Validation Master Plans
Unifies Validation Approach
Ensures Internal Agreement with Validation
Strategy
Improves Training of Validation Professionals
Facilitates Review/Approval Process
Provides chance to implement best practices for
lean project management
Validation Templates
Templates Improve document consistency and
efficiency of document creation
Templates Include:
•
•
•
•
SOP identified Sections (Objectives, Scope, etc.)
Instructions for completing specific sections
Template text with variable names
Accepted Formatting
Hint: Fewer templates are better than more templates.
Use variables to create constant text.
Examples of Variable Names
System Name
Program Version Number
File Name, Data File Name, Security File Name,
External Code Library Name
File Location
Company Name, Company Address
Owning Department, System Owner
Document Names, ID#s, Abbreviations, Titles
Protocol Names, Abbreviation
Breaking Down Systems into
Discrete Validatable Objects
Define Distinct Parts of the Program
• Forms, Reports
• Worksheets, Charts
• Workflows
Automatically Create List of Validatable Objects
Hint: Use software to analyze programs, record logical order
for forms and reports.
Collecting Requirements
Define requirements during project walk-through
Additional requirements can be gathered from key end
users, developers, system owners.
Note: GAMP 5 specifically allows combining/separating
of specification documents to facilitate validation.
Generating Design Specifications
Once requirements are defined, use software
tools to analyze objects and produce suitable
output
Only applicable to open source code.
Hint: Use a program to analyze common objects
and produce suitable content for the Design
Specification.
Example of Generating
Design Specifications: Data Input Form
Example of Automating
Spreadsheet Validation
April 27, 2016
16
Examples of Generating Design
Specifications: Spreadsheet Calculations
Determining Testing Strategy
Determine what you are going to do, then
execute your plan
Validation Professionals test according to
plan
Quality review/approve documents based
on existing standards
Define Testing for Each Object
Forms: Input, Processing, Output, Security
Reports: Data, Calculations, Formatting
Spreadsheets: Data Entry, Calculations, Data
Outputs, Labels
Charts: Data Ranges, Calculations,
Labels/Formatting
Workflows & Process Flow
Security Roles, including configuration settings
Test Cases and Test Plans
Testing based on Risk Assessments
Input Testing vs. Challenge Testing,
Workflow Testing
Automating with micro-test cases
Auto-creation of test scripts from your
requirements
Create specific test cases for use with change
control
How Much Is Enough?
Automating Testing allows more time for Testing,
Less Time spent on Documentation.
Defining specific requirements based on Risk allows
for use of different templates based on specific
functional needs.
Automated creation of Test Cases/Protocols allows
validation resources to focus on high risk sections of
program functionality
Required business functionality should receive special
attention.
Input vs. Challenge Testing
High Risk Fields Should have Challenge Tests
Status Fields
• Do not accept <NULL>
• If Drop-Down List Used, Status not accept other values
• Default Value should not be Pass, Success, etc.
Dates
• Do not accept clearly incorrect dates
• Do not accept unordered dates (Date of Test before Date
Received, etc.)
Workflow Testing
Extremely useful technique for systems with a
defined process or workflow (Sample
Management, Manufacturing Process, etc.)
• Verify that entered data can move through entire
workflow
• Verify that fail-points prevent data from moving
forward
• Verify that data cannot be accessed from
inappropriate parts of the workflow
Using Micro-Test Cases
Micro-Test Cases are test cases for common
requirements have pre-written verbiage (with
variables) which is inserted into appropriate
sections of the document.
Parallel sections inserted into Requirement
Documents, Design Documents and Testing
Documents
Smaller test cases combined into larger test
cases
Using Micro-Test Cases
Micro-Test Cases most useful for common, repeated
test steps.
•
•
•
•
•
•
Adding/Closing Forms or Reports
Data Entry into Fields
Screen Navigation
Printing, Generating Charts, Etc.
Confirmation of Formulas/Calculations
Any place in Test Protocol where similar operations are
repeated with similar results
Content of Micro-Test case modified with variables
Automatic Generation of Test Protocols
In certain systems, software can be used to
automate the generation of entire test
protocols based on defined inputs
Data Entry Screens
Individual Spreadsheet Worksheets
Security Testing
Automatic Generation of Test Protocols
Test Cases for Change Control
Certain Test Cases Are Particularly Useful
when verifying functionality after Change
Control
• Regression Testing
• Functionality Identified as High Risk
Wherever possible, Test Cases should be
written mindful of reuse during Change Control
Protocol Execution, Documentation and
Managing Deviations
Paper vs. Electronic execution
Capturing actual results and screen shots for
maximum effectiveness
Generating, processing and closing test and
script deviations
Identify Bottlenecks
Identify and correct bottlenecks in your process
April 27, 2016
30
Identify and Correct Bottlenecks
Identify Bottlenecks
• Look for slow areas in the overall validation process
Common Bottlenecks
•
•
•
•
•
Writing Documents
Executing Protocols
QA Review
QA Reviews
More QA Reviews
April 27, 2016
31
Tracking Edits for
Each Section of a Document
Paperless Validations
We have done exactly 1 protocol execution on
paper over the last 10 years.
• It was for a protocol that we have executed at least
50 times in the past
• It takes exactly 4 days to execute electronically
• On paper, it took 10 days
April 27, 2016
33
Paper vs. Electronic Execution
 Electronic Protocol Execution
• Replaces traditional pen and paper techniques
• Just as word processors improve writing, electronic
protocol execution improves paper executions
 Documents can be exported and reviewed by
traditional means (“Type-writer Rule”)
Note: Electronic Protocol Software must be
compliant with 21 CFR 11, with audit trails,
security, electronic signatures, etc.
Example of Electronic
Protocol Execution
Documenting Protocol Execution
with Screen Shots
 As each step is completed, collect screen shot
 Insert Screen shots automatically into finished protocol
 Configure Screen shots to include user name, date/time, etc.
which further document testing
Hint: We use screen shot software, integrated with Electronic
Protocol Execution software. This significantly reduces time
organizing and numbering screen shots.
NOTE: Screen shots do not “prove” a test case was successful,
only provides evidence that a test was performed.
Documenting Protocol Execution
with Screen Shots
Write Better
Teach people how to write documents and test
cases quickly and efficiently
Encourage input from the entire team
April 27, 2016
38
General Rules
April 27, 2016
39
Requirements
April 27, 2016
40
Test Cases
April 27, 2016
41
My Advice for You
Start your own Style Guidelines
Fill it with the knowledge you have learned
Let everyone review it & add to it
Use it to Write Better & Improve your entire
validation department
Save the World from
Bad Validation Documents!!!
April 27, 2016
42
Thank You!
Tyson M. Mew
(919) 844-2494
Ofni Systems Inc.
[email protected]
www.OfniSystems.com
References and additional materials will be available at
www.OfniSystems.com at the conclusion of the conference.
April 27, 2016
43
Additional
Reference
Material
Validation Status Update Reports
Validation Master Plans
• Validation Master Plans discuss validation activities
across an entire site. Validation Master Plans can
outline documents required in validation project and
general goals and strategies for all validation projects.
Having a validation master plan may dictate that
validation plans are not necessary for each validation
project.
• Validation Master Plans should be approved by the
Head of Site Quality, plus other department heads as
appropriate.
Validation Master Plans
Validation Master Plan outlines a sites
validation/qualification principles.
• Defines areas and systems to be validated
• Defines broad plan to achieve validation
• May be used to define how software testing should
be conducted, in both general terms and for specific
types of systems
Validation Plans
• Validation Plans define the scope and goals of a validation
project. Validation plans are written before a validation project
and are specific to a single validation project.
• Validation Plans can include:
 Deliverables (Documents) to be generated during the validation process
 Resources/Departments/Personnel to participate in the validation
project
 Time-Line for completing the validation project
• Validation Plans should be approved by the System Owner and
Quality Assurance. The collection of documents produced
during a validation project is called a Validation Package. Once
the validation project is complete, all validation package should
be stored according to your site document rules.
Policies, SOPs and Templates
 Policies
• High level directives
• Example: Validation Master Plan
 Procedures
• Specific directions for Computer System Life Cycle
• Specific directions for Validation documents
• Example: “Software Validation Testing”
 Templates
• Provide Outlines for specific validation documents
• Not required but recommended
• Example: “Template for Functional Requirement Specification”
Gathering Requirements
Interviews with:
• Key System End-Users
• System Owner
• Developer
Good communication between the validation team
and all end users improves all aspects of the
project and adds more value to the project.
Gathering Requirements
Screen Specific
• Input tests
– Make sure that all fields accept appropriate
data entry
– Use challenge tests where appropriate
• Processing
– Make sure all functions (buttons, etc.) operate as
expected
– Make sure all calculations produce correct results
Gathering Requirements
Workflow Specific
• Follow specific data entry throughout the workflow.
• Follow data transitions in workflows
Security
•
•
•
•
Define specific groups
Define specific workflows available
Define specific functionality
Test what users can and cannot do
Automated Document Generation
Extension of Templates
Includes specific verbiage with variables.
Variables are replaced with specific data.
Requirement lists automatically inserted into
appropriate sections of validation documents
 This approach works well with similar
programs with similar requirements (Groups of
spreadsheets, etc.)
Automate the Generation of
CSV Documents
Breaking down systems into discrete
validatable objects
Collecting testable requirements
Integrating Risk Assessments into the process
Generating Design Specifications directly from
source code
Automatic generation, tracking and updating of
the Requirements Traceability Matrix
Micro Test Cases
April 27, 2016
55
Paperless Validation
April 27, 2016
56
Deviations
April 27, 2016
57
Automating Deviation Reporting
When a test step fails, deviation is
automatically created
User identifies deviation type, based on this
determination, some fields are automatically
filled.
Deviations can be viewed in Real-Time
Deviations can be resolved according to each
companies procedures
Risk Assessment
April 27, 2016
60
Integrated Risk Assessment
Validation Resources are limited; a Risk-Based
Validation approach is necessary to focus limited
resources
Risk Assessment is performed at the requirement level
• Functional Specification
• Design Specification
Ask Developers/Key System Users/System Owners
where system risk is. They know and you usually get a
better ROI than trying to analyze every requirement.
Risk Assessment
Determine what Sections of the Document
create the most Risk and focus testing there.
Risk Assessment should be multi-disciplinary,
and should include validation professionals, key
end users, Quality review, developers and
system owner.
Once Risk Assessment complete, the Ideally
documented in a document included in the
validation package.
Common Solutions
April 27, 2016
63
Solutions for Common Software
Web Based Applications
Custom Databases (MS Access, FoxPro, SQL
Server, Oracle, etc.)
Spreadsheets (MS Excel, Quattro, Lotus, etc.)
Connections to LIMS
In addition to system specific issues, all systems
must demonstrate control of data, and meet any
applicable business requirements.
Web Based Applications
Define System Interfaces
• Input Screens
• Processing Screens
• Report Outputs
Define Key System Functions/Computations
Define Key Business Workflows
Define Security Requirements
Make sure all functions are tested
Custom Databases
Access, FoxPro, SQL Server, Oracle, etc.
Define System Interfaces
• Input Screens
• Processing Screens
• Report Outputs
Define Key System Functions/Computations
Define Key Business Workflows
Define Security Requirements
April 27, 2016
66
Spreadsheets
Excel, Quattro, Lotus, etc.
Define Data Input Cells/Ranges
Define Spreadsheet Processing
• Calculation Cells
• Internal Functions
Define System Outputs/Charts
Define Security Requirements
Case Studies
April 27, 2016
68
Validation Case Study Reviews
Ensure compliance with 21 CFR 11
• Technological
• Procedural
Risk Assessment
Gather Requirements
• Screen Specific
• Workflow Specific
• Security
Write Requirement Documents
Testing Documents
April 27, 2016
69
Case Studies
Spreadsheets for CRO
• Applied tools to spreadsheets to provide
technological controls, including audit trails, user
level security and electronic signatures
• Spreadsheet requirements generated from micro
templates.
• Requirements verified through protocol execution
Result: Entire spreadsheet library quickly and
efficiently validated
Case Studies
Engineering system for Pharmaceutical Manufacture
• Requirements gathered from key users and developers
• Software emphasized moving through exact workflows, so
testing plan modified to focus on workflows over data entry
screens with associated reports
• Requirements verified through protocol execution
Result: Emphasized workflows over input/output model
when appropriate, which improved testing and overall
validation effort.
Case Studies
Electronic Document Capture System
• Worked with developers to demonstrate compliance with 21
CFR 11
• Program used Client/Server model, with digital pen inputs
• Extensive use of virtual machines to model each section of
program – different OS, different sections
• Tested specific form events
• Requirements verified through protocol execution
Results: Validation effort allowed quick and efficient
compliance with 21 CFR Part 11 regulations. After the
validation process was complete, the validation results
were reviewed and approved by a third party auditor
Case Studies
Purchasing System for Pharmaceutical
Manufacturer
• Requirements gathered from company SOP and
instructions
• Worked with 3rd party vendor to add security
• Met with both End users and developers
• Requirements verified through protocol execution
Result: Third party software made compliant with
21 CFR 11, with implemented Security requirements
Case Studies
Clinical Research Database from CRO
• Clear requirements, no requirement drift
• Used existing templates, validation document
generation and electronic protocol execution
• Requirements verified through protocol execution
Result: Validation completed in 12 business days.
Automatic Requirement Traceability
Several Programs exist which will
automatically generate the Requirement
Traceability Matrix
Updates if Requirements are Moved/Edited
Displays if Functional Requirements exists
without corresponding Design specification
Displays if Requirements exist which do not
have a defined Test Case
<END>
April 27, 2016
77