Report - Drummond Group

Transcription

Test Results Summary for 2014 Edition EHR Certification
Version EHR-Test-144 Rev 01-Aug-2014
ONC HIT Certification Program
Part 1: Product and Developer Information
1.1 Certified Product Information
Product Name:
Crib Notes
Product Version:
3.0
Domain:
Inpatient
Test Type:
Modular EHR
1.2 Developer/Vendor Information
Developer/Vendor Name:
Grand Rounds Software
Address:
404 Valley Glen Drive Bryn Mawr PA 19010
Website:
www.cribnotes.com
Email:
[email protected]
Phone:
800-323-9167
Developer/Vendor Contact:
Michael Paulsen
Pa ge 1 of 11
Part 2: ONC-Authorized Certification Body Information
2.1 ONC-Authorized Certification Body Information
ONC-ACB Name:
Drummond Group
Address:
13359 North Hwy 183, Ste B-406-238, Austin, TX 78750
Website:
www.drummondgroup.com
Email:
[email protected]
Phone:
817-294-7339
ONC-ACB Contact:
Bill Smith
This test results summary is approved for public release by the following ONC-Authorized Certification
Body Representative:
Bill Smith
ONC-ACB Authorized Representative
Certification Committee Chair
Function/Title
8/18/2014
Signature and Date
2.2 Gap Certification
The following identifies criterion or criteria certified via gap certification
§170.314
(a)(1)
x
(a)(6)
x
(a)(17)
x
(d)(5)
(d)(9)
(b)(5)*
x
(d)(6)
(f)(1)
(a)(7)
x (d)(1)
x (d)(8)
*Gap certification allowed for Inpatient setting only
No gap certification
Pa ge 2 of 11
2.3 Inherited Certification
The following identifies criterion or criteria certified via inherited certification
§170.314
(a)(1)
(a)(14)
(c)(3)
(f)(1)
(a)(2)
(a)(15)
(d)(1)
(f)(2)
(a)(3)
(a)(16) Inpt. only
(d)(2)
(f)(3)
(a)(4)
(a)(17) Inpt. only
(d)(3)
(f)(4) Inpt. only
(a)(5)
(b)(1)
(d)(4)
(a)(6)
(b)(2)
(d)(5)
(f)(5) Optional &
Amb. only
(a)(7)
(b)(3)
(d)(6)
(a)(8)
(b)(4)
(d)(7)
(f)(6) Optional &
Amb. only
(a)(9)
(b)(5)
(d)(8)
(g)(1)
(a)(10)
(b)(6) Inpt. only
(d)(9) Optional
(g)(2)
(a)(11)
(b)(7)
(e)(1)
(g)(3)
(a)(12)
(c)(1)
(e)(2) Amb. only
(g)(4)
(a)(13)
(c)(2)
(e)(3) Amb. only
x No inherited certification
Pa ge 3 of 11
Part 3: NVLAP-Accredited Testing Laboratory Information
Report Number: GI-08042014-2085-A
Test Date(s): 7/23/2014; 8/4/2014
3.1 NVLAP-Accredited Testing Laboratory Information
ATL Name:
Drummond Group EHR Test Lab
Accreditation Number:
NVLAP Lab Code 200979-0
Address:
13359 North Hwy 183, Ste B-406-238, Austin, TX 78750
Website:
www.drummondgroup.com
Email:
[email protected]
Phone:
512-335-5606
ATL Contact:
Beth Morrow
For more information on scope of accreditation, please reference NVLAP Lab Code 200979-0.
Part 3 of this test results summary is approved for public release by the following Accredited Testing
Laboratory Representative:
Gary Isaac
ATL Authorized Representative
Signature and Date
8/18/2014
Test Proctor
Function/Title
Sarasota, FL
Location Where Test Conducted
3.2 Test Information
3.2.1 Additional Software Relied Upon for Certification
Additional Software
Applicable Criteria
Functionality provided
by Additional Software
Pa ge 4 of 11
Additional Software
Applicable Criteria
Functionality provided
by Additional Software
x No additional software required
3.2.2 Test Tools
Version
Test Tool
Cypress
2.4.1
ePrescribing Validation Tool
1.0.4
HL7 CDA Cancer Registry Reporting Validation Tool
1.0.3
HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool
1.8
HL7 v2 Immunization Information System (IIS) Reporting Validation
Tool
1.8
HL7 v2 Laboratory Results Interface (LRI) Validation Tool
1.7
HL7 v2 Syndromic Surveillance Reporting Validation Tool
1.7
x
Transport Testing Tool
179
x
Direct Certificate Discovery Tool
3.0.2
x
No test tools required
3.2.3 Test Data
Alteration (customization) to the test data was necessary and is described in
Appendix [insert appendix letter]
No alteration (customization) to the test data was necessary
3.2.4 Standards
3.2.4.1 Multiple Standards Permitted
The following identifies the standard(s) that has been successfully tested
where more than one standard is permitted
Criterion #
Standard Successfully Tested
§170.204(b)(1)
(a)(8)(ii)(A)(2)
HL7 Version 3 Implementation
Guide: URL-Based
Implementations of the
Context-Aware Information
Retrieval (Infobutton) Domain
§170.204(b)(2)
Guide: Context-Aware
Knowledge Retrieval
(Infobutton) Service-Oriented
Architecture Implementation
Guide
Pa ge 5 of 11
Criterion #
x
(a)(13)
§170.207(a)(3)
IHTSDO SNOMED CT®
International Release July
2012 and US Extension to
SNOMED CT® March 2012
Release
§170.204(b)(1)
(a)(15)(i)
(a)(16)(ii)
Guide: URL-Based
Implementations of the
Context-Aware Information
Retrieval (Infobutton) Domain
§170.204(b)(2)
Guide: Context-Aware
Knowledge Retrieval
(Infobutton) Service-Oriented
Architecture Implementation
Guide
§170. 210(g)
Network Time Protocol
Version 3 (RFC 1305)
§170.207(i)
The code set specified at 45
CFR 162.1002(c)(2) (ICD-10CM) for the indicated
conditions
§170.207(i)
(b)(7)(i)
HL7 Version 3 Standard:
Clinical Genomics; Pedigree
§170.210(g)
x
(b)(2)(i)(A)
§170.207(j)
CFR 162.1002(c)(2) (ICD-10CM) for the indicated
conditions
§170.207(a)(3)
IHTSDO SNOMED CT®
Release
§170.207(a)(3)
IHTSDO SNOMED CT®
Release
Annex A of the FIPS Publication 140-2
(e)(1)(i)
[list encryption and hashing algorithms]
§170.210(g)
(e)(1)(ii)(A)(2) Network Time Protocol
§170. 210(g)
Annex A of the FIPS Publication 140-2
(e)(3)(ii)
[list encryption and hashing algorithms]
x
Common MU
Data Set (15)
§170.207(a)(3)
IHTSDO SNOMED CT®
Release
§170.207(b)(2)
CFR 162.1002(a)(5) (HCPCS
and CPT-4)
Pa ge 6 of 11
Criterion #
None of the criteria and corresponding standards listed above are
applicable
3.2.4.2 Newer Versions of Standards
The following identifies the newer version of a minimum standard(s) that
has been successfully tested
Newer Version
Applicable Criteria
No newer version of a minimum standard was tested
3.2.5 Optional Functionality
Criterion #
x (a)(4)(iii)
Optional Functionality Successfully Tested
Plot and display growth charts
(b)(1)(i)(B)
Receive summary care record using the standards specified at
§170.202(a) and (b) (Direct and XDM Validation)
(b)(1)(i)(C)
Receive summary care record using the standards specified at
§170.202(b) and (c) (SOAP Protocols)
(b)(2)(ii)(B)
Transmit health information to a Third Party using the standards
specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(2)(ii)(C)
Transmit health information to a Third Party using the standards
specified at §170.202(b) and (c) (SOAP Protocols)
(f)(3)
Ambulatory setting only – Create syndrome-based public health
surveillance information for transmission using the standard
specified at §170.205(d)(3) (urgent care visit scenario)
Common MU
Data Set (15)
Express Procedures according to the standard specified at
§170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures
and Nomenclature)
Common MU
Data Set (15)
Express Procedures according to the standard specified at
§170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)
No optional functionality tested
Pa ge 7 of 11
3.2.6 2014 Edition Certification Criteria* Successfully Tested
Criteria #
Version
TP** TD***
Criteria #
Version
TP
TD
(c)(3)
1.6
(d)(1)
1.2
(a)(1)
1.2
(a)(2)
1.2
x
(a)(3)
1.2
1.4
x
(d)(2)
1.5
x
(a)(4)
1.4
1.3
x
(d)(3)
1.3
x
(a)(5)
1.4
1.3
(d)(4)
1.3
(a)(6)
1.3
1.4
(d)(5)
1.2
(a)(7)
1.3
1.3
(d)(6)
1.2
(a)(8)
1.2
(d)(7)
1.2
(a)(9)
1.3
1.3
(d)(8)
1.2
(a)(10)
1.2
1.4
(d)(9) Optional
1.2
(a)(11)
1.3
(e)(1)
1.8
1.5
(a)(12)
1.3
(e)(2) Amb. only
1.2
1.6
(a)(13)
1.2
(e)(3) Amb. only
1.3
(a)(14)
1.2
x
(f)(1)
1.2
1.2
(a)(15)
1.5
x
(f)(2)
1.3
1.7.1
(a)(16) Inpt. only
1.3
(f)(3)
1.3
1.7
(a)(17) Inpt. only
1.2
(f)(4) Inpt. only
1.3
1.7
(b)(1)
1.7
1.4
(b)(2)
1.4
1.6
(f)(5) Optional &
Amb. only
1.2
1.2
(b)(3)
1.4
1.2
(b)(4)
1.3
1.4
(f)(6) Optional &
Amb. only
1.3
1.0.3
(b)(5)
1.4
1.7
(g)(1)
1.7
1.9
(b)(6) Inpt. only
1.3
1.7
(g)(2)
1.7
1.9
(b)(7)
1.4
1.6
x
(g)(3)
1.3
(c)(1)
1.6
1.6
x
(g)(4)
1.2
(c)(2)
1.6
1.6
x
x
x
x
1.5
x
1.2
x
1.6
No criteria tested
*For a list of the 2014 Edition Certification Criteria, please reference
http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)
**Indicates the version number for the Test Procedure (TP)
***Indicates the version number for the Test Data (TD)
Pa ge 8 of 11
3.2.7 2014 Clinical Quality Measures*
Type of Clinical Quality Measures Successfully Tested:
Ambulatory
Inpatient
x No CQMs tested
*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov
(navigation: 2014 Clinical Quality Measures)
CMS ID
Ambulatory CQMs
Version CMS ID
2
90
136
155
22
117
137
156
50
122
138
157
52
123
139
158
56
124
140
159
61
125
141
160
62
126
142
161
64
127
143
163
65
128
144
164
66
129
145
165
68
130
146
166
69
131
147
167
74
132
148
169
75
133
149
177
77
134
153
179
82
135
154
182
CMS ID
Inpatient CQMs
Version CMS ID
9
71
107
172
26
72
108
178
30
73
109
185
31
91
110
188
32
100
111
190
53
102
113
55
104
114
60
105
171
CMS ID
CMS ID
Version
Version
Version
Version
CMS ID
CMS ID
Version
Version
Pa ge 9 of 11
3.2.8 Automated Numerator Recording and Measure Calculation
3.2.8.1 Automated Numerator Recording
Automated Numerator Recording Successfully Tested
(a)(1)
x
(a)(9)
(a)(16)
(b)(6)
(a)(17)
(e)(1)
(b)(2)
(e)(2)
(a)(13)
(b)(3)
(e)(3)
x
(a)(3)
(a)(11)
x
(a)(4)
(a)(12)
x
(a)(5)
x
(a)(6)
(a)(14)
(b)(4)
x
(a)(7)
(a)(15)
(b)(5)
x
x
Automated Numerator Recording was not tested
3.2.8.2 Automated Measure Calculation
Automated Measure Calculation Successfully Tested
(a)(1)
(a)(9)
(a)(16)
(b)(6)
(a)(3)
(a)(11)
(a)(17)
(e)(1)
(a)(4)
(a)(12)
(b)(2)
(e)(2)
(a)(5)
(a)(13)
(b)(3)
(e)(3)
(a)(6)
(a)(14)
(b)(4)
(a)(7)
(a)(15)
(b)(5)
x Automated Measure Calculation was not tested
3.2.9 Attestation
Attestation Forms (as applicable)
Appendix
x Safety-Enhanced Design*
A
x Quality Management System**
B
x Privacy and Security
C
*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16),
(b)(3), (b)(4)
**Required for every EHR product
3.3 Appendices
Attached below.
Pa ge 10 of 11
Test Results Summary Change History
Test Report ID
GI-08042014-2085
Description of Change
Add missing a7 module that was tested
Date
18Aug2014
2014 Edition Test Report Summary
Pa ge 11 of 11
USER CENTER DESIGN REPORT –
TEST REPORT UPDATE
This test report was updated in December 2015 to satisfy User Center Design Report specifications by
ONC.
The new Test Report ID is amended as follows:
“Part 3: NVLAP-Accredited Testing Laboratory Information: Report Number” plus the suffix “_Dec2015”.
Grand Rounds T M Software
Grand Rounds Software, LLC
TM
CribNotes Version 3.0
Michael Paulsen
Vice President Sales / Business Development
[email protected]
(816)246-5496
For public release:
Grand Rounds Software, LLC used the following usability design in developing and designing their HIT
system - NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records
TM
(7741) in developing CribNotes Version 3.0.
Michael A. Paulsen
December 30, 2015
July 9, 2014
Drummond Group Inc.
13359 North Hwy 183
Suite B-406-238
Austin, TX 78750
To Whom It May Concern:
This letter is to affirm that the documentation submitted to the Drummond Group
pertaining to Meaningful Use Certification Criteria 170.314.g.3 – Safety Enhanced
Design Practices is accurate and authentic.
The completed Usability Testing Report is accurate and authentic and represents the
results of Usability Evaluation Sessions on the following Certification Criteria:


170.314(a)(6) Medication list
170.314(a)(7) Medication allergy list
Robert L. Stavis, M.D., Ph.D., President
Phone: 610-812-5508
Advancing Newborn Care
www.cribnotes.com
800-323-9167
EHR Usability Test Report
Crib Notes
Version 3.0
Report based on NISTIR 7742 Template for Electronic Health Record Usability Testing as
Adapted from Customized Common Industry Format for Usability Test Report Template
Date of Usability Test:
July 8, 2014
Date of Report:
July 9, 2014
Report Prepared By:
Phyllis Barnett, Director of Professional Services
© Grand Rounds Software, 2014
Page 1 of 29
Table of Contents
Executive Summary ....................................................................................................................2
Introduction ................................................................................................................................4
Method .......................................................................................................................................4
Participants ..............................................................................................................................4
Study Design ...........................................................................................................................5
Tasks .......................................................................................................................................5
Procedures ...............................................................................................................................6
Test Environment.....................................................................................................................6
Test Forms and Tools ...............................................................................................................6
Participant Instructions ............................................................................................................7
Usability Metrics .....................................................................................................................7
Data Scoring ............................................................................................................................8
Results – Medication List .......................................................................................................... 10
Data Analysis and Reporting .................................................................................................. 10
Discussion of Findings ........................................................................................................... 10
Results – Medication/Allergy List ............................................................................................. 11
Data Analysis and Reporting .................................................................................................. 11
Discussion of Findings ........................................................................................................... 11
Administrator’s Guide ............................................................................................................... 12
Appendices ............................................................................................................................... 24
Non-Disclosure Agreement .................................................................................................... 25
Informed Consent .................................................................................................................. 26
Incentive Receipt and Acknowledgement Form ..................................................................... 27
System Usability Scale Questionnaire .................................................................................... 28
Page 2 of 29
Executive Summary
A usability test of Crib Notes Version 3.0 was conducted on July 8th 2014 in the NICU
Conference Room at Bryn Mawr Hospital, Bryn Mawr, Pennsylvania by Grands Rounds
Software staff. The purpose of these sessions was to test and validate the usability of the current
user interface and provide evidence of usability in the Electronic Health Records Under Test
(EHRUT). During the usability sessions, 6 healthcare workers matching target demographic
users served as participants and used the EHRUT in simulated, but representative tasks.
This study collected performance data on several tasks typically performed for the following
EHR certification criteria:
• Medication List (170.314.a.6) - add, edit, make inactive, view medications from list
• Medication/Allergy List (170.314.a.7) - add, edit, make inactive, view allergies from list
During the 15 minute usability test, each participant was greeted by the administrator and asked
to review and sign an informed consent/release form (blank form included in Appendix); they
were instructed that they could withdraw at any time. All the participants had prior experience
with Crib Notes being current users of the software at their health system. Participants were
given a verbal overview of the functionality prior to the start of the test. They were also given a
job aid that was typical of what would be used in a real situation. The administrator introduced
the test and instructed participants to complete a series of tasks (given one at a time) using the
EHRUT. During the testing, the administrator timed the test and along with the data logger,
recorded user performance data on paper. The administrator did not give the participants
assistance on how to complete the task but they did instruct the participants on which patients to
use and what medications and allergies to add and edit.
The following types of data were collected for each participant:
•
•
•
•
•
Time to complete the task(s)
Number and types of errors
Path deviations
Participant’s verbalizations
Participant’s satisfaction ratings of the system
Participant data was de-identified – no correspondence could be made from the identity of the
participant to the data collected. Following the conclusion of the testing, participants were asked
to complete a post-test questionnaire and were compensated with a gift card for their time.
Various recommended metrics, in accordance with the examples set forth in the NIST Guide to
the Processes Approach for Improving the Usability of Electronic Health Records, were used to
evaluate the usability of the EHRUT. Following is a summary of the performance and rating
data collected on the EHRUT for both criteria – Medication List and Medication/Allergy List.
Page 3 of 29
Medication
List
N
Tasks
#
Record Medication
Change Medication
Access Medication
3
3
6
Task
Risk
5=None
Task
Success
Task
Errors
Optimal
Path
Path
Deviations
Observed/
Optimal
2
1
4
Med/Allergy
List
N
Tasks
#
Record Med/Allergy
Change Med/Allergy
Access Med/Allergy
3
3
6
100%
100%
100%
Task
Risk
5=None
Task
Success
0%
0%
0%
Task
Errors
100%
100%
100%
0%
0%
0%
Task
Time
Task Time
Deviations
Mean
(sec)
Observed/
Optimal
Task
Rating
5=Easy
Mean
5
5
3
1.2
1
1
25
20
15
33
26
18
1.32
1.3
1.2
5
5
5
Optimal
Path
Path
Deviations
Optimal
Task
Time
Seconds
Task
Time
Task Time
Deviations
Mean
(sec)
Observed/
Optimal
Task
Rating
5=Easy
Mean
45
35
20
1.5
1.4
1.3
Observed/
Optimal
2
1
4
Optimal
Task
Time
Seconds
6
6
3
1.5
1.3
1.3
30
25
15
The results from the System Usability Scale scored the subjective satisfaction with the
functionalities based on performance with these tasks to be: Excellent
In addition to the performance data, the following is a summary of the most common qualitative
observations and comments that were made:
- Major findings:
 Participants have been using Medication List functionality for several years and were
very satisfied and comfortable with the functionality.
 Several of the participants questioned the need for medication-allergy recording for this
patient population (neonates) but understood that the requirement is needed for EHR’s in
general.
 Deviations were higher with the Medication Allergy functionality as none of the
participants had ever used the functionality prior to these usability testing sessions (it is
new to this version). None-the-less they expressed satisfaction and noted how it was very
intuitive, consistent with existing functionality and integrated well with the Medication
List functionality.
- Areas for improvement:
 Participants did not indicate or suggest any areas that needed improvement.
5
5
4.5
Page 4 of 29
Introduction
The EHRUTs tested for this study were the Medication List and Medication/Allergy List
functionalities in Crib Notes Version 3.0. Designed to present clinical information to healthcare
providers and workers in neonatal intensive care units, the EHRUTs test how users would enter,
edit and list medications and allergies on selected patients. The usability testing represents
realistic exercises and conditions of how users interact with the software.
The purpose of this study was to test and validate the usability of the current user interface and
provide evidence of usability in the EHRUT. To this end, measures of effectiveness, efficiency
and user satisfaction were logged during the usability testing sessions.
Method
Participants
A total of 6 participants were tested on the EHRUTs. Participants in the test were 3 nurses and 3
clinicians (nurse practitioners and physician). Participants were recruited by Grand Rounds
Software’s Director of Professional Services and were compensated with a gift card for their
time. The participants had no connection to the software vendor organization and in addition,
they had no involvement in developing or conducting the EHRUTs. The participants were all
current and active end-users of Crib Notes. These participants were recruited because they
represented a good sampling of the ‘typical’ Crib Notes user across demographics, roles and
experience.
The following is a table of participants by characteristics, including demographics, professional
experience, computing experience and product experience. Participant names were replaced with
Participant IDs so that an individual’s data cannot be tied back to individual identities.
Experience was self-rated from 1 to 5, ‘1’ being “not experienced” to ‘5’ being “expert.”
Participant
ID
Gender
Age
Education
Occupation/
Role
Computer
Experience
Product
Experience
A
B
C
D
E
F
Male
Female
Female
Female
Female
Female
50-55
45-50
55-60
50-55
45-50
30-35
Post Graduate
Post Graduate
Post Graduate
College Graduate
College Graduate
College Graduate
Clinician
Clinician
Clinician
Nurse
Nurse
Nurse
3
3
2
2
2
3
4
4
2
3
3
4
Page 5 of 29
Participants were individually scheduled for 15 minute sessions with 5-10 minutes at the end for
debriefing by the administrator and data logger and to reset the application to the proper test
conditions.
Study Design
Overall, the objective of this test was to uncover areas where the functionality performed well –
that is, effectively, efficiently and with satisfaction – and areas where the functionality failed to
meet the needs of the participants. This testing served as both a means to record or benchmark
current usability, but also to identify areas where improvements can be made.
During the usability test, participants interacted with Crib Notes. Each participant used the
system in the same location, and was provided with the same verbal instructions and job aids.
The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures
collected and analyzed for each participant:
• Number of tasks successfully completed within the allotted time without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant’s verbalizations (comments)
• Participant’s satisfaction ratings of the system
Tasks
This study collected performance data on tasks typically performed for the following EHR
certification criteria:
• Medication List (170.314.a.6)
• Medication/Allergy List (170.314.a.7)
Tasks were constructed that would be realistic and representative of the kinds of activities a user
might perform with this functionality in Crib Notes, including:
1. Record a medication
2. Change a medication
3. Access (view) medication list
4. Record a medication allergy
5. Change a medication allergy
6. Access (view) medication allergy list
Tasks were selected based on their frequency of use, criticality of function, and those that may be
most troublesome for users. (ie.: associated with the most user errors). Risk associated with each
of the task were also assigned using a scale of 1 to 5 where 1 was the riskiest and 5 represented
no risk.
Page 6 of 29
Procedures
Upon arrival, participants were greeted and their identity verified. Participants were then
assigned a participant ID. Each participant reviewed and signed Non-Disclosure Agreement as
well as an Informed Consent (blank forms included in Appendix). A representative from the test
team witnessed the participant’s signature.
To ensure that the test ran smoothly, two staff members participated in this test, the usability
administrator and the data logger.
The administrator moderated the session including administering instructions and tasks. The
administrator also monitored task times, obtained post-task rating data and took notes on
participant comments. A second person served as the data logger and took notes on task success,
path deviations, number and type of errors, and comments.
Participants were instructed to perform the tasks with the following instructions:
• As quickly as possible making as few errors and deviations as possible.
• Without assistance, administrators were allowed to give clarification on tasks, but not
instructions on use.
• Without using a ‘think aloud’ technique.
For each task, the participants were given verbal instruction of the task. Task timing began once
the administrator finished the verbal instruction. The task time was stopped once the participant
indicated they had successfully completed the task. Following the session, the administrator
gave the participant a post-test questionnaire, (System Usability Scale found in Appendix),
compensated them for their time and thanked each individual for their participation. Participants
signed a receipt and acknowledgement form (blank form included in Appendix) indicating that
they had received the compensation.
Participants' demographic information, task success rate, time on task, errors, deviations, verbal
responses and post-test questionnaire were recorded into a spreadsheet.
Test Environment
The EHRUT was conducted in the NICU Conference Room at Bryn Mawr Hospital, Bryn Mawr
Pennsylvania. For testing, the laptop computer used Windows 7. The laptop monitor was 15
inches x 10 inches with 1024 x 728 resolution. The application was accessed on an RDP
connection into Grand Rounds Software corporate server. A test database was used. The system
performance (i.e., response time) was representative to what actual users would experience in a
field implementation. Additionally, participants were instructed not to change any of the default
system settings. The participants used a mouse and keyboard when interacting with the Crib
Notes application.
Test Forms and Tools
During the usability test, various documents and instruments were used, including:
• Administrators Guide
• Informed Consent
• Non-Disclosure Agreement (NDA)
• Job Aid for Medication List
Page 7 of 29
•
•
•
Job Aid for Medication Allergy List
System Usability Scale Questionnaire for post-test assessment
Incentive Receipt and Acknowledgment Form
The Administrator’s Guide was devised so as to be able to capture required data. Both the
Administrator and Data Logger were in the room during the test and observed the session, with
the Data Logger recording comments and both recording the time it took to completed the tasks.
Participant Instructions
The administrator read the following instructions aloud to the each participant. (see complete
script in Administrator’s Guide)
“Thank you for participating in this study. Your input is very important. Our session
today will last about 15 minutes. During that time you will use an instance of Crib Notes
and I will ask you to complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few errors as possible.
Please try to complete the tasks on your own following the instructions very closely.
Please note that we are not testing you, we are testing the system, therefore if you have
difficulty all this means is that something needs to be improved in the system. I will be
here in case you need specific help, but I am not able to instruct you or provide help in
how to use the application.
Overall, we are interested in how easy (or how difficult) this system is to use, what in it
would be useful to you, and how we could improve it. I did not have any involvement in
its creation, so please be honest with your opinions. All of the information that you
provide will be kept confidential and your name will not be associated with your
comments at any time. Should you feel it necessary you are able to withdraw at any time
during the testing.
For each task, I will read the description to you and say “Begin.” At that point, please
perform the task and say “Done” once you believe you have successfully completed the
task. I would like to request that you not talk aloud or verbalize while you are doing the
tasks.
Participants were then given the tasks to complete. Tasks are listed in the administrator’s guide.
Usability Metrics
According to the NIST Guide to the Processes Approach for Improving the Usability of
Electronic Health Records, EHRs should support a process that provides a high level of usability
for all users. The goal is for users to interact with the system effectively, efficiently, and with an
acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user
satisfaction were captured during the usability testing. The goals of the test were to assess:
• Effectiveness of functionality by measuring participant success rates and errors
• Efficiency of functionality by measuring the average task time and path deviations
• Satisfaction with functionality by measuring ease of use ratings
Page 8 of 29
Data Scoring
The following table details how tasks were scored, errors evaluated and the time data analyzed.
Measures
Effectiveness:
Task Success
Effectiveness:
Task Failures
Efficiency:
Task Deviations
Efficiency:
Task Time
Rationale and Scoring
A task was counted as a “Success” if the participant was able to
achieve the correct outcome, without assistance, within the time
allotted on a per task basis.
The total number of successes were calculated for each task and then
divided by the total number of times that task was attempted. The
results are provided as a percentage.
Task times were recorded for successes.
Observed task times divided by the optimal time for each task is a
measure of optimal efficiency.
Optimal task performance time, as benchmarked by expert
performance under realistic conditions, was recorded when
constructing tasks.
NOTE: Target task times used for task times in the Administrator’s
Guide has been operationally defined by taking multiple measures of
optimal performance and multiplying by the factor 1.25 allowing for
some time buffer because the participants are not trained to expert
performance. Thus, if expert, optimal performance on a task was [x]
seconds then allotted task time performance was [x * 1.25] seconds.
This ratio was aggregated across tasks and reported with mean and
variance scores.
If the participant abandoned the task, did not reach the correct answer
or performed it incorrectly, or reached the end of the allotted time
before successful completion, the task was counted as a “Failure.” No
task times were taken for errors.
The total number of errors was calculated for each task and then
divided by the total number of times that task was attempted. On a
qualitative level, an enumeration of errors and error types were
collected.
The participant’s path (i.e., steps) through the application was
recorded. Deviations occurred if the participant, for example, went to
a wrong screen, clicked on an incorrect menu item, followed an
incorrect link, or interacted incorrectly with an on-screen control.
This path was compared to the optimal path. The number of steps in
the observed path is divided by the number of optimal steps to provide
a ratio of path deviation.
Task deviations were reported. Optimal paths (i.e., procedural steps)
were recorded when constructing tasks.
Each task was timed from when the administrator said “Begin” until
the participant said, “Done.” If he or she failed to say “Done,” the
time was stopped when the participant stopped performing the task.
Only task times for tasks that were successfully completed were
included in the average task time analysis. Average time per task was
Page 9 of 29
Satisfaction:
Task Rating
calculated for each task.
Participant’s subjective impression of the ease of use of the
application was measured by administering both a simple post-task
question as well as a post-session questionnaire. After each task, the
participant was asked to rate “Overall, this task was:” on a scale of 1
(Very Difficult) to 5 (Very Easy). This data was averaged across
participants.
Common convention is that average ratings for systems judged easy to
use should be 3.3 or above.
To measure participants’ confidence in and likeability of the
functionality overall, the testing team administered the System
Usability Scale (SUS) post-test questionnaire. Questions included, “I
thought the functionality was easy to use,” and “I learned to use this
functionality very quickly.” See full System Usability Score
questionnaire in the Appendix.
Page 10 of 29
Results – Medication List
Data Analysis and Reporting
The results of the usability test were calculated according to the methods specified in the
Usability Metrics section. The usability testing results for the EHRUTs are summarized below.
The results should be seen in light of the objectives and goals outlined in the Study Design
description. The data should yield actionable results that, if corrected, yield material, positive
impact on user performance.
Medication
List
N
Tasks
#
Record Medication
Change Medication
Access Medication
3
3
6
Task
Risk
5=None
Task
Success
Task
Errors
Optimal
Path
Path
Deviations
Observed/
Optimal
2
1
4
100%
100%
100%
0%
0%
0%
5
5
3
1.2
1
1
Optimal
Task
Time
Seconds
25
20
15
Task
Time
Task Time
Deviations
Mean
(sec)
Observed/
Optimal
33
26
18
1.32
1.3
1.2
Discussion of Findings
Effectiveness: Effectiveness of functionality is determined by measuring the participants’ success
rates and errors. As noted in the summary analysis, the success rate on all tasks associated with
the Medication List was 100% and there were no errors. It should be noted however, that this
functionality has been available in Crib Notes for many years and all participants are current and
active users of the system.
Efficiency: Efficiency of functionality is determined by measuring the average task times and
paths and noting the deviations. The tasks were accomplished in almost optimal time with few
deviations. Again the participants’ prior experience in this functionality contributed to the
measures.
Satisfaction: The participants unanimously rated all tasks surrounding this functionality as very
easy to use. Here it should be noted that users can access this functionality from several different
screens (paths) and that contributes to the satisfaction level greatly as medications can be added,
edited and viewed during other workflows.
Areas of Improvement: The participants did not note any areas that needed improvement.
Task
Rating
5=Easy
Mean
5
5
5
Page 11 of 29
Results – Medication/Allergy List
Data Analysis and Reporting
The results of the usability test were calculated according to the methods specified in the
Usability Metrics section. The usability testing results for the EHRUTs are summarized below.
The results should be seen in light of the objectives and goals outlined in the Study Design
description. The data should yield actionable results that, if corrected, yield material, positive
impact on user performance.
Med/Allergy
List
N
Tasks
#
Record Med/Allergy
Change Med/Allergy
Access Med/Allergy
3
3
6
Task
Risk
5=None
Task
Success
Task
Errors
Optimal
Path
Path
Deviations
Observed/
Optimal
2
1
4
100%
100%
100%
0%
0%
0%
6
6
3
1.5
1.3
1.3
Optimal
Task
Time
Seconds
30
25
15
Task
Time
Task Time
Deviations
Mean
(sec)
Observed/
Optimal
45
35
20
1.5
1.4
1.3
Task
Rating
5=Easy
Mean
Discussion of Findings
Effectiveness: Effectiveness of functionality is determined by measuring the participants’ success
rates and errors. As noted in the summary analysis, the success rate on all tasks associated with
the Medication-Allergy List was 100% and there were no errors. Note that this functionality was
new to the participants. They had never seen or used it prior to this usability testing session.
Efficiency: Efficiency of functionality is determined by measuring the average task times and
paths and noting the deviations. Though the participants were unfamiliar with this functionality
they were still able to accomplish the tasks efficiently because it was consistence with other
entry, edit and view functionality in the system. It should be noted that the task times for
recording and changing medication allergies were affected because the users spent time
reviewing the ‘pick-list selections’ for several new fields (e.g.: reaction type and severity)
whereas in normal use they would be familiar with those options.
Satisfaction: The participants rated all tasks surrounding this functionality as easy to use.
Again, it should be noted that users can access this functionality from several different screens
(paths) and that contributes to the satisfaction level greatly as medications can be added, edited
and viewed during other workflows. This is especially true for viewing medication allergies.
Areas of Improvement: The participants did not note any areas that needed improvement.
5
5
4.5
Page 12 of 29
Administrator’s Guide
EHRUT Usability Test
Administrator ________________________
Data Logger ________________________
Date _____________________________ Time _________
# of Participants ___________
Location ____________________________
Prior to testing:
 Confirm schedule with Participants
 Prepare copies of consent forms and job aids
 Ensure EHRUT lab is setup properly
 Ensure Crib Notes application is accessible from lab
Prior to each task:
 As applicable, reset application to starting point for next task
Page 13 of 29
Orientation
Thank you for participating in this EHR Usability Test. Our session today will last no more than
15 minutes. During that time you will work with two features of Crib Notes – the Medication
List and Medication/Allergy List.
I will ask you to complete a few tasks using these functionalities and then ask you to answer
some questions. We are interested in how easy (or how difficult) these functionalities are to use,
what in these processes are useful to you, and how we could improve them. You will be asked to
complete these tasks on your own trying to do them as quickly as possible with the fewest
possible errors or deviations. Do not do anything more than asked. If you get lost or have
difficulty I cannot answer or help you with anything to do with the system itself. Please save
your detailed comments until the end of a task or the end of the session as a whole when we can
discuss freely.
I did not have any involvement in the design of the functionality, so please be honest with your
opinions.
Each of you will be assigned a patient to work with. You will notice that we have pre-entered
some data on those patients to better simulate a real scenario. For some tasks, we will also tell
you specific data to enter in. As you complete various tasks surrounding the functionality, we
will be recording time data on your activities. All of the information and feedback that you
provide will be kept confidential and your name will not be associated with your comments at
any time.
Do you have any questions or concerns? Okay, let’s get started then.
ADMINISTRATOR’S NOTE:
Participant ID (Clinician) A will work with Baby 1, Jones
Participant ID (Clinician) B will work with Baby 2, Jones
Participant ID (Clinician) C will work with Baby 1, Jones
Participant ID (RN) D will work with Baby 1, Jones – view only
Participant ID (RN) E will work with Baby 2, Jones – view only
Participant ID (RN) F will work with Baby 3, Jones – view only
Page 14 of 29
Task 1: Record a Medication
As applicable, take the participant to the starting point for the task.
In this task you will be recording a new medication on your patient. The medication you will be
adding as follows:
Infant weight: 2500 gms
Add: Ampicillin 125 mg IV q 12h
Add: Gentamicin 7.8 mg IV q 24h
Add: Acetaminophen 35 mg PO q 6 hours PRN
Success: A: _____ B: _____
C: _____
Easily completed
Completed with difficulty or help, Describe below
Not completed
Comments: ____________________________________________________________
Task Time:
A: _____
Optimal Path: A: _____
Correct
B: _____
C: _____ Seconds
B: _____
C: _____
Describe below
Describe below
Comments: ____________________________________________________________
Observed Errors and Verbalizations:
Comments: ____________________________________________________________
Rating:
Overall, this task was: A: _____
B: _____
C: _____
Tell participant scale: “Very Difficult” (1) to “Very Easy” (5)
Page 15 of 29
Administrator/Data Logger Notes & Comments:
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Page 16 of 29
Task 2: Change a Medication
In this task you will be changing medication on your patient. The medication you will be
changing is as follows:
Change to: Ampicillin 85mg IV q 8h
Change to: Gentamicin 4 mg IV q 12h
Change to: Acetaminophen 45mg PO q 8h PRN
Success: A: _____ B: _____
C: _____
Easily completed
Not completed
Comments: ____________________________________________________________
Task Time: A: _____
B: _____
C: _____ Seconds
Correct
B: _____
C: _____
Describe below
Describe below
Comments: ____________________________________________________________
Comments: ____________________________________________________________
Rating:
B: _____
C: _____
Page 17 of 29
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Page 18 of 29
Task 3: Access (View) Medications
In this task you will be accessing the medication list of your patient.
Participant ID (Nurse) D will work with Baby 1, Jones
Participant ID (Nurse) E will work with Baby 2, Jones
Participant ID (Nurse) F will work with Baby 3, Jones
Success: A: _____ B: _____
C: _____
D: _____
Easily completed
Not completed
E: _____
F: _____
Comments: ____________________________________________________________
Task Time: A: _____
F: _____ Seconds
B: _____
C: _____
D: _____
E: _____
F: _____
Correct
B: _____
C: _____
D: _____
E: ____
Describe below
Describe below
Comments: ____________________________________________________________
Comments: ____________________________________________________________
Rating:
F: _____
B: _____
C: _____
D: _____
E: _____
At this time we ask that you complete this short “System Usability Scale Questionnaire” for the
Medication Functionality. Again your comments and information will be confidential.
Your Participant ID is ________.
I will now collect the questionnaire and we will go to the next functionality.
Page 19 of 29
Task 4: Record a Medication-Allergy
In this task you will be recording a new allergy on your patient. The allergy you will be adding
is as follows:
Active: Enter Ceftriaxone allergy, Reaction-rash, Severity- Mild. Details per user
Active: Enter Clindamycin allergy, Reaction: Respiratory Distress, Severity: MildModerate, Details per user
Active: Enter Methicillin allergy, Reaction: Anaphylaxis, Severity: Severe, Details per
user
Success: A: _____ B: _____
C: _____
Easily completed
Not completed
Comments: ____________________________________________________________
Task Time: A: _____
B: _____
C: _____ Seconds
Correct
B: _____
C: _____
Describe below
Describe below
Comments: ____________________________________________________________
Comments: ____________________________________________________________
Rating:
B: _____
C: _____
Page 20 of 29
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Page 21 of 29
Task 5: Change a Medication-Allergy
In this task you will be changing an allergy on your patient. The allergy you will be changing is
as follows:
Change: Change status to Inactive
Change: Change Reaction to Urticaria, and Severity to mild, Details per user
Change: Delete Methacillin allergy, and add Metroclopramide allergy, Reaction: rash,
Severity: Mild, Details per user
Success: A: _____ B: _____
C: _____
Easily completed
Not completed
Comments: ____________________________________________________________
Task Time: A: _____
B: _____
C: _____ Seconds
Correct
B: _____
C: _____
Describe below
Describe below
Comments: ____________________________________________________________
Comments: ____________________________________________________________
Rating:
B: _____
C: _____
Page 22 of 29
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Page 23 of 29
Task 6: Access (View) Medication-Allergies
In this task you will be accessing the medication-allergy list of your patient.
Participant ID (Nurse) D will work with Baby 1, Jones
Participant ID (Nurse) E will work with Baby 2, Jones
Participant ID (Nurse) F will work with Baby 3, Jones
Success: A: _____ B: _____
C: _____
D: _____
Easily completed
Not completed
E: _____
F: _____
Comments: ____________________________________________________________
Task Time: A: _____
F: _____ Seconds
B: _____
C: _____
D: _____
E: _____
F: _____
Correct
B: _____
C: _____
D: _____
E: _____
Describe below
Describe below
Comments: ____________________________________________________________
Comments: ____________________________________________________________
Rating:
F: _____
B: _____
C: _____
D: _____
E: _____
At this time we ask that you complete this short “System Usability Scale Questionnaire” for the
Medication-Allergy Functionality. Again your comments and information will be confidential.
Your Participant ID is ________.
I will now collect the questionnaire. This concludes the EHRUT and we thank you for your time
today.
Page 24 of 29
Appendices
The following appendices include supplemental data for this usability test report. Following is a
list of the appendices provided:
• Non-Disclosure Agreement (NDA)
• Informed Consent Form
• Incentive Receipt and Acknowledgment Form
• System Usability Scale Questionnaire
Page 25 of 29
Non-Disclosure Agreement
THIS AGREEMENT is entered into as of _ _, 2014, between ____________________
(“the Participant”) and Grand Rounds Software, LLC.
The Participant acknowledges his or her voluntary participation in today’s usability study may
bring the Participant into possession of Confidential Information. The term "Confidential
Information" means all technical and commercial information of a proprietary or confidential
nature which is disclosed by Grand Rounds Software, LLC, or otherwise acquired by the
Participant, in the course of today’s study.
By way of illustration, but not limitation, Confidential Information includes trade secrets,
processes, data, know-how, products, designs, drawings, computer software, ideas,
improvements, inventions, training methods and materials, marketing techniques, and plans.
Any information the Participant acquires relating to this product during this study is confidential
and proprietary to Grand Rounds Software, LLC and is being disclosed solely for the purposes of
the Participant’s participation in today’s usability study.
By signing this form the Participant acknowledges that s/he will receive incentive compensation
for feedback and will not disclose this confidential information obtained today to anyone else or
any other organizations.
Participant’s printed name: ___________________________________
Signature: _____________________________________ Date: ____________________
Page 26 of 29
Informed Consent
Grand Rounds Software, LLC would like to thank you for participating in this study.
The purpose of this study is to evaluate selected functionality in Crib Notes. If you decide to
participate, you will be asked to perform several tasks in Crib Notes pertaining to the selected
functionality and give your feedback. The study will last about 30 minutes. At the conclusion of
the test, you will be compensated for your time.
Agreement
I understand and agree that as a voluntary participant in the present study conducted by Grand
Rounds Software, LLC I am free to withdraw consent or discontinue participation at any time. I
understand and agree that the data collected from this study may be shared with outside of Grand
Rounds Software. I understand and agree that data confidentiality is assured, because only deidentified data – i.e., identification numbers not names – will be used in analysis and reporting of
the results.
I agree to immediately raise any concerns or areas of discomfort with the study administrator. I
understand that I can leave at any time.
Please check one of the following:
Signature: _____________________________________ Date: ____________________
Page 27 of 29
Incentive Receipt and Acknowledgement Form
I hereby acknowledge receipt of a $10 Starbucks gift card for my participation in an EHR
usability test run by Grand Rounds Software, LLC.
Printed Name: ___________________________________
Signature: _________________________________
Date: _______________
Usability Test Administrator: __________________________________
Signature of Usability Test Administrator: ________________________ Date: __________
Witness: ________________________________________
Witness Signature: __________________________________ Date: _______________
Page 28 of 29
System Usability Scale Questionnaire
Functionality: ___________________
Participant ID: ____
1. I think that I would use the functionality frequently.
Strongly Disagree
1
Strongly Agree
2
3
4
5
2. I think that the functionality was too complex.
Strongly Disagree
1
Strongly Agree
2
3
4
5
3. I think that the functionality was easy to use.
Strongly Disagree
1
Strongly Agree
2
3
4
5
4. I thought that the functionality was way too cumbersome to use.
Strongly Disagree
1
Strongly Agree
2
3
4
5
5. I found that the functionality was well integrated.
Strongly Disagree
1
Strongly Agree
2
3
4
5
6. I thought that the functionality had too much inconsistency compared to other functionality.
Strongly Disagree
1
Strongly Agree
2
3
4
5
7. I learned the functionality easily.
Strongly Disagree
1
Strongly Agree
2
3
4
5
8. I felt very confident using the functionality.
Strongly Disagree
1
Strongly Agree
2
3
4
5
User-Centered Design Practice
Crib Notes
Version 3.0
Date of Report:
June 30, 2014
Report Prepared By:
Grand Rounds Software User-Centered Design Practice
Principles of User-Centered Design
“Usability” is defined as the ease of use and learnability of an object, in this case EHR software.
Usability includes the concept of user satisfaction and user experience as well as usefulness of
the product. In a user-centered design (UCD) paradigm, a software product is designed with its
intended end-users foremost in mind.
Grand Rounds Software applies a UCD framework as discussed in the NIST Guide to the
Processes Approach for Improving the Usability of Electronic Health Records (NISTIR 7741). 1
The principles and how Grand Rounds applies them follows:

Understanding user needs, workflows and work environments.
Crib Notes was originally designed by physicians with decades of NICU experience and a
deep understanding of neonatal provider needs and NICU workflow. The chief designer of
Crib Notes is a working Neonatologist and current active user of the system as is another
member of the development team. There is also an active community of users who frequently
provide input on how to enhance functionality and address the emerging needs of NICU
providers and patients.

Engaging users early and often.
Because many of the enhancements and new functionality are often first suggested by users,
users are frequently brought into the process to review the functional specifications prior to
coding.

Setting user performance objectives.
The design team creates user performance objectives for each task pertaining to new
functionality as part of use cases that are developed at the onset of the design phase.

Designing the user interface (UI) from known behavior principles and familiar user interface
models.
Grand Rounds Software follows the standard best-practices in UI design including
navigation and system conventions to promote familiarity, consistency and visual clarity.
1
NIST Guide to the Processes Approach for Improving the Usability of Electronic Health
Records


Conducting usability tests to measure how well the user interface meets user needs –ANDAdapting the design and iteratively test with users until performance objectives are met.
Grand Rounds Software conducts usability evaluations at several points in the software
development life cycle. First, in the “analysis and design” stage, again, in the “testing”
stage, and finally as feedback after “implementation/deployment”.
Evaluate usability
design with users
(functional
specifications
including mock-up
UI screen shots)
Design functionality
(often based on
suggestions from
current users)
Assess
feedback *
Develop
functionality
Conduct usability
test with users
* Incorporate usability feedback and depending upon the
severity, the feedback is either applied to the development or
design work.
Grand Rounds Software Usability Evaluation Methodology
Grand Rounds Software employs a “Cognitive Walkthrough” approach for usability evaluation
and testing. This method involves a small usability evaluation group typically including the lead
designer, an application analyst and one or more users in the appropriate user role (eg: clinician
vs. nurse) reviewing the user interface. For the first session, the user interface is presented on
paper as part of the functionality specifications with mock-up screen shots. For subsequent
usability sessions, the UI is working and available. During the sessions, the evaluation group
“walks” through a set of the most typical user tasks supported by the new or enhanced
functionality, one-step-at-a-time. 2
This approach to usability evaluation is especially suited for the Crib Notes software product as
the approach is specifically task-orientated and most closely matches what the users experience
in the real healthcare setting environment, the NICU.
2
Wharton, C., Rieman, J., Lewis, C., and Polson, P. (1994). The cognitive walkthrough method:
A practitioner’s guide. In Nielsen, J., and Mack, R. (Eds.), Usability inspection methods. New
York, NY: John Wiley & Sons, Inc
www.cribnotes.com
Robert Stavis, Ph.D., M.D.
President
[email protected]
August 4, 2014
Mr. Gary Isaac
EHR Test Proctor
Drummond Group, Inc.
13359 North Highway 183, Suite B-406-238
Austin, TX 78750
Dear Mr. Isaac:
The accompanying QMS Practices document is responsive to Meaningful Use Certification
Criteria 170.314(g)(4) and describes the Quality Management practices of Grand Rounds
Software regarding the development, testing, implementation and maintenance of Crib Notes
across all of the modules that we have presented for certification.
Sincerely,
GRAND ROUNDS SOFTWARE, LLC
By:_____________________________
Robert Stavis, M.D., Ph.D.
President
404 Valley Glen Drive
•
Bryn Mawr, PA 19010
•
(800) 323-9167
•
[email protected]
Quality Management System Practices
Crib Notes™
Version 3.0
Date of Report:
July, 2014
Report Prepared By:
Grand Rounds Software Quality Management System Practices
Principles of Quality Management Systems
The quality management system (QMS) of Grand Rounds Software (GRS) describes the
organizational structure, policies, procedures, processes and resources principles and processes
surrounding the design, development, client implementation, and maintenance of Crib Notes™
that promote and ensure quality. Elements of the GRS QMS include:
•
•
•
•
•
•
•
•
•
•
Personnel training and qualification
Control of product design
Control of documentation
Control and definition of the development process
Validation of processes
Product acceptance
Control of nonconforming products
Institution of corrective and preventive action when errors occur
Record keeping
Statistical techniques
Grand Rounds Software has incorporated internationally recognized methodologies and
standards including those of the Rational Unified Process, SEI CMMI Level 5, Agile
Development Management, Program/Project Management, and ITIL into its software
development life cycle. These methodologies intrinsically address QMS and primarily impact the
following areas:
•
Management Responsibility: Policies and processes in this area, directly driven by senior
management, provide for the administration and adherence of quality management to all
processes in organization.
•
Resource Management: Policies and processes in this area provide resources and/or
infrastructure to for software development.
•
Software development: Policies and processes in this area provide the processes and tools
used in software development following the QMS framework and customers’
requirements.
•
Measurement, Analysis and Improvement: Policies and processes in this area provide
mechanisms to monitor and measure the effectiveness of Crib Notes™ in meeting client
requirements, and to analyze nonconformities that lead to decisions for improvements to
achieve higher levels of effectiveness and client satisfaction.
Quality
Objectives
Resource,
Support Needs
4
Human Resource
Management
Review
Work Environment
Security
Resource, Support
Purchasing
Continual Improvement
Corrective and Preventive action
Data Analysis
Satisfaction
Customer satifastion
Internal Audit
Noncoforming product control
Hardwares,
softwares
Software Product Realization
Monitor and Measurement
Deliverables
Customer
Resource
Management
Customer
Training
3
Internal
Communication
Management Responsibility
2
Requirements
Responsibilities
and Authorities
Improvemen
t decisions
1
QMS Planning
Process/
product
Organization
Quality Policy
Measurement - Analysis - Improvement
Customer
Focus
QMS Applications in GRS Software Development
Each Crib Notes™ build cycle proceeds along a clearly defined pathway, summarized by this
project management schema:
Requirement Definition
•
•
•
•
Software designers develop functional requirements and specifications document and add
to the JIRA issue tracking system (JIRA).
The development team reviews and clarifies requirements.
The development team Project Manager (PM) reviews changes and inputs impacts using
Traceability Matrix and comments on JIRA.
Specifications are finalized.
Risk Management
•
The development Team PM identifies the major project risks and the risk mitigation
actions associated with the development and maintenance of the application. Identified
risks are communicated to relevant project stakeholders and tracked to closure.
Estimation Effort and Scheduling
•
•
The Development Team PM uses Ticket Point model to estimate effort which is reviewed
by PM and team members.
PM uses Microsoft Project to create schedule plan that is reviewed by team members.
Test Case Development
•
•
Quality Control Team defines test cases for new requests.
Test cases are reviewed by QC Team, PM, and team members.
Software Development and Initial QC
•
•
•
Developers follow Java coding standard and apply all defined best practices.
Developers follow the test cases created by QC to do unit testing.
Code review is conducted using Code Review Checklist.
Quality Control Testing
The test stages are as follows:
•
•
•
•
•
•
•
System Testing
Integration Testing
Acceptance Testing
Testing types include:
o Smoke Testing
o User Interface Testing
o Acceptance Testing
o Functionality Testing
o Regression Testing
o Performance Testing
QC follows the test cases to verify the ticket.
Tickets that do not have a defined test case are verified using the Testing Checklist and
Traceability Matrix.
The test status is updated in Testing Report
Causal Analysis and Resolution (CAR)
•
•
•
After a production release, the PM collects defects, problems, and best practices for
analysis. The analysis drives appropriate corrective and preventative actions to enhance
project performance.
Actions proposed from the CAR process are tracked through completion using the PPDM
tool.
ANOVA statistical techniques are used to analyze root cause problems, and measure
effectiveness after applying corrective/preventive actions.
Implementation
•
•
Crib Notes™ installations follow industry standard, best-practice healthcare IT
implementation methodologies. Deliverables include an agreed upon Implementation
Project Charter, Project Workplan, and Training materials, all adapted for the specific
client install.
Project scope, risk, issues and quality are addressed continually throughout the duration
of the implementation.
Maintenance
•
•
•
Features and functions that are required by client requests, changing clinical and
regulatory practices, and/or hardware/software infrastructural changes are identified and
specifications for changes are developed, posted, and tracked on JIRA.
The software development processes and monitoring tools described above are applied to
these change requirements.
The issues are incorporated into production releases as agreed upon by the senior
management and development team.
www.cribnotes.com
President
[email protected]
July 23, 2014
Mr. Gary Isaac
EHR Test Proctor
Austin, TX 78750
Dear Mr. Isaac:
With respect to the Crib Notes EHR system for which certification will be sought for
170.314(d)(2), we provide this attestation:
1. Are default settings for audit log and audit log status record are enabled by default? Yes. Crib Notes audit records are created as part of the base programming of the system, and there is no provision within the application to turn off the audit logs. 2. Is encryption of electronic health information on end-‐user devices is enabled by default? Yes. Documents provided to parents (i.e. Discharge Summary, Growth Charts, Discharge Instructions) are encrypted by the application, and there is no provision to not encrypt these documents. 3. Does the EHR SUT allow a user to disable the following? • audit log: No. • audit log status: No. • encryption status: No. 4. Does the EHR SUT permit any users to delete electronic health information? In general, no data may be truly deleted. Some types of data may be removed from the GUI, but such data remains in the audit logs and database. 5. Describe how the audit logs are protected from being changed, overwritten or deleted by the EHR technology. 404 Valley Glen Drive
•
Bryn Mawr, PA 19010
•
(800) 323-9167
•
[email protected]
The EHR application provides read-‐only access to the audit files; there is no provision to manually create, edit, or delete audit records. 6. Describe how the EHR is capable of detecting whether the audit logs have been altered. The audit files in Crib Notes cannot be manually created, edited or deleted within the Crib Notes application, so the need to determine whether the audit logs have been altered does not apply. Audit records are stored within the Microsoft SQL database and can only be accessed by a database administrator with edit permission. Changes would be forensically evident in transaction logs and backups of the database. Sincerely,
By:_____________________________
President
•
Bryn Mawr, PA 19010
•
(800) 323-9167
•
[email protected]
www.cribnotes.com
President
[email protected]
July 23, 2014
Mr. Gary Isaac
EHR Test Proctor
Austin, TX 78750
Dear Mr. Isaac:
With respect to the Crib Notes EHR system for which certification will be sought for
170.314(d)(7), we provide this attestation that the AES-128 bit technology is used for
encryption on end-user devices. This technology is included in the specification of Annex A
of the Federal Information Processing Standards Publication 140-2.
Sincerely,
By:_____________________________
President
•
Bryn Mawr, PA 19010
•
(800) 323-9167
•
[email protected]

Report - Drummond Group

Transcription

Similar documents

Rangemaster - JWWalker.net

Lloydminster Grand Opening Invitation

40mm multiple grenade launcher mk il

^ Sand Creek Wildlife Management Area

Joan AYLING 16 rue de Liffol 88350 GRAND France www.apollon

Columbus KKC Flyer 2013 - Columbus Cocker Rescue!

MOVIE DAY

grand re-opening - Deerfoot Meadows

enlarge