How to Integrate NERC’s Requirements in an Ongoing Automation and Integration

How to Integrate NERC’s Requirements
in an Ongoing Automation and Integration
Project Framework
Jacques Benoit, Cooper Power Systems Inc.,
Energy Automations Solutions - Cybectec
Robert O’Reilly, Cooper Power Systems Inc.
Energy Automations Solutions - Cybectec
Abstract
This paper addresses the challenges faced by utilities and/or integration
companies during deployment and engineering phases of automation and
integration projects, with regards to complying with the new cyber-security
requirements set out by NERC. This paper will focus on approaches to these
new challenges to ensure the project stays within schedule and budget, from the
point of view of substation requirements, management and of the different
SCADA systems.
Introduction
This technical paper will discuss the challenges of minimizing the impact of
adding NERC CIP compliance to an ongoing project consisting of updating a
substation’s automation systems. Originally aimed at providing faster access to a
higher amount of operational and non-operation data within a substation
framework, the changeover is an opportunity to upgrade some of the protection
and metering devices. But now, the project must also include compliance with
cyber-security requirements.
While at first glance NERC requirements may seem to be an insurmountable
task, when one takes a closer look at the standards, it becomes obvious that
proper planning and best practices are the key to accomplishing compliance.
Moreover, proper planning will minimize the impact of NERC CIP compliance on
the project’s budget and timeline.
From a project implementation point of view, NERC CIP mainly describes what is
required from utilities, but does not provide any technical information on how to
implement a project to meet those requirements. This leaves a lot of room for
interpretation and implementation.
From a project viewpoint, one must decide quickly which requirements would
normally be addressed outside of a project scope and hence would not impact
adversely its timeline or budget. Since they should be the responsibility of other
groups within the organization, we will not discuss the following CIP standards in
this paper:
CIP-001
CIP-008
CIP-009
Sabotage Reporting
Incident reporting and Response Planning
Recovery Plans for Critical Cyber Assets
Instead, we will discuss how the following key CIP requirements have a direct
impact on your ongoing project and should be addressed in any ongoing project:
CIP Reference #
CIP-002-R3
CIP-003-R4.1
CIP-003-R5.1
CIP-003-R6
CIP-004-R2
CIP-004-R3
CIP-004-R4
CIP-005-R2
CIP-006-R2
CIP-007-R1
CIP-007-R2
CIP-007-R3
CIP-007-R4
CIP-007-R5
General Description
Critical cyber asset identification
Critical cyber asset information to be protected (items
defined by management team)
Access control (personnel cleared to access
protected information)
Change control and configuration management
Training of all personnel (operation, technical,
contractors, etc.)
Personnel risk assessment
Personnel access to critical cyber assets
Electronic access controls (ensure electronic access
is only permitted to approved personnel)
Physical access controls
Test procedures (supplied by others)
Ports and Services (ensure only the required ports
and services are active, all others are turned off)
Security patch management
Malicious software prevention
Account management
For the readers’ convenience, we have summarized the different CIP
requirements in the appendix.
The Original Project
The example chosen is that of modernizing an existing substation automation
system. This type of project was selected because it is probably the worst case:
not only is new equipment added, but legacy equipment is also kept in the
substation. The implementation must be done while keeping legacy systems in
operation. Moreover, the project must allow for compliance with all applicable
CIP requirements, and be able to pass a compliance audit near the end of the
project.
We will use the example of a typical legacy substation which has been in
operation for more than fifteen years and undergone normal additions required
by increased client demand. In most cases, such substations would resemble the
following diagram.
Fig. 1 - Existing substation automation before project
The first order of business when moving an ongoing project towards NERC
compliance is to plan for the substation’s auditability
As the project engineer, one must keep in mind that the plan must be approved
by the company’s NERC committee, and that deadlines and budgets are not
expected to be impacted.
A Review of the Project
Usually, projects are planned and budgeted with preliminary engineering
performed more than 18 months before actual implementation. This delay can
create an issue relating to equipment and software costs, as well as delivery lead
times.
The first project review for CIP compliance will require the retrieval of all
information on the previously selected components for the project. Then all
potential critical cyber-assets will need to be documented. The list will finally be
reviewed to ensure that the security requirements can be met with the equipment
that had been originally selected.
The initial substation planned architecture is presented below:
Fig. 2 - Automation overview diagram of planned project
It is important to review any potential new features of the equipment that had
originally been selected in the planning stages of the project. Quite often, the
product contains new features/capabilities that will in the end save time in the
detailed engineering and commissioning phases. Hence, although one may feel
the operation is time-consuming at first, it will probably save time by the end of
the project. What remains to be examined is the increased paperwork and
preliminary audit added to the factory acceptance test (FAT).
The risk assessment portion could be performed during the audit and FAT. The
project should be executed with a best practice approach which should bring the
risk within a manageable context.
Establishing the Security Perimeter
The connection between the substation and the corporate WAN had been
planned using a router and firewall. This setup had been approved by the IT
group. In view of the NERC CIP requirements for an electronic perimeter, this
configuration can no longer be considered adequate. For instance, this device
does not meet the access control and logging requirements.
Most substations also contain older devices such as power meters and DFRs
with limited communications capabilities. These devices require some form of
protocol converter. Also, in addition to the main access points, the EMS group
requires the use of a dial-up connection for remote access to the metering
equipment. Dial-up access is flagged as a major potential security risk by NERC
CIP standards.
Now it is clear to the engineering team that using only a router will not comply
with NERC CIP’s required electronic security perimeter.
One might recommend a gateway device in addition to the router. Gateway
devices usually provide secure communications capabilities using modem
connections, serial, and TCP/IP. They create a single point of access to the
substation making it easier to secure the electronic perimeter. Although they will
vary form vendor to vendor, these gateway devices usually also provide an
additional firewall and security features.
Isolating the substation’s critical assets and physically installing them in strategic
and secure locations within the substation also helps to meet the CIP physical
perimeter security requirements.
Equipment Inventory
Once the electronic security perimeter has been defined, the inventory of
equipment must be established and documented.
Although this seems a difficult task on the onset, it is more easily prepared than
one might think. All the information required is already available so that
equipment data is brought back to the central systems (be it SCADA, EMS, Asset
Management, or others) via the intelligent gateway. Designing how information is
to move from substation to control center will also help define what information is
more important.
During this phase it is recommended to have short and to the point brainstorming
sessions with the different groups wishing to have access and to have them
document their requirements. One might be surprised how demands are reduced
when written versions are required.
Once this information has been identified, the intelligent gateway can be used to
limit access to this information. Access levels and user groups should be used to
only allow specified systems and users read or write access. Any other system
should not be allowed to retrieve/operate on the information.
For information which is made available via the intelligent gateway; the unit’s
security environment should be configured to let only the specified computer
system(s) access the specified and approved information. This information
should be documented for future auditing requirements.
Access Control, Personnel Risk Assessment, Access to Cyber
Assets and Account Management
Before NERC CIP standards, these points were not normally part of a project.
However, CIP standards make their assessment and documentation mandatory.
Fortunately, help usually can be found in other groups within one’s organization.
Human resources and senior management can define access levels and the
personnel who will have them, as well as perform the personnel risk assessment.
This should not impact the project’s budget. Only documentation of those
accesses would remain to be produced.
One can use a central security server or the intelligent gateway’s security
features to manage accounts. Obviously, central account management is much
more efficient in providing comprehensive authentication and simplifies meeting
the NERC requirement of being able to remove access rights rapidly. Central
user management may however require new servers and software, which would
normally be expensed from the IT budget.
Change Control
Although change control and configuration management may seem new, most
project managers who have been through a number of projects understand this
as the mandatory documentation process to control risk during an automation
upgrade project. Hence it is usually planned in the original weekly review list.
At this point already seven items of your CIP requirements list have been
addressed or planned for:
CIP-002-R3
CIP-003-R4.1
CIP-003-R5.1
CIP-003-R6
CIP-004-R3
CIP-004-R4
CIP-005-R2
CIP-007-R5
Critical cyber asset identification
Critical cyber asset information to be protected
Access control
Change control and configuration management
Personnel risk assessment
Personnel access to critical cyber assets
Electronic access controls
Account management
So far, there was very little impact on budget or timelines, except for delays
regarding reviews of personnel risk and their security clearance. However, this
requirement is usually the responsibility of human resources for personnel and of
the purchasing group for the contractors.
Security Patch Management and Malicious Software Prevention
The manufacturer of the gateway device will usually provide the tools to properly
handle any patch management and prevent malicious software. Many techniques
exist and it is not in the scope of this paper to decide which approach is better for
this facet of the CIP requirements. Suffice it to mention that today, tools and
equipment are available for this purpose. However, it is still up to the project
team to validate that these tools will perform as required by the project and
corporation.
Test Procedures and Port Blocking
During the final engineering phase, one should prepare a framework of the
testing methodologies that could be required to validate the new automation
system and its integration into the current substations operations. This is usually
done with the help of the vendor or the integrator. When dealing with a
change/addition to an existing substation, careful planning must performed to
ensure that the system will interface and react properly and promptly to the
substation operation requirements.
This detailed testing phase is the most appropriate time to check that all of the
ports and services not required by applications are turned off. This can be done
remotely, since it is easy to forget this type of work during commissioning of the
systems at the substation. The IT group can provide the tools required for these
tests. However, vulnerability testing should not be performed on a live system as
it may render it inoperable.
If possible, one should change the default ports. For example, DNP3 via TCP/IP
uses port 20000 by default. With newer systems and applications, this can be
changed, hence preventing anybody from accessing your system by trying to
ping the standard ports.
Fig. 3 – Overview drawing of the final concept for the new automation systems
Personnel Training
Training is usually the last item on a project’s list. In today’s complex operational
environment it should not be neglected. Personnel training has always been a
priority for most organizations and is planned and budgeted accordingly. NERC
CIP standards simply require more detailed documentation regarding training
sessions, attendees and the personnel’s ability to react appropriately in different
situations.
Training should be a requirement from all vendors providing the
software/hardware for the project. The training should include detailed hands-on
lessons with the applications, hardware and general software.
Security Software
Security software should be chosen together with the IT group and should
provide a centralized approach, where it is easier to manage access rights and
users, data logging, intrusion monitoring and system health monitoring.
Local security should also be implemented in the substation, for onsite
personnel. Local security must provide the capability of being integrated into the
centralized approach to simplify overall user and application management but
also to provide the capability for the security to be available at the local level
when connection(s) to the centralized systems is not available.
Conclusion
Proper planning is the key to minimizing the impact of NERC CIP standards on a
project’s timeline and budget. Individual steps towards NERC CIP compliance
are not complex: they simply require a little more effort on the documentation and
planning sides. When one has experience with retrofit projects, proper
documentation and training become a life-saver at project’s end.
Appendix: CIP Standard Solutions Breakdown
Requirement
Description
CIP-002-R3
Critical cyber asset
identification
CIP-003-R4.1 Critical cyber asset
information to be protected
CIP-003-R5.1 Access control
CIP-003-R6
CIP-004-R2
CIP-004-R3
CIP-004-R4
CIP-005-R2
CIP-006-R2
CIP-007-R1
CIP-007-R2
CIP-007-R3
CIP-007-R4
CIP-007-R5
Solution
Reuse project inventory
Review with different groups
requiring access to information
Seek access models from upper
management, use centralized
authentication model
Change control and
Reuse project change
configuration management
management infrastructure
Training of all personnel
Improve documentation
Personnel risk assessment
Human Resources and
Purchasing to conduct
assessments
Personnel access to critical Seek access models from upper
cyber assets
management, use centralized
authentication model
Electronic access controls
Use centralized authentication
model
Physical access controls
Install card reader or video
camera
Test procedures
Use exhaustive FAT procedures
Ports and Services
Reassign ports when possible,
use intelligent gateway to restrict
access
Security Patch Management Use intelligent gateway with
security patch management
feature built-in
Malicious software
Use intelligent gateway with
prevention
malicious software prevention
feature built-in
Account management
Use centralized authentication
model