hello and welcome
to the defcon 29 ics village
where today i'm going to talk about
crippling the grid
an examination of dependencies and cyber
vulnerabilities
just what that means so first who am i
well my name is joe sloick
i'm currently the cti and detections
lead for network security monitoring
company gigamon
previously i performed cyber threat
intelligence research for domain tools
and dragos and prior to that
i've held multiple roles in the us
government including at los alamos
national laboratory in the u.s navy
but enough about me what's our agenda
for today
so today we're going to first talk about
understanding what the grid
actually means as a concept as
a really a machine
and then with that high level
understanding we'll then
dive into some of the particulars of how
the electric
system actually operates and potential
dependencies and injection points
that could lead to cyber nexus attack
routes in order to cause
disruption and disruption in a way
different than what we see with most
headline events that have taken place
over the last 10 years
and finally we'll conclude on a more
uplifting note in talking about what is
actually required to defend against
attacks of this nature
so first when we talk about the grid we
have to realize that
we're really not talking about one thing
from a north american perspective we're
really talking about several things
within the united states alone we have
the
western texas and eastern interconnects
as well as
expanding a bit out uh similar
sections covering alaska and quebec so
we really have different sections
different geographic regions
that while not completely independent of
one another
certainly do create segmentation when it
comes to the overall
generation transmission and distribution
of electricity
which actually gets us to another point
that the electric sector is not a
monolithic entity
but really consists of entities spanning
several distinct phases
from generation what people typically
think of when they talk about
the production of electricity
but that generation has to be
transmitted across substantial distances
in order to reach consumers where final
sort of last mile distribution takes
place
by other entities and these entities can
and often are
uh distinct from one another
representing different companies and
it's important to note that
if we look at historical events and
impacts
we've seen attacks take place at
different phases
of the electric system so the two
ukraine events the first
in 2015 targeted distribution the second
in 2016 targeted transmission
and we're not quite aware of any actual
in the wild
impact on generation at least not at
this time
it's not to say it hasn't happened it
just means that no one's really talked
about it or proven that it is the case
although there certainly have been some
tests like the aurora generator test
that have touched on the subject
but to close this section up there's no
single grid
we have multiple phases of operation
throughout the
process of delivering electricity to end
users
and different organizations often
operate at different phases of the
this operation that already gets us a
pretty complex attack surface between
different regions different
organizations operating within that
region
and how they might interact with one
another but there's a lot more going on
than just
this high level view of the electric
system
because but wait there is more
because there are additional system
characteristics that tie into
how the electric system operates as a
system of systems
as one continent-spanning machine so to
speak
that gets us into items such as
requirements for generation
the stability of the system in order to
operate and operate in a way that's safe
as well as the actual delivery of
services to consumers
we're not going to touch on this last
one that much but it is something to
bear in mind
especially as we see increasing adoption
of things like smart metering
and other uh ways of transmitting more
control and more interaction
all the way to end users that previously
just didn't exist before
that's an entirely different topic that
we can do an entire presentation on
on its own but we'll talk about though
are
generating sources in fuel so while it
would be ideal
that electric generation
was more renewable in uh nature
or even nuclear i would argue primary
ways of generating electricity across
much of the world right now
consist of fossil fuels principally coal
and natural gas and because we're
talking about burning
a substance in order to generate steam
that then turns a turbine
we need fuel in order to make this
process work
well that's a pretty important
vulnerability
in how these systems work because if you
can somehow cut off or disrupt the
supply of fuel
that can be problematic very quickly so
certainly with coal
we have coal storage yards that could be
quite large and have
maybe several days worth of supply but
natural gas is typically a just-in-time
very low storage at least on-site uh
proposition so a disruption in these
sorts of markets can be quite
catastrophic quite quickly
furthermore we have balancing
authorities when it comes to things like
system stability so we already looked at
the grid as consisting of at least three
high-level distinct entities but within
those entities
we have a further set of
responsibilities for balancing
the frequency phase and similar
characteristics as well as
matching the actual uh amount of
electricity generated
with demand at any given point in time
uh in order to make sure that the system
functions
electricity has to be consumed
essentially as it's created and has to
do so within
fairly well-defined uh tolerances in
order to do so safely without damaging
equipment
and that's what these authorities are
set up to do to organize these systems
and make sure that everyone is playing
along the right set of rules
because the thing is is that if we start
getting a unstable or an unmanaged
grid within one of the balance authority
geographies as well as through the
entire interconnect
bad things can happen so for example in
north america
we generate around 60 hertz in terms of
frequency
uh corresponding to you know really just
literally
how a generator is turning and how many
uh revolutions per unit of time
and the thing is is that as we start
getting away from
that frequency even by relatively small
amounts we induce the possibility for
uh either errors in the system but also
set the stage for
outright equipment damage that needs to
be mitigated through various controls
the same applies for things such as the
frequency around which the
we talked about frequency the phase
around which
electric generation takes place and how
this is then communicated throughout the
system
onto transmission and even distribution
levels
with other physical equipment balancing
authorities step in to make sure that
everyone participating in these systems
is doing so in the right fashion and
doing so in a way that
allows the system to function normally
now i've just talked a lot of very very
high level these are complex topics and
we could talk about these
for quite an extended period of time but
you didn't come here to talk about power
engineering
you came here to talk about cyber and we
haven't really touched on cyber at all
just yet but the thing is
is that cyber actually has a role to
play in this and it's one that's
increasing
and that leads to several risks for
example
just recently a few months ago we had
the colonial pipeline ransomware
incident
while that didn't impact all the
electric sector
particularly this was a petroleum
pipeline so we're talking about
vehicle fuel aviation fuel and similar
this does show that pipelines can be
fragile to a certain extent now in this
case the disruption to colonial pipeline
never really resulted in
ransomware-induced shortages
mostly panic buying and quite frankly
human stupidity
produced the impact in this scenario but
it does show that pipelines can be
impacted in ways that have physical
repercussions through something
like relatively indiscriminate
ransomware operations
well what does that mean when we start
talking about natural gas and not
petroleum
natural gas the sort of natural gas
revolution
in uh power generation in north america
is astounding you can see in the chart
here that
where natural gas used to be sort of a
lower level contributor
to the electric generation mix of north
america
it has rapidly uh grown to become the
number one source of electricity
within the united states and the thing
is is that while that is fueled by
things like the
shale gas boom and other sort of
operations it still requires the
movement of natural gas from its point
of extraction
typically places in texas in the
northern uh midwest and other areas to
get to generating facilities
typically typically closer to
populations and as you can see from this
map
there are some pretty interesting choke
points we're starting to look at this as
a system
so certainly around major uh gas
development regions again like texas and
the gulf coast we see lots and lots of
pipelines and
uh lots of uh reliability or at least
fault potential fall
failovers within the system but as we
start getting into areas like the
western united states
the eastern seaboard the florida
peninsula and other areas
start seeing fewer and fewer redundancy
uh within this system
so what if we were able to induce some
sort of disruption to pipeline systems
on the gas sector well the thing is
we've already seen some of these events
take place but
they did so at a fairly small level
so in uh
2019 there was a notice put out by the
united states coast guard of all
entities
that a relatively small pipeline
facility was impacted through a ryuk
ransomware event that resulted in a
disruption so a very minor event given
the size and scale of the facility
involved
but it shows that we can get disruption
of petrochemical including natural gas
infrastructure
through something like ransomware
another item to keep in mind is that
just recently cisa announced or at least
attributed
and made publicly available a report
that it existed for some time
a widespread intrusion campaign from
2011 to 2013
linked to the people's republic of china
that was targeting pipelines at a very
thorough uh very widespread sort of
basis so adversaries are either
sort of unintentionally from the
ransomware perspective or very
deliberately through
very long duration intrusion operations
involving pipelines exploring this field
to what end you may ask well if i can
identify
a critical pipeline or segment maybe not
getting quite down to an individual
compressor station but at least
identifying the
control center that would control
multiple uh compressor stations and
similar infrastructure for a given
pipeline
i could potentially disrupt that control
center or similar functionality through
a cyber event
and result in a disruption of pipeline
information
operations that would restrict the flow
of gas to follow on generation stations
as well as things like home heating and
other
items that could start to create a real
physical impact
now certainly some of these systems are
well segregated and there are defenses
in place but
given the trend line away from
very local very hands-on sort of
management of these systems
and instead to having distributed
management from a
single or handful of remote control
centers
this sort of scenario starts become
significantly
more impactful and realistic than it may
have been even just 5 or 10 years ago
but in addition to things like
feedstocks we also have stabilization
impacts
so these are two examples that took
place
in early 2019 ironically both on the
same day
uh in 11th of january of 2019
of frequency deviation events one in the
united states the other
in the european union in both cases
there were essentially unexplained at
least at the time
significant frequency deviations that
resulted in
non-trivial impacts to the overall
electric system
shedding load which means people not
having power
uh taking plants offline in order to
prevent physical damage and other things
while some explanations have emerged for
why these events took place it's still
very strange when you start looking at
either nsoe
or this project from oak ridge national
laboratory in the university of
tennessee
that tracks these sort of anomalous
events that these are quite common but
systems can step
in in order to prevent the worst case
scenarios from developing
as a result of these uh grid
wide phenomena that take place so
examples of those sorts of impacts
include just simple operational
efficiency as well as wear and tear
so having things like protective relays
stepping in to
take a system off or put a system back
on when there are significant frequency
deviations
does generate wear and tear on that
equipment when things get a little bit
more
dicier and sort of what we saw in the
european example
we can start seeing assets become
isolated so
removing equipment from the system or
shedding load in such a fashion that
consumers
experience a blackout in order to make
sure that the system remains stable
finally we can even get into potential
physical damage scenarios so the aurora
generator test that i
spoke about earlier involved a
manipulation
in the electro the uh protective relay
system
for a uh generating asset
and by manipulating that in just the
right way so that you re-sync the
generating asset
out of sync to frequency or phase you
can cause it to basically
shutter uh and do that enough times or
with a
large enough deviation you can actually
induce physical damage
now we haven't really seen anything like
that take place
uh at least not to a breathtakingly
destructive fashion because there are
many systems
that step in in order to prevent these
sorts of scenarios from occurring
but it is something that the grid has
been designed around because it's a very
real
risk to equipment well from a cyber
perspective though we're starting to see
adversaries become more interested
either incidentally or very deliberately
to the types of systems that step in
from a
balancing and stability uh perspective
when it comes to
electric system operations so for
example within the last two years
really actually just within the last
year this took place last year
there were two events of interest the
first involved in intrusion
at the european entity enso e
which essentially links the transmission
system operators
uh on the european continent together to
allow for better communication and
whatnot
so nsoe doesn't really control anything
itself but it's an awfully nice
springboard to getting access to
the tso organizations that are
underneath it to facilitate follow-on
targeting
uh not what you would typically expect
for an adversary to really go after
for an intelligence value or anything
else and there was no ransomware in this
case
but a very intriguing target for what it
might enable for follow-on operations
around the same time there was also a
ransomware event
that targeted a entity in the united
kingdom called alexa
that's responsible for essentially
market operations
to match electric generation with
electricity demand
while it's very debatable that taking
down an electron would result in any
actual physical
damage to the underlying electric system
it certainly would increase the amount
of friction
in play in trying to operate deregulated
electricity markets
and so we can't quite ignore that either
if we go back a little bit further too
and it's not on this slide
we could look at the 2016 ukraine power
event as another example of at least
approaching some aspects of this
frequency and stability uh question when
the attack resulted in manipulation and
actually disablement
of the protective relays on the victim
power well
transmission station as part of that
incident so adversaries are starting to
learn that not only are these systems
present uh and represent a sort of soft
underbelly to the overall electric
system
but they can also be reached targeted
and potentially even manipulated
through computer network operations and
we have examples of this
taking place albeit at very low and at
least not yet
or at least not not known destructive
levels for
over five years if we go back to the
2016 ukraine incident
now the possibilities for an adversary
in this sort of grid stability
attack scenario range from denial
walking access to system-wide resources
and removing
system-wide safety and other sort of
checks that allow for
efficient and uh very integrity focused
operations of the electric system to
disruption
where we can outright disrupt operations
as they shut down
in order to preserve safety in an
unstable or
uh just uncertain operating environment
uh to outright destruction so like we
talked about with things like the aurora
test and
something like the 2016 ukraine event
there are possibilities to induce
physical damage
this would be really really really
flipping hard to try to actually execute
and practice because
it would require not just setting off
the event so to speak by creating the
network-wide instability but also
removing protections and other controls
that are in line
to prevent uh worst-case scenarios from
developing so an adversarial would have
to do a number of things
to make this possible however the
increasingly
networked nature of a lot of our grid
operations
systems as well as individual uh
portions of the electric
system as it may exist mean that there
are increasing possibilities for
adversaries to reach out and touch
some of these elements so we can't quite
ignore it even if this represents a very
distant right or left tail possibility
compared to some of these other
scenarios
so that's all very concerning and
present some problems but as lenin would
say
what is to be done well the first thing
is we have to really
remember and recognize that the grid is
a big complex
project whether we're talking in europe
in north america
etc and not only is that you know in the
classic case of like oh there are
actually three grids in the united
states
but also comes to understanding those
dependencies and other items
that allow for the grid to function in
the way that it was designed to
that means that we need to invest in
security visibility
and redundancy across all phases of
operation in order to
really be able to respond to or counter
high-end threats to this environment
part of that means making sure that
we're increasing our visibility to try
and match threats
if we have that increased visibility we
can then have greater assurance of
deploying root cause analysis on any
incidents that may emerge like those
frequency deviation events we talked
about earlier
and by applying this continuous analysis
of our own environment
we could then marry that to continuous
analysis of the threat environment
what lessons can we learn from what
adversaries are doing
out in the wild such as the targeting of
entities like nso e
or alexan and pipeline operations etc
based on that understanding fusing
internal and external perspectives
we then adjust defenses in light of that
knowledge and ideally begin
sharing that knowledge not just with
others within our sector such as other
power generators or similar
but also sharing data with partners and
other entities that make this system
function
uh at a massive scale emitting
the balancing authorities and omitting
fuel suppliers and transportation
functionality
means that we're essentially extending
vulnerabilities out beyond where we can
really control or see them
and allowing for sort of roundabout ways
of impacting
electric utility operations
another part of this and since we are at
defcon it's important to
bring this up is that we also need to be
a little bit more creative when it comes
to our security testing in red teaming
just saying like oh the bad actors can't
get to our generator controls that we're
safe
omits ideas like some of these stability
attacks and other
items where we're looking at attacks
that originate outside of our own
perimeter
how do we identify and test for these
scenarios
such that we can identify when controls
exist and what controls need to be
improved
as well as for evaluating things like
our business continuity and disaster
recovery plans
should one of these scenarios manifest
so that service can be restored
in a timely fashion so this is a very
short talk
and we only touched on things at you
know the 10 000 put
foot view i hope this was very
insightful and really identifying that
from a defense of electric utility
operations
it's not just about the utilities it's
not even just about transmission
generation and distribution
there's an awful lot that we need to be
concerned about and making sure that we
have eyes on
so with that i hope that this has been
enlightening
and uh leading to some interesting
thoughts about where to go forward
this few resources available on this
topic have also written and presented on
these topics elsewhere so plenty of
other things out there that can be
referenced
but with that if there's time we will
try to take some questions
so thank you