hi everyone my name is dan gunner and
today i'm here to present to you
consider the data source a journey
through an industrial attack
[Music]
disclaimer up front on this the there
are a few companies brands and
trademarks in here the views are
completely my own
and all information is used at your own
risk
so every information security journey
really starts with preparation and when
you think about it
preparation is the calm before the storm
when you're dealing with an event or
with an intrusion
and during preparation there's obvio
there's tons of different controls you
can do there's things like host and
network monitoring data centralization
all of the standard things we're used to
there's offensive analysis and stem
simulation red teaming penetration
testing all of the other
similar services there and you know
there's the process side too
so security is people process technology
but all of this makes up really that
range of events you can do
during preparation um but at some point
you know stuff's going to happen and
when stuff does happen
you know preparation is still worth it
you have to be ready though for
hey there's a lot you can plan for
there's things you can model
there's approaches that you can kind of
run through
and scenarios you can run through but
there's never certainty that that's
going to cover
what you know the full scope of what you
think it will
um this is a quote from a while back
right
um but you know it's it's held true
whether
it's been something digital or whether
it's something now
um you can plan for kind of that first
encounter
but once you get past that first
encounter there's always going to be
those
unknowns and those details um that'll
catch you by surprise
and so you know it's kind of easy to say
as you're planning for that first
kind of for that first iteration to say
okay well
you know i'm going to i'm going to put
all of my eggs in this basket right i'm
going to
you know come up for really focus on
this one control
and just kind of stick with that because
hey you know no matter what i do to
prepare
it doesn't matter there's also risk
though to
not always constantly improving or to
doing that because
you know while there is risks and cost
action i mean you're risking sometimes
your reputation you could be risking
your job if you get
breached and someone gets fired over it
but in the long run you know there's
really this belief that you know far
less
or the the risks of long-range fixing
things
is actually less than the risks of not
doing anything
particularly in our space right i mean
there have been industrial intrusions
for a while on the screen is
really just a handful of even the ones
we know about there's a lot of
intrusions
that likely we don't know about we
haven't heard about either companies
have
kept it in for reputation um or just
honestly we don't know about it it was
never detected
and you know sometimes um so stuff can
happen
and sometimes literal stuff can happen i
mean here's an example that
some people talk about if you've been
around the community for a while
um but others don't know about this so
back in 2000 and marucci shire
there was an inside threat um he
this uh person wasn't happy because they
weren't hired
for a position there was some kind of
some
employment drama between them
and you know with his technical
knowledge and with his abilities he
actually ended up
doing a series of wireless a series of
rf attacks um against
this public sewage it ended up spilling
sewage a lot of places
this ended up in kind of a knife fight
actually between him and other operators
on site that we'll talk about
a little later some of the outcomes for
that but really what happened
is you know he he had the equipment he
had the knowledge to access this
system remotely so what he would do is
drive
you know he was driving up to different
sites
connecting wirelessly there was some
stuff and if you read through the court
documents there was stuff that he had to
know to actually connect
um his inside knowledge did help some um
but what's interesting when you look
through this
is actually you know the knife portion
was
you know through this network he was
just using uh legitimate applications to
generate the traffic well
they'd be like okay well this traffic's
coming from site
14 and so he would emulate okay i'm
going to
configure this hot my my software to
pretend to be
site 14 and mess with the pumps um
and so an operator was like hey this
isn't right someone might be messing
with us well
they changed to site three um
in changing to site three they noticed
hey well the actual plants now
um you know site three is now trying to
do it and i still have this
rogue site 14 off in the corner
how this was actually and you know
what's interesting there is that's kind
of really the first known example of
where we
see an operator that's actually um
you know detecting these events um but
this is all to say that
stuff literally happens in this domain
and it has for a while i mean 2000
is 21 years ago if you do the simple
math there so
um this is all to say that
response is inevitable and this isn't to
i mean i'm not saying this to sound
hopeless i'm saying that
in today's world there's not an if
um you don't control the if and you
don't control the wind
but what you can control if you're on
the defense side of this
is how fast you can detect it and how
fast you can respond
to it and so we're going to really focus
on those two areas as we talk through
the attack
there are many models for for
modeling attacks one of the popular
adversary actions one of the popular
ones and hot ones today we're going to
use this miter attack
um i'm sure most of you are probably
familiar with miter attack for the
people that might not be what it is is
miter attack breaks down
adversary tactics and techniques
into a series of buckets so what they do
is they've taken
events in the past modeled them out and
tried to kind of
cluster the activity or the intent
throughout the whole attack cycle so
attack definitely
google it look it up it's a pretty
big knowledge base there that you can
use
what recently happened specific to the
industrial community
was the triton evaluation so for those
now familiar with crisis and triton
there was a breach into middle eastern
oil
site the compromise to safety system
you know a year or two back here maybe
three years
back what mitre did
and miter does these attack valuations
where they bring in different vendor
products
they um run the same scenario through
them
there's technically no winner but what
they do do what they do
is publish out the results to kind of
show hey you know
here's kind of this central mitre
created scenario
um and here's what we saw each of the
tools
um they don't declare a winner they say
hey you should
declare you it's up for people to decide
basically how people do
um and so what miner did here is they
calculated
or they uh they emulated the behaviors
of what was known about
um that intrusion into that middle
eastern um
uh gas and oil site
and how they did this is they they had
real hardware
doing this so they set up the control
system it was a rockwell control system
uh
the burner management environment the
graph you see there is from the miter
site
um they set up the control system
components there
and they did simulate some of the
physical process so obviously they might
they weren't um producing barrels of oil
i don't think but what they were doing
they had the real
software connected to a simulator there
to try to get
you know as accurate as you can within
you know a reasonable budget
um while still exercising those security
tools
what you have there what this shows is
that bingo card and again we talked
about
attack being that the collection of
tactics and techniques
um you know this is kind of that bingo
card and green is what they exercised
and we'll talk through
different ones of these shortly um but
as you can see they picked
really a portion of the attack
techniques
tactics and techniques to choose against
this
you know as we get into the scenario of
what they emulated and again this was
based on that trisys flow if you look at
the link at the bottom of this slide
it talks through um you know how they
throughout this scenario
we're going to start from the attacker
point of view so as a defender
you might catch the initial compromise
as the route into it
chances are as a defender you'll see
later on we're going to go with the
attacker perspective just because it's
more chronological
um and so the one of the early things
that they
that miter emulated through this was an
attacker using
you know valid credentials that they
stole right so maybe they spearfish
these maybe they got these um
through watering hole through other
means but what they did is they jumped
from the intersect enterprise
side into the industrial side of the
plant they used rdp over 33.89
pretty common this can still be enabled
at a lot of sites a lot of people allow
this
over the firewall still and they said as
this that it
is you know standard operating behavior
um what they did with this was they did
a program upload
um save some files um and we'll talk
about this a bit more in there
you know one of the first things that
we're going to point out as we look at
this and the importance of data
is out of the bat you might say hey well
you know going from one network trust
zone to another
um maybe i can just do this with bass
lines right um
it's rdp and that's true bass lines can
be helpful
um but it depends on those thresholds
right um
you get a lot of rdp from the box a to
box b
if box a has no reason to talk to box b
then yeah that's really good baseline um
you know if box a though is an
engineering station inbox b is a j or an
engineer station on the corporate
network
and box b is actually the jump server
inside
you know it's going to be harder to just
do a
you know matter of fact well an engineer
talk to a jump host okay cool
was it done out of business hours was it
done when that engineer was off shift
you know um you know you have to
consider
what actually is inside of that baseline
and how you drive it
you know after hours is actually really
cool and if you can find a way to
get your shift schedule into there you
can get better baselines right and what
i mean by getting your shift schedule in
is
trying to remove it to where you can do
it human-wise you're going to have
problems scaling
but if you can get your baselines better
by hey let me go ahead and import a csv
of my operator schedules um you can
begin to get better baselines and
this is where it can be futile other
futile otherwise because
if if the attacker is acting within that
mathematical baseline or within the
logical
the logic of that baseline you're going
to struggle with this
and you know another important thing to
say on this is and we'll see this later
in there
is um host sources can provide
details that the network sources can't
see this is a really important takeaway
from the miter attack eval as we push
through
and definitely something you should
investigate both from miter's results
but also just as you evaluate products
and look through what people are doing
when we talk about data sources today
we're going to really talk about three
areas we're going to are three different
types um this is how we're bucketing
them
we're going to talk about process
network data this is a lot of
when you look at the industrial market
what you see so passive network analysis
this
includes people that are doing kind of
netflow style
summarizations your five tuple of you
know source and desk
ip and port and then protocol
including down into some of the
summarization so
when you get into looking deeper into
the protocol
process network data is that really
efficient
way of um you know digging into
some of the fields that they decide to
extract why it's the process side as the
vendors decided like hey
you know we're going to write logic on
this field or it might not be a vendor
vendor or open source product decided
hey let's summarize this
specific field in the protocol when we
talk about process network data the
efficiency here is analysis and storage
um there's only so much processing power
um
that you have and especially as you you
know ruggedize a box and you push it
forward in a plant
you know you can't often have racks and
racks of servers looking at a single tap
point
at some point um you know you have to
you have to cut data down in the storage
size too
obviously storage also can take up quite
a few racks um
if it's not process data depending on
the flow of that
the con on on this is encryption so
you know with the 3389 and with
like tls with https um
if the data is actually encrypted you
can see the fact of communications
but it's going to limit you later on if
you actually want to see
what was communicated what was the
context inside of it
and also protocol support so like we
said for summarization it can depend on
you know how deep and what um
you know in a protocol that's actually
pulled out that the logic is written on
protocol support's really important
especially in the industrial field when
you're dealing with
very proprietary protocols some that are
known about and the whole spec might not
be known
um or there might be you know protocols
with products that you don't
even know about and vendors don't talk
about much
process host data is the next one right
so this is
things that come after endpoint agent
collection this could be things like
windows host logs this could be things
like
you know av logs your endpoint
protection
um application logs really anything you
can pull off the disk
where this is processed again is it's a
log source
so um you know there might be some
filtering in there
like you filter on the network side to
extract
out the data of interest um so again
the pro here is it's not network events
and what i mean by that and i don't mean
to that sound like a really vague
inverse of
network data what it is is you can begin
to see like hey
here's information on here's a processes
command line you're not going to see
that on the network here's
you know other other um you know file or
process
metadata when you look like sys
internals and other tools
um some of the cons here are
configurations so when you get into host
data
as the number of hosts grow you kind of
have this maintenance tail that you have
to deal with um
and also making sure that your
configuration um
and even managing your configuration can
be a challenge
um and then integrity right so
you know when you're listening to data
over tap point
there are things an attacker can do to
try to
screw with the data you're getting with
host data if the attacker is on the host
indicator removal is its own actual
miter
ics ttp attack ics ttp there
so with host data you do have to
consider hey is
is root getting involved am i hooked
somewhere
what are the issues is this data i'm
getting the true data or could it be
manipulated
um third data source we'll talk through
is our will uses raw host and net
network data
unlike its process friends this is your
packet capture
your disk and memory image the pro on
this is it's not
filter limited so you're not getting the
opinion of the uh
you know open source project or of the
vendor of what they decided to put in
there
the con on here we talked about the
processing space and
um you know um yeah the processing
capacity and disk space needed
um the con here is you do need a mature
analysis pipeline because the more data
you bring in
you have to be ready to officially deal
with that
that is kind of the pro con of
not dealing with filtered data but the
real pro again to go back to that
is there's a lot more analysis
opportunities as you dig into that
so digging farther into miter end of the
miter eval
starting with initial compromising more
than network data is often needed to
dissect behavior is the takeaway here
the criteria down at the bottom so the
criteria to score on this minor point
was
seeing tcp 3389
so again you know if we talk about
process network data towards the top of
the slide that's something i can see
i'm going to see that tcp on udp traffic
potentially
and i might see the username with rdp
you can get the rdp cookie that
sometimes includes username
depending on if the network tool you're
using is actually extracting that and
you actually have to see
a certain part of the rdp handshake to
get that
that can answer the first part of
scoring but to completely have scored on
this
and you actually had to have the second
part which is
you know via the ms via that specific
process and having the user
may or may not be present this is where
the network data helps but
you need that host data to say okay
here's the process information here's
the login record with my
with the username if you go through the
miter results you'll see some of that in
there
third one on there raw host and network
you can of course pull this
out of memory out of disk out of the
pcap
it's also going to be there but again
you're going to have to
um you know be dealing with a tool that
either processes
that for you efficiently um or it might
take you a bit of time
um or you keep if you're doing it
manually you're going to have to keep
the number of boxes you're doing on it
low
as we get outside of initial compromise
and we start to talk about persistence
um i'm not going to go through every
step of the miter
of the attack we actually only chose the
few that we wanted to talk about to
really
bring out some salem points on this but
for employing persistence what was done
in the attack
value eval was installing a scheduled
task
and then what this scheduled task did
was initiate it or
reverse shell and they did this ssh
reverse shell over
port 445. why they did this was to get
past firewalls so if a firewall is not
actually application aware if it's just
doing it off the port number
this would get past it they were trying
to guide it as
smb traffic and we'll talk about this in
a minute
when we hop into this um you know let's
go into the first criteria right so a
scheduled task was created
it's not legitimate it was imported into
the task manager
for to score on this and the minor
attack eval
you would not have gotten this point
right if you were just going off process
network data
um and this is where i think mitre
actually does a pretty good job because
if you
you can look through now the results and
say okay well who's actually
processing host data um the link at the
bottom actually
i'll point out um should point you
straight to this employee persistence um
to where you can see how the different
vendors did but in this case
to score here it's like wait okay who's
looking at cessnon file create a lot of
the
vendors that scored on this were using
sysmon file create or they were doing
some type of command line analysis
command line analysis coming out of the
fact that the windows event logs do give
you the command line
all of the competitors were scored out
of windows um
out of the windows event logs um again
this is where raw host and network data
comes in
but it raises the question and really
you know
brings to the point what if no network
data is generated by the tactic
definitely something to consider and
also something to consider here is you
know if all you're responding with
is network data how are you going to see
this right and what i mean by
that is okay cool i have a network alert
um you know maybe from initial
compromise maybe from a
later step well i'm going to want to
correlate this persistence and so how do
i correlate this persistence to
that network event that i got on either
side of this tactic
the approach i offer for this and the
approach of mitre showed is
definitely consider those windows event
logs
the point we want to show here is low
collection diversity creates blind spots
like we said to correlate the whole
stream of attack
you're going to need window or you're
going to need network
and host data and
you're going to need um the process and
capability to
um be able to analyze that moving on
so we talked about that reverse shell
being created
um and it going over port 445 um and
that passing of firewall
rules this is something to where you
might
see it on the network side right so
there is an app player mismatch um
sara cotto one of the attack um
participants caught it via seracotta
rule you might
catch this by just saying hey i'm
looking at the protocol going over this
port and it doesn't look like smb
it looks like ssh that might have caught
it at the process network data level
on the host side the windows event logs
and specifically what was scored in here
was event 4672
um that's you know that's that's where
the participants found it in here
again you can do it with ron um raw host
and network
there's lots you can do there again the
same scout
scaling issues apply but this is where
details matter
because you know this might be a weak
signal where you're like hey i get a
whole lot of
app layer mismatches on my network side
and in hindsight i can see okay yeah
this was part of persistence
but you have to be ready to deal with
okay
you know what's my threshold of dealing
with
app layer mismatches if i'm just using
network security
or if i'm using both or network data
or if i'm using both network and host
data i can say hey there's a mismatch
and there's this really weird command
line down here um
you know that's that might be a more
that might be a better threshold where
you're like hey this clearly isn't right
you know another example where details
matter is when we got into collection
and discovery or when you look at the
collection
discovery on here so again an attacker
used a custom executable for a stealth
scan on the network
44818 is a port associated with rockwell
for those not familiar with rockwell and
what they were doing here was uh
identifying devices talking enip
which is the protocol that these devices
talks they did this multiple hours to
limit detection
process network data right your baseline
again going to the baseline threshold
thing
the baseline threshold might have helped
you here
the other interesting thing to point out
here and
a little trices knowledge and where
mitre
did a good job evaluating or uh
emulating prices was uh the attackers
did
use uh what's called pi installer pi dxe
basically it's a way to turn
python scripts into cross-platform
capable
binaries you can if you look at the
filed
metadata going over the wire of how pi
dxe and pi installer works
there's you can there are yara rules and
there are certain rules out
there to look for these flying over the
network so
in dealing with this and process network
data space that's
definitely something to look for in a
detail that certainly matters there
on the hosts on the hosts data side
again this
is one to where command lines can be
super important
um you might do actually a baseline of
command line deviations in this case if
logixmaps.exe is an existing good
binary and you know the command lines
that are normally run with that
if the command line changes and the
network behavior changes
you know that might be in a more
advanced threshold
that you decide to go out and look at it
when you go down to the raw
host and network data side we talked
about yara for process
this would be the same deal if you pull
a hard drive in it or if you pull a disk
image or if you
pull files out of memory and start
running yara against it
you know if the process is running you
might actually
catch um you might catch this tactic
um you know also in raw disk
that way but this is another case where
details really do matter because again
it's
you know here's a here's a binary that's
named uh
or binary name that i accept but wait
it's pi installer where's pi installer
used
um there are vendors out there um
there are industrial vendors that are
using python um i don't know of many
using
pi installer or pi dxe at this point um
and so that's where there is you know
kind of those baselines or
um d opportunities to apply detail to it
as we move on um you know details at
that network
level don't just matter as we get into
expanding access um here an attacker
um you know did scp over 2223
to move some tools um and then uh
um yeah yeah and so when we talk about
process network data
our challenge here is going to be
um the data's encrypted so there is the
protocol mismatch again
if you understand enough of the protocol
going over port 2223 you might catch it
that way you're not going to see the
file contents so like the sarah cotton
yar rule here won't work as much
um on the process host data side and
this was interesting to my
to the attack eval there were a ton of
file creation events
i know hold on uh what you might be
thinking on that
um but file creation events are there
and if you look through the results
you'll see
a lot of file creation events were
created on the raw side you'll see a lot
of
or you could do this via file metadata
let's go back to that file creation
events um
certainly depending on the size of your
network and depending on the capability
of your collection stack and your
ability to analyze that
file creation events are definitely
interesting
that being said you have to filter them
because the next question
on that is can you track every file and
registry right to this system without
being flooded
um again certainly um
it's it's valuable to bring those logs
in it's certainly valuable to log those
but
you need to consider that those
baselines and thresholds that you say
hey wait this file right shouldn't
happen here maybe it's a file right to
um a weirdly named directory directory
maybe it's a file right to something
like c
windows you know system32 that would
definitely be
potentially a weird file right um but
you know context things like that
because tracking every file and registry
right
won't scale you're going to flood
yourself out with that
where where you know what is important
is and we've said this a few times
throughout
you know as you as you turn up your
automation and as you
dig your baselines deeper
you know you're going to need context to
scale scale requires context
some of this can be done on the industry
level some of this
might be done specific down to your
plant level but
at the end of the day to scale this and
to really detect
sophisticated adversary like was seen in
trisys in this
that scale is going to require context
and and even that that context might not
just be
technical context so as we moved into
infecting the safety system control
logic
um in this case um you know an adversary
initiated a program upload action um
and for those not familiar program
uploads and downloads are actually
um a little counter-intuitive to people
first in the industry so an
upload is actually going from a
programmable logic device to your
computer
in the challenge you can see here they
did it between the safety plc and the
safety ews
download is actually pushing the program
logic from the
engineering workstation is what an ews
is it's pushing it from the engineering
downloads pushing the engineering
workstation to the programmable logic
controller
in this case right we have two boxes
that
should be communicating this really
right so the the engineering workstation
it's probably in the established
threshold that it should be talking to a
plc
um so in process network data you're
going to see this file upload
at least in this case you will so this
use the the
protocol rockwell uses is called sip
sip you can buy this spec there's also
versions of the spec that float around
the internet
this is a known protocol not all devices
will you know the protocols
used for file uploads some of the
advantages of some of the vendors is
they do actually dissect some of these
very proprietary protocols a lot of the
major
vendors do have support on this but you
know you might see
on process network data you might see
evidence of the proprietary file upload
command
um in this case again it might not
trigger a threshold because these are
computers that should talk it um you
know but
maybe you have multiple thresholds maybe
this happened in the middle of the night
when engineers aren't there maybe you're
doing more correlation on there
where this is also helpful can again be
in where people are scoring on this and
the challenge
is the windows event logs with the
process information
and when you get down into this some of
this does
begin to get into getting into packet
captures with the industrial traffic
this does you know depending on the
protocol again
you might need reverse engineering
skills you might need to
figure out what the protocol actually
looks for and do things like riding a
lua or um you know a lewis dissector for
wireshark or other things if you're
trying to do this manually
um but in this case too the point to
prove here is outside
validation might be required an easier
thing is to just say
hey engineering team we saw a program
upload go between
you know this is kind of weird you know
with or
you know we thought this was kind of
weird did you do this
um and you can say and they might say oh
well no we weren't there
and you can double check that with you
know closed circuit tv if it's if it's a
monitored site
um you know door access readers you
could do it actually with the windows
event logs of who
who logged in when and trace it down if
someone on vacation did it um
and you know either their accounts
compromised or it's not them
but the point here is outside validation
might be required to
prove these out with certainty
um you know moving into the last two
steps of what we'll talk about today
um protocol depth doesn't matter until
it does
and when we get into the disabling the
safety function again we are in
very specific operations very specific
tags or data fields within the device
in process network data when we're
dealing with
you know in this case it's an adversary
initiating a right tag
action they're writing to a very
specific
part pardon memory of that device with a
very specific
value to get down to this level um
you have to do deep packet inspection of
the protocol
in this case it was a sip and ethernet
ip
um again it's important to understand um
the protocols in your environment and
what um you actually have visibility
over because
even if you take a big product line like
a you know honeywell experience like uh
emerson delta v or mrs innovation
there's a lot of protocols
and so just saying like hey we cover
we cover this product line no you need
to know like hey
does this product line is this product
line the one that does my program
uploads is this product line
you know or yeah is this protocol the
one that does my
program uploads is this protocol the one
that's writing tags to the device um
and even what's enabled on the device
right so you know i've worked with
devices to wear both
enip was enabled and modbus so you know
even if you're watching enop over here i
can do it via modbus over here and if
you're not watching you're not going to
see it
um so understanding the deep packet
inspection to
understand these rights to tags is very
very very important
this is a case where host data
a process host data isn't going to help
you so
some of the vendors do use the
application logs if they use the windows
api for logging
you might be able to go into the windows
event logs
into the application log or um into the
logs they write to and find that
you can certainly do this with raw data
but or with raw
packet capture but again you're going to
need um
to either have a dissector to have a
little bit of reverse engineering skills
um depending um on the protocol and how
the tool was written
but protocol depth does matter
uh finally the last one that we'll talk
about is really the last step
when you were looking at the ics eval
this was on a burner management network
these tag rights towards the end is
actually how mitre was emulating the
um effect on target phase of this
um towards the end there was another um
group of tag rights that happened um
very specific changes to the
devices what we see here is air damper
settings and
cascade control removal again process
network data
shines here assuming you have that
deepak and
inspection and you can see the tag data
and
you know what's interesting when you go
through the uh miter
eval results on their site miter does
include these screenshots of what
the vendors showed in their uh in their
uis
and you know it's definitely interesting
because one of the things you can do is
see hey well
what vendors were actually able to see
the values that were written in and the
tag names that they were written to
um you know that's that's definitely
something when you look at mitre to
consider to do
or if you're just doing your own
evaluation or process um
but in this case even if you're not
evaluating a product
um to see this type of impact um you're
going to need to go down here so
if if you're doing it yourself it's
still important to say okay
with whatever i'm home rolling to do
this i need to be ready to do this um
you could also do this with static and
dynamic analysis as you're going through
stuff so throw it in the same box throw
it on your digital twin or range or if
you have a device
but protocol depth at this point really
does
um matter and at first you might not say
well
you know why would i care about this
specific tag and again this goes back to
filtering well
i'm going to filter out tag names
because that's like if i track every tag
there's no way i'll store this
that's right and you know that's right
to a point it's right to the point to
where you need it like in this case
where
you know the protocol depth does start
to matter
you know something we want to wrap up
and something you know i want you to
leave with with this
is when you look at the ics attack valve
it's really easy to enter this with
hindsight bias and you know
kahneman quote hindsight bias makes
surprises vanish
what you see here on the right of the
slide is the scada system faults
at marucci shire so that sewage plant
that
overflowed um in a normal state so
before those february
before the january february sites this
site had about two
two to four alarms a day was the average
when the insider started um the wireless
attack they went from
two to four events a day you know up
into 10
15 20 you know you see a spike over 40
there
in addition they were having pumps that
weren't turning on
they were having pumps that are pumps
not turning on when they should they
have pumps
um you know they were turning on when
they shouldn't there was a lot of
weirdness
um and if you look in there this unknown
system faults are red and it's about
from
january to the end of april all of that
time there was a lot of weird going on
and there wasn't this correlation
um that this was that this was an
insider that was uh
you know connecting in and causing these
faults on there um
only when did only when that operator
started getting suspicious
and getting in that knife fight um you
know were they able to say okay well
we know this guy we know he has the
knowledge ultimately they ended up um
you know suspecting it was that guy they
hired private investigators to
follow him they found out he was
actually park near
sites uh they sent the police and
arrested him found the stuff in his car
um he was sentenced i think to two years
in jail and
like a 13 or 15 000 fine um
but you know it's a prime example of
it's easy in hindsight bias to say oh
well yeah my threshold would have been
there
um but in cases like this um you know it
really was the operator that saved the
day
and it was a non a non-um
you know a non-technical um
it was a non-technical trigger that even
caused
um you know the due diligence to look at
that it's important to consider as you
look across this
where um you know those alerts might
come in
and where those alerts that actually get
you to
you know figuring out where the event
starts and how to respond to it
all of this at the end of the day the
success depends on detection of the
chain of events um
you know you might say well you know
we'll start with network because with
network i just have to find one of the
links of the chain
and that's true you might find some of
the links that chain just going network
you might find some of the links of the
chain just going disk
but you have to have enough of the chain
of events
to properly isolate to properly contain
and eradicate um because if you don't um
you know ultimately you won't actually
secure your network on that
um and the correlation of events right
um
if you can't correlate events together
like we saw there were multiple steps
where both
post and network data was needed if you
can't correlate you're really going to
struggle
the right data is going to put the
dumpster fire out quicker and i put
right in quotes
because like right there's no solid
definition to what the right data is
there's a lot of difference between
networks between
plants but not having the right data can
limit your detection
entirely um you know if you if you don't
have the data like you can't
you can't collect data from the past
that you didn't collect before
um you might have some metadata or other
options um but
i mean um you can't go back in the past
and get data
you never got so it's really important
to consider both the host and network
data that you use as you go across
how you can improve your success
conditions we
offer kind of six steps or six
recommendations for that
you know what do you know about your
environment and your visibility how
reliant are you on host only or network
only how deep are you going in the
protocols how do you know your
devices how they behave how your
operators work in the plant
understanding the people process the
technology the full scope of everything
you do
is really important because it's really
hard to set good baselines if you don't
understand yourself
and you know you can work with you know
you can work with a lot of people to try
to do it but
ultimately you're going to know more
about yourself than they will
and then it goes into constantly
improving right so how quickly
can you dig into those data sources if
you need to
and if you find out you can't dig into
those quickly you need to go back and do
it
or figure out how to solve that um if
if going through the miter ics eval
results um
you know you say oh yeah we would not
have seen that or hey you know there's
good opportunities here
um definitely use those preparation
level things to
improve before the event actually
happens redundant sources right
so you know we talked about host logs
the risk being that someone can mess
with them but
you know you need to have redundant data
sources to have that failure margin
um you know i've been multiple places to
where
during an incident or before an incident
someone
you know lost their entire splunk stack
or they lost their entire seam
um you know if you lose your entire seam
or if you lose a part
of what you're responding with or what
your visibility is
you know if you lose enough visibility
or network you need to address that you
need to have situational awareness
even through a failure margin the
process
side is designed for failure to a point
you need to design your security side
for
failure collecting the right data like
we said you need to understand your
collection limits um
you know you might not be able to you
might have
a very high throughput bandwidth channel
you might have
um you know collection challenges
because it's out in the field and you
can only use rugged options that don't
that limit the uh processing or
collection power you have
you need to understand what right data
is for you
both for how you analyze data and for
the realities of your operations
um you need to understand your tool
limits so you know
how you use your products right you know
every product's great when it's getting
sold to you but how are you
actually using that for analysis and how
does that make you better and fit in
that overall strategy
and then challenging your status quo
right look at your baselines constantly
validate them and check them you know
use you know tabletop exercises use red
team assessments use
you know um use the smart operators i
mean we've had cases to where
we've worked with customers and talked
with them um
and you know just in a conversation with
a customer on being like well how would
you do this you know
the whole chain of events and the whole
chain of success starts with a
five-minute conversation because they
know
what the status quo of the environment
is better always challenge the status
quo both
you know of your technical um you know
of your
you know technical stack but also your
baselines and how you do things
so the takeaway here preparation matters
so it's not a panacea it does accelerate
response and gets you
to you know that faster mean time to
detection mean time to response which is
important
um but you need to also prepare for when
you have to act
um silver bullets can breed complacency
right it's important to
validate that the tools that you know
what you think your tools do actually do
what they do and that you can use them
in the way that
they're being advertised to you response
is inevitable right
you don't choose if and when you respond
but you can control the speed as we said
data diversity matters right we saw
multiple cases to where both the host
and network side was needed
to um to get to the conclusion or to in
the minor case to actually score the
point
um context is key right
tracking every file right you won't i
mean if you if it's a small network you
might be able to do that
if it's a big network you know you're
going to have a big storage bill so if
you're putting that up in aws it's going
to be a big aws bill
you need to think about the data life
cycle though around your data to say
okay
you know let's store file rights of this
subset of data let's store
command lines of this subset of data if
they start to and we'll store them for
this long and this
you know in this area and you know this
how you'll access them especially if
you're using cloud you can
consider well we'll hot store these cold
store these you know you can go all day
with the design on that but
at the end of the day what it's
important to do is not flood an
analyst while also making sure that your
um
you know the analytics if they're
automated are able to run over that
corpus of data
an analysis step doesn't matter until it
does you know we talked about
both the file context and the network
context mattering
analyzing sophisticated attackers are
going to challenge some of your
collection and analysis limits um
like we saw in the impact stage of miter
attack
um you had to dig deep in deeper into
this hip
protocol you know as you take this kind
of into the production world it's going
to be a challenge
because again you're dealing with
proprietary protocols depending on
how often tags are written how often um
you know how often how loud the network
is you know
you have things like you go into a
refinery it's not just one data bus
there's two data buses
i've seen even seen even more
complicated networks out there
um your analysis step and your
breadth of analysis all matters
and you should constantly be
understanding and stretching those
limits
so i want to say thanks for attending
the talk it was
it's really great this is actually my
first defcon talk to do
and it's great to do it with ics village
definitely stay in touch
that's our twitter and if you want to
work or if you want to work together
feel free to reach out um
you know there's always fun stuff going
on so again thanks and have a great rest
of the day