hello everybody uh thank you for taking
the time to listen today
my name is otis alexander i'm a cyber
security engineer
um at mitre
and i lead the effort for attack
evaluations for ics uh so today i'll be
talking about uh the importance of
detection context uh
in regard to our recently released
results for
minor ingenuities attack evaluations for
ics
so just as background
this is attack evaluations first
move into
a new evaluation domain industrial
control systems and we just completed
our first round of independent
evaluations for ics and the results were
released last month on the 19th
so the evaluations examined how cyber
security products from five vendors or
ics vendors armis
clarity dragos iii and microsoft
were able to detect ttp
associated with the trident malware
so
just as background for attack
evaluations our goals are to improve
organizations against known adversary
behavior by first empowering end users
with objective insights so they know how
to use the participating uh products
uh
providing transparency around the true
capabilities of these products
and driving vendors to really enhance
their capabilities
it's important to note that these
evaluations
uh
don't result in a winner we're not
declaring a winner uh they're not meant
to be a competitive analysis
we're simply documenting what we
observed in regard to detections
and it's just important
to remember across vendors there's no
similar way that you know we know of
analyzing ranking or rating solutions so
we look at how each vendor approaches
threat defense within the context of
attack
so for
this adversary emulation that we did for
this evaluation
we were emulating
ttp associated with
the well-known trident attack
and
we launched our adversary emulation
against an evaluation environment
uh that is our lab which functions as a
burner management system
so we launched 100 sub steps against
that environment and the real goal of
this was
to disable safety functions and to allow
for unsafe state to be induced so the
unsafe state that we were trying to
induce in this burner management system
was a release of gas without the system
tripping at all and once we
got enough gas in the simulated facility
uh we
went ahead and ignited that gas to cause
uh
destruction to the infrastructure
so
it's important to note here that many of
the things that we did were just
standard actions that were
leveraged against the environment
the adversary emulation consisted of a
lot of remote access
actions it
also from the ics perspective it
consisted of like status requests to
pocs or online edits to change
configurations of plc's or even forcing
points to
try to drive the process into a bad
state
so
if you want to learn more about what we
did i encourage you to go to the attack
eval site to look at our operational
flow and we'll go through the individual
steps
so
each vendor
that was participating
sent us a physical appliance
with their solution installed on it for
us to
install in our lab and they all received
network traffic was which was
distributed to them by a network
aggregator
uh in addition uh windows event logs
were centrally collected and then
forwarded via syslog to any of the
vendors that
had the capability to collect those with
the appliance that they sent to us
we also offered the opportunity to
actively pull the plc's for
configuration changes this is a feature
that some of the vendors have and we
wanted to see how it worked so for those
that offer that to their customer base
we offered the chance to do that but at
the end of execution so that uh that
traffic to the plc's didn't interfere
with the network traffic going to
everybody else
so important concept is uh detection
categories and it's important because
each of these vendors has their own way
of
describing uh detections they have their
own
uh windows and their ui where they
collect these things it may be called
you know alerting window or detection
window or even uh notifications or
something like that it also a lot of
times have a back end
which collects a lot of the atomic data
that's being parsed um
from the particular uh collector or data
source that that um they're ingesting so
um it's important for us to be able to
abstract that data
so that we can talk to about it in a
similar way across the vendors and
that's really what detection categories
are
so across the top we have our uh main
detections uh n a just means that the
particular participant
didn't have a solution to collect a
particular data source so we just write
n a by that
uh none means that we didn't see any
detection at all
uh telemetry is just minimally processed
data with
you know little to no context associated
with it a general tactic and technique
represent context that the vendor
product has added to
the detection
to kind of talk about why it's malicious
we also will include two modifiers which
are config change and correlation so
config change
is uh
added to a detection or to a point in
time to show that the
vendor made a change to their product so
they have to
come to us and let us know what that
change is and if we approve it then
we'll add that
to that particular sub step
that it
corresponds with
correlation uh which is going to be a
important topic coming up
is
i add it to a detection to show that
a detection was associated with another
detection or other data that had been
seen
uh and this can add more
context to
a detection so that you can better
understand surrounding events
um so let's talk about that a little bit
uh let's talk about how we can add more
core
context to
particular detections by leveraging
correlation and even leveraging
correlation across data sources
so
like i said before uh we were talking
about the abster emulation
this was
built off of a lot of standard actions
and
these actions are not malicious on their
own you really have to provide context
to them to understand that they're
malicious so
rdp ssh sftp if there's things that you
do in your environment those are just
standard actions that happen
uh status requests in
regard to
uh the sip protocol or online edits or
force points or you know the standard
actions that you see uh in control
systems are
uh not malicious in and of themselves
uh so it's important to add context to
them so that
analysts can understand that this is
something that really needs to be looked
at or
responded to
so to prove that action is bad uh we
really need to show
something else some some other context
we need to give some more information so
maybe it's that this particular action
will negatively impact the control
system
or it should not have been performed uh
at a particular time
or by a particular person or at all in
this environment or you know another
thing that we can do is we can
use correlation or tie it to some other
malicious activity
that it's related to and therefore we
have more context
about it
maybe potentially being bad
so
what we're really looking for is a more
holistic view of detections what we see
a lot is telemetry
and singular analytic detections
uh which all increase visibility in an
environment
but alone they don't always provide
enough context for ants to know that
they're bad or or that they should be
investigated
so correlation can be really used to tie
together the standard actions to
malicious ones
so
during uh
the execution and while we were
collecting evidence and processing
results we did see correlation we saw um
correlation
in our regard to singular data sources
so
um
windows events or network traffic but
not both at the same time so just to
give you some examples of
things that we saw
uh we saw windows host data sources
which were tied to related events
associated with the execution of
malicious programs
usually
consolidated to a single asset something
that happened on one asset so this was
good because
what it did was it built a bigger story
around a single action so that we saw
related events and we got more context
of what happened over a particular time
period
or uh how a parent process launched
under other processes
and you know a tree structure of events
so that we as an analyst can
uh really understand a a sequence of
events
uh the other thing that we saw were
events pulled together in a story format
more of a a narrative talking about
network sessions that may relate to each
other or
other information that provided root
cause analysis to really explain why
something may have been happening
both these are great examples of
correlation
over a single data source but what we
wanted to see is more narrative about
what was happening across data sources
to really tie some of this stuff
together so that we had more context as
an analyst
so
i provided a couple of examples of
things that we could
potentially do to provide more
correlation across data sources
so
if you look at our adversary emulation
it's broken out into sub steps
and
each subset has some criteria something
that we're looking for
in terms of the
detection
uh to say that this is related this uh
will count as evidence of
this particular step of adversary
emulation
so some of our sub steps are related
together uh and it's usually in in terms
of numbering but overall it's a a
similar action so for instance 4b
1 and 4b 2 are uh and 4 4 c 1 are all
pretty much
a similar goal of the atmosphere
similar action
uh 4b1 is evidence that
the executable s p client uh is not
legitimate so
this particular executable is
masquerading as something related to smb
but it's truly not
and 4b2 is evidence of an established
network connection over tcp port 445
uh between the adversary machine and the
control engineering workstation
and this is an outbound ssh tunnel uh
request through the firewall
and uh what we see here is the s p
client is actually plink
under the hood and it's creating this
outbound ssh
request
and then 4c is evidence of an
established network connection over tcp
port 3389
uh between the adversary machine and the
control ews
so if we look at all of these uh sub
steps
what we see is that they have uh
differing data sources so the first one
is uh we're looking for a windows event
uh the next one we're looking at network
traffic for evidence of the network
session
and then the third one we're looking for
a windows event again
so
what we kind of saw is like for 4b1 we
may see an alert that talks about
masquerading and
uh may tie together
certain processes uh that are related to
each other to correlate and kind of tell
a story about how these processes were
being executed but what we did see is
the windows event related to the network
traffic
so on the right-hand side what we have
is you know it's a narrative but it's
something that we would
like to see in terms of correlation
across the data sources to to kind of
tell a narrative so it says behavior was
observed indicating network restriction
bypass through rdp tunneling based on
the executable s b client i execute
masquerading as plink die execute
being used to create an ssh tunnel over
port 445
so
we don't expect this exact um you know
sentence to be
stated within a particular platform but
something
that uh
leverages
information from all three of these sub
steps
uh to correlate it into some
coherent story or tied together so that
the analyst knows
that these things are related and
that these standard actions are tied to
something
malicious in this case uh masquerading
for smb client
so i have another example here of uh sip
communication and uh correlation across
uh these types of events
so these span across sub steps 19 b1
20 a1 and 282
so 20 our 19b1 is evidence that a newly
created uh
newly created files from
a zip that are have been extracted into
the temp directory
are not legitimate so now we're kind of
dealing with this whole masquerading
again
and then 20a one is evidence that rs
logic 5000. that execute which was
recently uh extracted from this uh
uh was executed on the safety
engineering workstation and this may
also include uh things like
their uh parameters associated with this
particular executable and things like
that 2082 is evidence of an adversary
initiated get attribute single sip
request
for the status attribute associated with
the plc
so again we have three separate sub
steps with their own criteria and they
have differing data sources on uh the
middle column here so the first one is
associated with windows events as is the
second one but the third one is network
traffic that we may be uh looking for
so again we saw
a correlation for windows events
associated with masquerading or
scripting and different things like that
we saw
event trees being created or events
being tied together based on
certain criteria but what we didn't see
is
windows events related to network
traffic
so on the right hand side again we have
uh correlation
across the data sources and what we uh
would expect to see in terms of tying
together these events so
behavior was observed indicating that a
newly created file rslogix 5000 is
masquerading as an allen bradley
executable
and is an issue a sip request for status
attribute over the network
so now
we have this standard action again uh
that we may see in an environment a lot
this sip request for a status attribute
for the plc
uh which
may or may not be malicious we don't
know but combined with something that
relates in time or
some other criteria
to this newly created file that's
masquerading as an allen bradley file
uh now we can kind of build a story
about why this may be malicious and it
takes on a whole new meaning
to
an analyst looking at the screen
so
by leveraging these different data
sources and tying to them together we
can actually tell a better story about
why standard actions may warrant some
more investigation
so this really is our call to vendors as
you know provide more context to some of
these detections
we saw a lot of
good context provided in regards to
singer singular analytic detections we
saw correlation across singular data
sources
uh but we didn't see
um
the ability uh to leverage approaches to
correlate the textures across multiple
data sources so we gave some examples of
host-based and network-based
data sources but also plc
based data sources so that's the
capability of your
platform to
actively pull
plc's to get configuration information
maybe all these things could be
correlated in some form or fashion to
tell a bigger story about what's going
on
uh
in these environments so
this will not only improve the analysts
understanding of the activity but it
will also help uh you know make sense of
standard actions that may uh go ignored
or be misunderstood in these
environments without more contacts
and it really defines why a standard
action uh may be malicious so
we encourage vendors to
see how you can leverage
a correlation across these data sources
to provide more context to
uh analysts
and for those who are looking at the
results we encourage you to really dig
deep into them to understand
uh how these um
vendors are adding context to detections
um
what they're presenting to you as
analysts in the context of this
particular evaluation because that's
really important uh to understand you
know maybe what you will be wanting to
look at as an analyst uh versus
uh
you know things that may or may not
provide as much context so we think it's
very important to dig into the results
to learn more
so
that's all i have uh feel free to reach
out to me
i'll also be answering questions on
discord so if you have any questions
about the evals or process or some
things that we learned
please join me there to to
for a question and answer so thank you