thank everybody
for joining me for structured analytic
techniques for improving information
security analyses i'm rabbit
no really i am uh i'm sometimes also
known as jj
professionally if i need a boring name
for resumes and things um
i'm currently the head of product
security and security operations for
a company called latch latch makes smart
access devices
and smart home integrations and a number
of things for residential buildings that
are sort of
the internet of things variety we like
to think that
we make things that are a step above
what you'd probably typically think of
when you hear the term internet of
things
uh in background a former biometric
device
penetration tester i've done some
medical
security hospital engineering i
entered information security from the
field of epidemiology and public health
research
so my previous year and a half has been
a real interesting ride having some of
that
become national uh
you know stories international stories
my
informal introduction to information
security is when i completely wrecked
the pdp 1184
at my high school you never forget the
sound of the heads hitting the platters
on those old death drives that's
all i'm saying about that and an
interesting thing about
me uh i was raised around my father's
a very weird and large collection of
robots
so i'm basically like a budget version
of shinji akari
so what are structured analytical
techniques
structured analytical techniques are a
set of
tools tricks uh
fun ways to manipulate data that
can help you discover and eliminate
biases
they can improve the accuracy and
reliability of your
conclusions uh they can ensure that all
data is being equally considered and
including
included during analysis and they can
enhance your creativity
so where did this come from a few years
back
i was perusing files on the cia.gov
website as one does and i came across
something called
a tradecraft primer structured analytic
techniques for improving
intelligence analysis uh written by
prepared by u.s government so i never
found out who actually wrote this thing
but i assume it's you know one of these
people
um just ahead there's no connection
between me and the cia or me and these
authors
i just happened to be a guy who found
this uh so
yeah nothing yet i do think though that
maybe uh one of them might read it
eventually so just in case
so it's a good paper and also just to
point out uh
anything that the cia publishes with the
words a tradecraft primer on it
i am absolutely going to read that
so what portion of this is mine what
have i done
i initially thought about this mostly it
was rattling around in my head in terms
of how you might use it for
open source intelligence or for threat
intelligence so that's kind of a natural
fit just intelligence analysis
but as i like all of us i probably don't
do anything that cool most of the time
most of my time i'm spent
you know doing some more normal infosec
stuff for my orgs
and so i started thinking how could i
apply these same techniques to stuff i
do on a more day-to-day basis or
you know or sort of more mundane or
might have a more
widely applicable uh you know user base
uh there are three types of structured
uh
analytic techniques the first one is
diagnostic techniques
those are used to improve uh quality and
make sure that your analyses are based
on quality information
the second type uh are contrarian
techniques
these are for making sure analyses are
not biased or based on limited thinking
and the last type is imaginative
thinking techniques
these are for developing new insights
breaking out of the box or
seeing things from potentially new
viewpoints
so the first structured analytical
technique we'll take a look at it's
called just a key assumptions check
it's kind of an easy one to start with
it's very easy to describe
it's literally taking uh uh
thinking about all the assumptions you
made when coming to a conclusion or
performing an analysis
and writing them down creating a list of
things you've just assumed
you may review the assumptions
periodically as you do the analysis just
to see if your opinion of them has
changed
maybe you discovered you made an
assumption you don't remember making or
didn't think as of
as an assumption at the time or maybe
you know you need to do
some rethinking about some of the stuff
you had assumed in the first place
an example of how this might help you in
information security is let's say you're
replacing an old layer 3 firewall with a
new next-gen
you know blinky light layer 7 firewall
it's got
deep packet inspection and it does
protocol identification and all kinds of
cool things that your layer 3 didn't
what are some assumptions you may have
made that may come back to bite you
in having made that decision well you
may have assumed that newer is better
often is not it isn't always
uh you may have assumed that the default
settings on the new firewall
would block malicious traffic maybe you
know you
brought it out of the box and put it
into place and it's expecting you to do
a lot more tuning than you realized
um you may have assumed you really only
needed to worry about
incoming packets versus exfiltration or
outgoing
traffic uh or you
may not you may assume that it doesn't
pass traffic until it knows that traffic
is good or bad
that's the one that uh particularly why
i chose this example is because i've
encountered layer 7 firewalls that
allow 32k or 64k of data to move
before they decide if a protocol is bad
or not
and at the time as someone who was
trying to exfiltrate all that really
meant to me was if i cut my data into 32
or 64k
chunks i could pass right through the
firewall by just restarting the session
every 32 or 64k
and it would never clamp off on that
protocol as
being malicious because it didn't have
time
uh so another technique is a quality of
information check
that is basically where you go through
all of your information
and judge uh judge it for its strength
its weaknesses
its importance and its confidence
it's something you can use throughout
the analytical process and
you should be performing it probably
periodically
it helps you detect errors in processing
or collection of information
errors in translation or errors in
interpretation
it can identify attempts at deception or
denial strategies if you happen to
discover that maybe there's some
specific problem with your data and it
can assist in communicating the amount
of confidence
you have in your assumptions and key
information
so it's pretty easy to do all you do is
take each information you're basing your
assumption or conclusion on
identify which of them are critical to
that result
and then consider uh if it involves
interpreted information has that been
interpreted in the right context or with
you know
a complete understanding of how that
should probably be interpreted
an information security example so
you're working instant response and
analyzing log files sent to you by an
ops team
what are some quality of information
checks you could do
to make sure you're working with quality
source material
well you might check that the timestamps
are accurate they probably aren't
uh are you confident that the log hasn't
been tampered with is there some
mechanism you can use to
prove that it has some you know veracity
to it
uh are you confident that log is real in
the first place you know
maybe there's someone left one for you
to find uh
and lastly does does the log even
capture the type of information you're
looking at maybe you have your device
configured such that the event you're
looking to correlate
isn't even captured in that log those
are all things you could check to
improve
the quality so another technique
indicators or signposts of change this
is where you create
a list of events or monitoring targets
that will tell you if your situation has
changed
essentially you identify a bunch of
competing hypotheses you know ways that
a situation may go and then you create a
list of potentials at potential
activities or statements
or events for each scenario right if you
start going down one of these paths
you may list out what would you expect
to start seeing happen
and then you regularly review and update
your your indicators
uh to make sure you're watching for
these things and so if you start seeing
these indicators in your environment
you've got some idea of what path a an
event might be taking
so uh information security example uh
you may want to protect your
organization
from a system-wide ransomware attack you
think you're in good shape because your
org uses cloud storage and your domain
controllers are regularly patched
um you might uh what might be some
signposts
that this assessment you've made that
you don't really need to worry about it
too much
might be changing and you might need to
rethink that scenario or rethink that
risk or conclusion
well for one if ransomware attackers
start
leveraging or attacking specifically
cloud storage providers a lot more
that's a fundamental change to their
tactics and and their
you know ttps by your estimation
so that should cause you to rethink your
your conclusion
or what if there's a vulnerability in a
domain controller say in a principle or
where microsoft really
releases a patch that doesn't fully fix
the issue that would
never happen but knowing that exists you
would have to
uh then consider will your
conclusion or or uh scenario of
or decision that you were protected hold
up over the next one
six 24 48 hours
another diagnostic technique uh analysis
of competing hypotheses
so this one is uh basically identifying
all of the reasonable alternative
explanations
and then just weighing the evidence
against them to determine whether your
evidence supports or
or refutes these potential hypotheses
and this this is specifically done
without
conjecturing about the probability of
the hypothesis
being true hypothesis danger uh and
the goal is to take
your data uh your your bits of evidence
weigh it against each of the hypothesis
and decide if each piece of evidence you
have
either supports it supports that
particular particular hypothesis
refutes that particular hypothesis or
maybe doesn't contribute to it in any
meaningful way
you go through one at a time and you
create a matrix of
uh which bits of evidence support or if
you
which hypotheses you have with the goal
of
refuting or disproving hypotheses
not with the goal of proving hypotheses
and that's to prevent you from
uh grabbing on to one particular result
you might
favor for one particular reason or deal
is the most
likely you're not trying to do that
you're trying to make sure that you've
identified
all of the possible potential hypotheses
uh it reminds me of a quote a sort of
famous one when you have
eliminated the impossible whatever
remains however improbable
improbable must be the truth which is of
course from house md
uh an infosec example you're asked to
investigate the defacement of an
internal wiki page
someone has replaced all the tutorial
videos with links to hardcore form
the logs don't show any edits in fact
you don't see any activity on those
pages for several
years so how might you go about using
ach to
figure out or help you come up with a
conclusion as to what might have
happened
well you might uh conjecture that could
an admin
have made the changes and scrubbed the
logs
that might be possible could an engineer
have made the changes in the back end
and that's why your logs
don't match up could an external
attacker have compromised the wiki
system
in some way that you don't know about
yet or could the videos themselves have
changed somehow
and hopefully it's clear to a lot of you
that i'm alluding to a recent event
where
a company purchased a domain that used
to host fairly innocuous videos
um and on that domain now hosts hardcore
porn and as a result it's
showing up in a lot of old pages that
haven't been touched in a while because
it's true the actual videos themselves
are what have changed
so let me get to uh contrarian
techniques
uh probably the one that people are are
most familiar with is
devil's advocacy however i've noticed
that some people
some people understand the term devil's
advocacy to mean
arguing the opposite viewpoint or or
taking an intentionally contrary
viewpoint
to argue with in the context of the
original
government us government document uh
they actually
use this as saying uh it's more like red
teaming your analysis
it's taking your own analysis and then
having another
faction or you playing you know another
role in this analysis process
attacking your own analysis to find
weaknesses
things that might not be supported
things that might be
not quite as reliable uh as
you are using them in the final analysis
it's uh intended to allow you to test
whether your key assumptions will hold
up under different circumstances and it
helps you identify
faulty logic
so an information security example of
devil's advocacy
uh there's a bit of a scenario for this
one you're wrapping up a report
on what has happened during an event
you're mostly confident you've arrived
at the right conclusion as to how an
attacker was able to gain access to a
database and exfiltrate some data
but the stakes are high um let's just
say
if this happens again your company's
going under it's really important
you have to be 100 certain about this
given that it's so important
how what can you do to ensure that the
result that you've came up with
or uh your your hypothetical scenario
is as accurate as it possibly can be
well that's where you bust out
devil's advocacy so maybe you would take
one of your really good analysts
or uh maybe it's you uh sort of
functioning in this
this particular role but for for this uh
purposes of this one that
you have one of your other analysts look
at your report they specifically start
going through your report looking for
weaknesses in the things that you've
assumed
weaknesses in the evidence potential
faults that might disqualify some
infinite evidence that you're using
they just basically tear it apart and
while they're doing that to create a
report
of these are all the things that are a
problem with this analysis
and then they present that to the group
now
the group can take a look at that and go
now you know they can
they can argue with they say well you
know this really is founded because
of whatever reason or they might say all
right you're right we need to go find
some additional corroborating evidence
to support this one particular piece
because it's sort of a weak pillar of
the final result
and that helps you make sure that your
final product is based on the strongest
of possible determinations
from the process very similar
to devil's advocacy is uh team a team b
analysis a lot of people get these mixed
up and in fact if you do a team 18b
analysis wrong you can end up turning it
into devil's advocacy
but uh there's a slight difference in
that um
team 18b is used when you have two
potential
results or final conclusions that are
sort of equally probable or at least
equally
of equal confidence and you're not quite
sure which one
is the stronger one what you might do in
that situation is to take each of those
hypotheses
and give them to two different teams and
allow each team to pursue them
as their own analysis
one of the often uh uh
used additional sort of tricks in for a
team 18b analysis is to take
somebody who is in favor of one
particular outcome and put them on the
team of the opposite outcome
and it isn't just to sort of make them
angry it's because they may have
insights
or understand connections between the
data they're working with
that the other team hasn't realized it
allows you to sort of do some
cross-pollination of that understanding
of the issue
so how about you use that in information
security so you and your team are trying
to decide how best to reduce the number
of
malware incidents you're experiencing
within your organization
uh let's say you have limited budget and
time and engineering resources
and you can only really go with one
thing
one uh group of your analysts wants a
strategy involving
endpoint policies updates changes to
something on the endpoint
and the other group of analysts is
really uh thinks that the right way to
go is to put money into some
network wikibox uh anti-malware boxes
that can
stop stuff before it even gets to the
endpoint so one thing you might do
in that situation is you might decide to
give each of those two groups
or tell each of those two groups to run
with that idea and produce
an analysis why their solution is the
best option
why we shouldn't consider the other
option
and you might use the trick of sort of
cross-pollinating by taking people that
are an advocate of
endpoint uh restrictions and put them on
the network uh restrictions game and
vice versa and sort of get that um
that cross-pollination but the final bit
is you have sort of an agreed upon
sort of formal uh judgment method
uh maybe it's just you know the relevant
manager decides which one was most
compelling or maybe you have a
a team of experts who listens to the
arguments and they decide which one was
the most compelling or
who had the the best argument for that
particular
way forward it's a bit of a debate
tactic um but it can allow you to
pull the best out of two potential
in very close situations
so uh i impact low probability in that
analysis this is for when you have an
event
that would have dire consequences if it
happened but no one thinks it will
happen
it's very useful when decision makers
are convinced that an event is unlikely
but maybe haven't given sufficient
thought to the consequences
of its occurrence they've kind of
dismissed it because it's it's very
unlikely
it can be used to uncover hidden
relationships and it can help uh
analysts develop signposts from kind of
like we were talking earlier
uh which can provide early warnings of a
shift in
in the situation how do you do it you
define the high impact
outcome clearly like what is actually
going to happen if
this occurs uh and then you devise one
or more pathways
of series of events that could
potentially get you there you start from
uh known good and work your way towards
the high impact event that's important
uh when comparing this with a future
technique
uh you insert possible triggers or
changes that could affect the outcome
right maybe
it seems very unlikely because um
no one has the proper keys or access to
the proper keys so maybe a triggering
event might be
fees get released or you know keys are
exposed somehow
uh that might change this whole uh
scenario and now you need to rethink
our actual uh low probability it is
anyway you identify a a set of
indicators for each pathway that can be
monitored
and then you set about monitoring those
things so that you know
assuming you can't do anything to
fundamentally reduce this risk or remove
this risk
at least now you have a set of
indicators so you may get some advanced
warning of it coming
and be able to take corrective actions
an example of this say you're
concerned that your organization is not
taking the risk of having too many
unnecessary domain admins seriously
most of the domain admins don't need
that level of permission
but the stakeholders believe it hasn't
been a problem and it makes changes
quicker in an emergency
obviously something i've heard in the
past it also makes keep security from
being a bottleneck because they don't
have to ask for permission if they've
got
full threads to everything you're
concerned because they're of the risk of
a compromised admin credential or uh
what kind of damage an internal attacker
might be able to do
so maybe you conduct a high impact low
probability analysis
uh you highlight the huge impact that
happens if this very low priority uh
probability event occurs you identify
several pathways that could occur right
financials get leaked there's weak
passwords chosen by the people who have
these
maybe an internal attacker is having a
really bad day or get some news they
don't like you know
that might in fact impact things
you identify potential indicators that
these events are occurring right at
least now you've got some idea of what
to look for you may not be able to fix
the situation
hopefully you're working on the
organization to reduce the the number of
admins there are
but you've got some things you can watch
for you can look for indications that uh
leak credentials or credentials in a
week
you can test for very weak passwords in
some ways
there might be some stuff you can do
so a what if analysis a what if analysis
is often confused with a high impact low
likelihood analysis
but it really is almost a high impact
low likelihood analysis
done backwards with high impact low
likelihood analysis
you start at a base state and construe a
path where you may arrive at the high
impact event
with a what if analysis you start with
the high impact event
and then work your way back to identify
the things that would have had to have
happened
for that scenario to occur um it's
basically just
coming from the other direction so you
you start by assuming the event has
happened you select some triggering
events that may have permitted the
scenario to
unfold um you develop a chain of
argument as to how the output
output outcome could have come about you
work backwards as i mentioned
and you generate a list of indicators
for each of those stages
so now you've uh with your high impact
low likelihood analysis identified
things you might monitor or or consider
important for
detecting if it's starting or in
progress
you've now got additional uh indicators
for if activity is happening in each
stage and so maybe the event has
occurred and now you're finding it
before it works its way back on
the chain an information security
example uh
consider again that same scenario where
you're concerned about
potential for domain admin credentials
to become compromised
instead this time assume they're already
compromised
how did that happen right and so maybe
as a result of that
you might identify the same vectors as
before like leak pack
elite passwords or inside attackers
um however maybe because you're thinking
the other direction this time it might
occur to you that
maybe the help desk dumped some caches
off of something because someone logged
in with admin creds and
whatever it was was set up to cache that
uh you know
you might as a result of that decide
well
i'm going to add to the list of things i
can wash watch for or do
check for excessive ash caching
especially
domain admin creds and uh uh you know
maybe there's something you can do to
make sure you're catching if people are
using domain admin credits for like
needless local logins you've now got a
little more information of things you
might look for in your environment to
detect these specific types of problems
and on on to the uh the imaginative
thinking techniques i think these are
probably the most
obvious uh to people or the easiest for
people to wrap their head around
people are used to the idea of
brainstorming they've heard that
uh term a lot um
the the interesting thing is again the
source material talks about
brainstorming as a group activity
um in in obviously because they're doing
sort of structured
beams and things to bring about
brainstorming i think when we
just say brainstorming we we think of it
often as just
coming up with things off the top of
your head you know that might be
off the wall and that's true and you can
do that by yourself but in this context
they're talking about it as a group
activity
basically it's it's it's
quickly generating a range of hypotheses
that can then be refined
tested further developed or discarded it
just creates raw material you can work
from some ideas
things that you can then later go and do
you know um
test your evidence against and see which
of those you can discard as right off
the bat as well well that's not going to
happen
that couldn't be the result um but it
just generates you things to work with
gives you
more creative points to start your
analysis with
um typically in terms of a structured
analytical technique
this is done by assembling a group of
analysts with others relevant
uh or others with relevant knowledge you
free form generate ideas
which can plausibly fit this supplied
evidence and you never censor an idea
you take whatever people throw at you
and no matter how outlandish and you
throw it up on the board but if it's
real outlandish
you kind of use that as an opportunity
to find out why where did that occur to
you what are you thinking
maybe there's some information in that
connection right it didn't occur to you
to think that but they might have some
uh understanding that caused them to you
know
to reach that conclusion or maybe
they're thinking about the problem space
in a completely different way and that's
why it seems oddball to you but when you
talk to them about it
it'll seem less oddball
how might you use that in infosec so
your new internet of things
product must store credentials and
tokens so that it can communicate with
your service
however the selected hardware doesn't
contain a secure
secure element um it doesn't have any
way to securely store the material or
resist recovery
we'll just assume in this this fictional
device you can always get these creds
off of it
how might you address the risks of
credential theft
or cloning what i don't know but maybe
we would use
brainstorming you might come up with
some ideas maybe there's some
other immutable characteristic of the
device we can use to identify it
maybe we don't need to store a specific
token maybe we just don't have
credentials and tokens in the first
place what if we re-thought
how the product works and just got rid
of them somehow or
maybe it's a feature maybe we want
people to clone them maybe that's a
great idea
and it enables you know some sort of
other business model for us with this
same widget
um so red team analysis i think
it's it's another technique but i think
it's specifically
in this context we have to be careful uh
to understand
what red team analysis uh actually means
uh it's more in the context used
in the structured analytical techniques
of what i would consider a threat
simulation
or an adversary simulation meaning uh uh
you are determining you're analyzing
what the red team might be able to do
uh instead of red teaming your analysis
so uh how would you do it uh analyst
takes on the position of a potential
adversary
just you know rising on uh in the
thought space
and and tries to keep in mind the
motivations and the constraints
the adversary is driven by or restricted
by
and then they develop a list of
first-person questions the adversary may
be trying to answer for themselves
how would i interpret this particular
result or what would my next prior if i
reach a certain
point of exploitation what would my next
priority be because what is my ultimate
goal it might be specific to these
threat actors
uh you then lay out and prioritize all
the possible next steps
an advertary an adversary may take based
on the answers you came up with
those questions when you were sort of
play acting in that role
and of course the reason you do that and
lay out and prioritize all those
possible next steps is now you've got
another
set of great things you can look for and
monitor that will help you figure out if
these these sort of activities are
occurring in your environment
an information security example so
you've set up
several high interaction honeypot
honeypots in various places on your
network periphery
you notice a potential attacker always
connects sort of at the same time they
connect real quick
they get in they run a few meaningless
seeming commands where they just check
the user id
to make sure they're root uh and you
know maybe they enumerate some processes
and
dump some uh information about you know
what kernel it's running and then they
leave and they just do that over and
over and over
and never do anything more with it so
you might find yourself well what's
wrong
what's wrong with my honey pots right
well you might conjecture well
maybe the attacker is detecting it's a
honeypot and just dropping off
so maybe i need to change something up
or you might um
decide that the attacker is really
looking for a specific type of host
maybe they need a certain type of
processor
certain you know code revision to pull
off whatever it is they're looking to
exploit this host for
and that's why they're not interested in
it or
does the attacker have a lot of
compromised systems already
and they're just doing this because they
want to know if it's still active
they're just keeping it in the loop in
case they ever need it someday and maybe
it's all automated maybe no
human has ever done this some bot just
popped your honey pot and reported home
yeah i got something great
and it goes into some list and c2 and
they'll use it someday maybe
um knowing uh the viewpoint of your
attacker like why they may be doing that
in this scenario i threw in oh the ips
tips you off that it's some specific
actor
i don't know i think that normally
happens but let's assume
you know you could say oh well i
understand uh from this
attacker's point of view you know
they've got a ton of these things so
likely if we keep watching it eventually
they'll come back and want to actually
use this for something useful to them
uh so what have we learned uh we've
uh learned if you're concerned about
information quality
then use diagnostic techniques if you're
concerned about information biases
use contrarian techniques you're
concerned about information completeness
use imaginative thinking techniques and
where can you learn more the document
that i started with is fantastic
it goes way into more depth you should
really read it if you found any of this
interesting
there's another great document from the
rand corporation where they analyze the
first document and then decide whether
they think it's good or not
that's fantastic and then there is a
book structured analytic
techniques for intelligence analysis uh
by richard hewer and randolph
pearson uh it's 900 on amazon so i
didn't read that at all
i hear it's really good if someone can
get me a copy of that book i would love
to read it but i'm definitely not paying
900 bucks
um so if you have any questions i'm
rabbit i'm pretty easy to find on
twitter
uh you can email rabbit at rapid.com use
the right sixes
uh i will respond to the pgp's around
and uh thank you for listening to me
blabber for a half hour about
structuring