connection firewalls for threat hunting
i'm sean wallace a cyber security
research strategist
for five years i've invested time
designing and implementing a passive
mixture and firewall
this has led me to understand that it
can be used as a great thread hunting
tool
and in this talk i'm going to spend some
time showing how you can achieve
visibility with these nation firewalls
test new features to help mitigate risk
from your production traffic
and apply bleeding its functionality
signatures
to those next-gen firewalls to get some
early visibility
and enhance and complement your threat
hunting tools
first a bit about myself again i'm sean
wallace i'm a research strategist at a
top u.s bank
i'm a network security domain
specialists things like
firewalls ids thread hunting tools
packet capture tools
and email are things that i have a lot
of history and experience working on
i'm an advisor to a threat intel as a
service company by the name of silent
bush
i'm a defcon goon and a black hat
volunteer and i live in the san
francisco bay area
so how did this come about as i was
looking at next-gen firewalls
i noticed and observed an operational
risk that
application id could potentially cause
impact of production traffic
and i started to look at ways to
mitigate that risk
a traditional firewall has firewall
rules
in what's called the five tuple looking
at source ip
source port destination ip destination
port and protocol
with application firewalls we add
application id
as well as some other features such as
user identity which can be used
to write those firewall rules
so in this example that i'm showing
below we can apply that only http proxy
traffic could be allowed
through a rule meaning application
traffic matching ssh could not thus you
can block ssh traffic
you could of course also put in a rule
above that says
deny ssh traffic for the application
so now that i have this application
logic
what happens if that logic changes on
the firewall
applications are ever changing and what
if the firewall either can't keep up
or what if the firewall starts to think
that that application is something else
we had a similar incident that happened
very early on where we had some impact
with a
tool just something such as tacx where
the application logic of that changed
in the firewall and started to block the
traffic
so you know again we considered how can
we
mitigate that risk and i thought of
putting a passive next-gen firewall
side-by-side that we could update
quickly
and monitor such changes you might ask
why couldn't you just do this in a lab
environment
well with a large network with so many
different applications it's hard to test
every single one of them in the lab
so having this allows us that luxury of
being able to see what's on the
production network and observing those
changes
as well as nothing ever works in the lab
like it does in production
so the solution i came up with again is
let's deploy a passive nixon firewall
this gave us that mitigation to observe
application ids
as well is now we can test new
configurations new features
new signatures or deploy custom
signatures
or new versions the added benefit of all
this
is we can turn on all the security
features the passive security features
we want
and get the most value of our next gen
firewall to help complement our threat
hunting tools
this isn't necessarily a brand new idea
this has been done with ids vendors
before
a lot of them call these their various
lighthouse systems
where these systems are observing
traffic on the network
and reporting back so they roll out
these new signatures to various
lighthouse systems and make sure that
they aren't seeing various false
positives
before rolling them out now that a lot
of systems are moving to cloud
they pretty much do this without even
having to tell you
but you might ask don't i already have
other threat hunting tools
what's really the benefits from why do i
need this
well you know i'd like you to ask
how many of your threat hunting tools
have application id intelligence
i know we have quite a few and they just
haven't gotten there yet with app id
intelligence yet
unless they're from a big vendor that
already has
this baked into an action firewall where
they've been able to
port that over to various tools
merging threats how often do these
various tools push down
new signatures or new content and are
they at the pace that you would see in
ids
box or what not pushing content
most of these tools don't have ids
signatures themselves either
most packet capture systems i know of
don't
either although some have been
investigating going in that route
file movement i know a couple packet
capture systems as well as other thread
hunting tools which
have issues to coding various files on
the network if they are non-web apps
however what i've seen with um
connection firewalls is it's
very uh very key to be able to decode
that
as it's needed that for some of its uh
feature and functionality
strategic placement you may decide that
where i have my firewalls and where
um i have my packet capture or other
thread hunting tools maybe i have them
at different choke points or different
locations so you can get some added
benefit
out of this because of that and
decryption
surprisingly there's still quite a high
value
without encryption here so i think
that's an option to consider
now i'm going to look at a couple
different architectures and discuss them
that could
that you could deploy in in this one
this is without
decryption you can see the user traffic
is um accessing either a router or a
packet broker
and then that traffic spanned off to
your passive next-gen firewall for
inspection
however your production traffic your
active traffic's going through number
three here
and flowing out to the internet as
normal
here we have one with decryption where
we add that packet broker to
parse over and decrypt that traffic and
then send it back to the packet broker
in this case a passive next-gen firewall
is getting that decrypted traffic
but your active traffic is still going
through next-gen firewall inspection as
normal
one thing i'll add here is the next-gen
firewall can also do
encryption and wouldn't it be great if
instead of having this added box of a
packet broker that the next-gen firewall
could just span
that traffic over this is something we
definitely have been working with
various vendors on figuring out if we
can do to help
simplify this diagram and reduce the
number of hops traffic has to access
and the number of times we have to
decrypt traffic
so the high level concept here is with
all of these features
can we deploy a passive nicshan firewall
essentially as a pre-prod environment
where we can see live traffic but also
leverage
all these benefits of features
functionalities
and versions coming from a next-gen
firewall
again you can have early adoption of
hardware software
new threat signatures new features and
new functionality
the one other thing i really like about
this approach
is most threat tools or packet capture
systems that i've worked with
really have leveraged a custom ui where
you're always in that ui doing your
hunting and looking
but nixon firewalls are really good
about logging and we're interacting with
a sim
so this gives you the benefit of giving
all that data over the sim
leveraging reports and alerts and
anything that you have in there
as well as other logic or other sub
searches that you can
enrich your data with that you may not
get
in other threat hunting tools
so now we're going to go through and
kind of look through some of the
features and some of the things that
stand out
obviously the firewall logs are going to
allow you to see all the traffic and
this is important because if you don't
enable them
you'll maybe only see all the threat
traffic that can be good or bad
depending on your solution
if you want the app id enrichment though
you're going to need all of that traffic
logged
depending on what your other firewall
solution is this could either be new
traffic
or just enrichment
or a different source depending on again
what that
other traffic is if you do have other
firewalls this traffic can also be
duplicate
that's something to consider because
it's essentially is doubling your log
size or even more if you're starting to
enable a lot of features and
functionalities
but if you really want to compare your
application id traffic it's very
important to enable this
you need to be able to see if the
application traffic between your passive
environment and active environment are
the same
i should note here that various vendors
log differently
some of them log at the beginning of a
session some of them log at the end of
the session
the end of the session you're going to
find out how did the session terminate
how many bytes were in the session
sometimes you'll find out
whether it's in timeout or not you don't
necessarily get that in the beginning of
the session data
all the firewalls do allow you to enable
both
logging at the beginning and in so this
is a strong consideration that
maybe you want that data to be able to
really kind of help you
application id nixon firewalls
introduce application id now they're
giving you more contacts than just a
destination ip and a port
they're giving you what is this
application and what is it doing
so if you look at the example here i
have some sample common applications
or uncommon i should say you see spn go
you see facebook facebook social plug-in
you see google cloud storage download
you see the context of what that user is
doing they're downloading or uploading
that can be very valuable in thread
hunting if you want to start
writing your searches to filter on
people uploading data
or perhaps downloading data then you can
also start to pair that with your
various threat data to see exactly
where it came from and what they were
doing so
some ideas how i use the application
data is looking at those rare ones i
always like to see
what am i what would i expect on my
network what am i seeing that's
different
or not expecting and why why are people
suddenly going there
you can also leverage the context that i
mentioned to see what people are doing
within those applications maybe they're
uploading when they shouldn't be or
maybe they're uploading to a place you
didn't expect
and then of course you can look at
various anomalies this is very important
as you can start to see those sharp
increases and this can lead you to
various
uh dlp type information or perhaps c2
in this example i'm showing we're
looking at github traffic and again you
can see that context
um this data is from palo alto it's
publicly available and it's totally free
from applopedia.palo optonetworks.com
most firewall vendors have this
information for free
i just pick palo i kind of like the the
interface of how it shows and it made it
easy for me to talk about
so in this case you think about this if
you want to
write the same content on another thread
hunting tool
essentially you probably could do it
right we've done this before in
various packet capture systems where i
want to focus on
uploading or posting of information but
you know
if i if i'm looking for uploading i'm
looking for posts
but then i have to go and do additional
leg work how do i distinguish between a
post and an upload
you know here or edit of a github system
here um the next gen firewall vendor's
done all the legwork for you
so it really can simplify and you can
focus more on the content rather than
you know getting the content
and another example here i'm looking at
ssh
you can see in ssh you can start to see
how you can leverage looking at the
uncommon ports you can see
the ports i would expect to see on my
network 22 80 22
and some of the 22 22. now i start to
see some more of the uncommon ones are
ones that i'm
not really expecting the one that stands
out to me obviously is 8080 because
that's my web proxy
that means somebody is essentially
probably trying to tunnel over ssh
with any of these i can now pivot off
the ip addresses that are provided
perhaps see what other things they're
doing or whether applications they're
triggering or maybe threats
and that's very valuable
moving on to ids threat and malware
i think they greatly speak for
themselves obviously ideas can be used
to identify potential threats
malware inspection can inspect files
traversing from malicious malware
you can set up immediate updates as i
mentioned before
and as soon as zero day threats come out
start to monitor those
hunting for those new vulnerabilities
you can find out
internal threats perhaps new networks
that are um
have scanning that you didn't expect to
or who are the people out on the
internet who are really hitting you
with zero-day scanning
one thing i would mention be very
cautious of
enabling low and informational
signatures
that data can be overwhelming noise the
reason for that is
um a lot of those are just meant to
identify various types of traffic not
necessarily meant to
look for threats for threat signatures
one example of that
is we had enabled all loan information
when we first started it out
and one of the signatures was
identifying various
active directory traffic it was
essentially saying
hey there's an active directory server
here there's an active directory server
here
every single packet and i'm like yeah
there's supposed to be an active
directory server here
so it's just one of those that i'm like
okay
i'm not sure that i needed to know this
information but
it was an awful lot of noise and also
had a huge cpu impact due to that
moving on the url filtering categories
nixon firewalls also have the capability
to log all urls
and then take those urls and
match them to a category that can be
very
beneficial if you're using a web proxy
you probably already have this
functionality
but what if they don't match
in this example you can see one of these
is not like the other
here we have three public categories
heights
cisco checkpoint broadcom and palo alto
and three of them are showing this
category this particular site
is shopping whereas we have the fourth
one palo alto showing it as malware
i'm sure if you talk to each of the
vendors they're going to say that their
threat intelligence is probably better
or comes from a different source than
others
however and they probably have more
insight than others and that's
perhaps why one is different it's for
you to decide
but taking as many different sources as
possible
could help you with better enhancement
of your threat data on your network
moving on to file protection this is my
favorite part
file protection is allowing the ceo
files traverse across your network
not just for web applications which i
think is very exciting
you can start to see other traffic such
as smb and other
applications that leverage this in this
example here i can show you
i'm showing you how just looking at the
file name
searching for password star essentially
you're seeing various file names that
have passwords in them
these obviously look very suspicious and
um the first case here you can see that
sober smb
v1 v1 is not encrypted
so i can go in my packet capture tool
pull this file out
and guess what i have some 2021
passwords
and probably so do other people if they
essentially are trying to pull these
files off our network
this is just an example um that i'm
showing here but
i wanted to kind of give that of seeing
how this could be valuable
further going in this example again i'm
showing palo alto's applopedia here
where it distinguishes between v1 v2 and
v3
this can help you search on your network
and find out
do i have v1 on my network where i
shouldn't and you can track down
those various systems and have them
cleaned up
then you can extract those juicy files
from the system
uh one thing to note is um
and i learned this i didn't necessarily
know a lot about smb when i went down
this path
so this was a good learning experience
for me
the file pointer in smb is not encrypted
so even in v2 and v3
i am seeing that file pointer so i can
see file traversal across the network
smbv3 and such also introduced
multi-channel i put a link in here that
is a
really good breakdown of what that means
however the long and the short of it is
it uses now multiple
sessions to exchange files so we lose
some of this visibility
various vendors are still working
through support for multi-channel
so when using this uh in the future we
will be able to see
the same visibility we have again and
that i think is very exciting
i do know there are some vendors that
already have this rolled out
so i i would encourage people to look
into this if this is something that
you know little about and may be missing
visibility on your network
moving on to dlp there are dlp policies
that you can enable
uh in my experience though i leave that
to the experts i attempted to turn this
on and it was nothing but noise
it was going to be very difficult for me
to write a lot of these signatures
and a lot of the advanced eop tools
already have
better detection than the signatures
that i had the capability of writing
i would encourage you to partner with
your experts and see if maybe there is
some benefit to do this and maybe it's
worthwhile to reach out to them
and chat about it
moving on to email email is also one of
the weakest areas
the reason for this is the next-gen
firewalls don't run as an email server
they run simply as a active we're gonna
pass the traffic through
in an email server you're terminating
the session gathering all that metadata
and then starting up a new session it's
very possible for the
next-gen firewalls to capture this
information
i just don't think they've invested the
time or effort um nor maybe
do they want to to kind of do this
for that reason there's been a lot of
parsing issues with the data that i've
seen in my experience
i do have a lot of experience on the
email front and i've noticed
the log messages are just so much better
coming out of the email gateway that i
tend to result in looking at that
one area i can note about this though is
that you can
leverage email for looking at malware
and threats that are coming across email
for what data you do have so that is
very valuable as another source maybe
it's a different type of sandbox maybe
it's a different type of technology
more sources the better in this case
two other areas are dns and user
identity
dns protections in most cases are active
protections
on next-gen firewalls so there isn't a
lot you can do here nor a lot of value
of enabling that
it requires you to actually point to the
next-gen firewall
then actively kind of take that data and
do something with it
and if you have a passive environment
well you can't necessarily do that it
requires that kind of
active interception uh in that case i
haven't spent
a lot of time looking on that list i
wish there was more
kind of anomaly detection for a passive
environment built in
user identity on the other hand is is
great you can tie this
after directory and start to get a
relationship of what your users are
what your user anomaly behavior is and
etc
i haven't spent a lot of time on this
either because
it requires a somewhere to grab that
database from
i've noticed that various different
implementations
do this differently there's you know
there
hasn't been necessarily a common trend
to do this
and that's been one kind of turn off for
me on doing this
the other is this is an act of
connection and it's talking actively to
after trihetry
um that can be in a large environment
that could be very impactful and have
some production risk
so we kind of stayed away from that as
well until
we're a little more ready and active to
look into this
overall um i want to you know say in
conclusion
i really think next gen firewalls have a
legitimate place here for next
for thread hunting i really want various
vendors to kind of take this seriously
kind of
consider that hey yes we may have a
use case here that other people and
businesses may find as well as a risk
mitigation
part form networks to make sure if
they're
a little scared or a little shy to get
their feet wet because
of these various impactful changes
they're all alternatives to do this i
also really want to thank the community
for letting me speak i'd really love to
hear from more people
who have similar experiences or further
interest in the subject
i definitely you know as i said i'm very
passionate about this and would love to
move forward myself with this
there are any questions please let me
know feel free to reach out
thanks so much