nearly a decade of
working as a network engineer there have
been a number of different things that
have
really gotten to me when it comes to the
intersection of networking and security
typically when i push back on these
things it tends to get people just
get tens it tends to make people squirm
and sometimes even grab their pitchforks
and torches
these are three of those things things
which really need to be thought more
about
things that with which that don't always
improve your security posture within
your network
and these are uncomfortable networking
topics
hi i'm charles rumford and i'm here to
tell you about some uncomfortable
networking topics but first
a little bit about myself for the last
10 years
i have worked as a network engineer in
both higher education and in the
corporate world
and before that i was assistance
administrator probably my most formative
years
in aiming in these topics is was working
as
a wireless network engineer in higher
education where the
um where there's a strong convergence of
usability
security and networking and flexibility
along with manageability to make sure
that the
system is working i've also had i've
also held
i've also done work as a network
architect automation engineer
linux system men i've also been an
adjunct professor and then outside of
the workspace i've i do a lot of
photography knitting
and if you really want to go look it up
i am a campanologist
we'll be curious to see if anyone asks
me any questions about that
before we get deep into the topics i
really want to put some constraints
around
the area in which i'm talking about i'm
in this
talk i'm not talking about client
networks i am talking about
networks that are providing services
to clients typically this would be a
data center of some sort
spaces where you might have web servers
you might have
database servers dns things of that
nature
that is the primary focus of this talk
in the primary areas in which i am
talking about
a lot of these a couple of these don't
really apply to
client networks but can be used in those
spaces
so i think everyone's familiar with the
project management triangle
choose to you can either have a project
be qual good quality and low cost but
it's going to take a lot of time
or it's going to take a little bit of
time and be good quality but it's going
to cost you a lot of money
or it's going to be low cost and low
time
but it's not going to be great quality
and this is something that we think
about a lot when it comes to
projects right but we don't think about
it we don't
what do we think about when we think
about design so i've kind of
changed the triangle and typically when
we deploy something in the networking
perspective
we choose to either it's manageable and
secure
but it's not really usable or it's
usable
and secure but it becomes a management
nightmare
or it's usable in management manageable
but it's not always the most secure
option
right and we think i've run into this
numerous times while working in the
networking space especially around
security
when deploying security um architectures
right but what if i were to say there
are a couple of things that we can think
about that we can do
inside of our networks that increase our
manageability
right and then also increases our
usability
while either holding security at a
current
current stance or possibly even
improving it
right you're probably saying that crazy
but just hear me out okay
right so for a number of you so let's
dive in
right and for the number of years um
there's one thing that has always gotten
to me
and that is when i can't ping something
right icmp is your friend
now before you go get your pitchforks
and torches and come after me
hear me out icmp is an
important functional part of the
internet it is the how the internet
passes messages from one from one space
to another right
and they're not just messages of hey are
you there yes yes i am
but they're also messages of hey that's
prohibited hey
you can't do that hey that thing don't
exist right
and we even we've even used icmp
for routes in the past right all all of
these are
ic our icmp messages and they're all
good things
right but over the years we've come to
we've come to um block icmp because
everyone thinks that
icmp is bad right that icmp is going to
make their system
more vulnerable people are going to be
able to find it
that really isn't true right
so i have some for some questions for
you right
what is the first thing you do when you
when you think that you
have are having some network
connectivity issues i'll give you a
moment
now if you thought to yourself you ping
something
that's probably what everyone comes up
with right i do it all the time
the first thing i do is ping the gateway
ping google ping
cloudflare ping something right can i
get a ping through
right we also use the uh pings
or icmp messages for reachability
latency and jitter right which are
all things that are easy to figure out
using a simple ping
right and in my mind as a network
engineer if something is not pingable
then it's down and it's not connected to
the network as far as i'm concerned as a
network engineer
is if i can't ping it then it's low
level networking isn't working
right so let me ask you another question
right oh there we go let me ask you
another question
what if google cloudflare and quad9
one day decided to block all icmp
inbound traffic to their services
right or better yet as a network
operator i decide to block all icmp
to your network gateway i basically i
say
no more pinging the gateway that is not
allowed in this network
right take a moment to ponder this that
default test that we all use
for determining whether there's
networking issues or not
goes away we no longer have that
very standard test that we use
right it's become a test that we take
for granted and yet at the same time
we [ __ ] it on a regular basis so why
do we
stop icmp traffic to servers to services
right typically it's because if it ain't
pingable that it ain't fast
it can't be found well i'm here to tell
you that that is
false that is quite false right there
are
many many ways to find a service on the
internet
without using icmp right security
through obscurity has never been the
answer
it never has been and it never will be
and all you're doing is adding
complexity to your situation
right and so as a friend i'm going to
tell you something
right an open port to the internet is an
open port to the internet
right if you have a resource on the
network
and you have an open ssh port or an open
http port
or anything someone knows about it i can
tell you that
right now at one of the universities
that i worked at
one of the researchers on a regular
basis scanned the entire internet
with in under 40 or under 24 hours
on a regular basis every port on the
open internet
and able to map out what's going on and
she wasn't using
icmp right
if a port is open on the internet then
it is open
so i say allow all the icmp and let's
just make our lives so much easier
it's easier to call it's easier to
figure out
what's up what's down it makes
troubleshooting the first lines of
troubleshooting
so much easier right and plus how many
times have you run an nmap command
with the dash pn flag to just bypass
icmp
we know the port's open why are we
hiding something that's already
available right
so some things to consider when turning
on icmp right
the first thing from a network
engineering perspective is can you at
least turn it on for the local network
right if you lease allow it in the local
network then
troubleshooting inside the network
becomes easier
yes i would prefer to icmp be open
everywhere
but if you just start in the local
network awesome
right but some other things to consider
you can rate limit icmp
especially at your firewall or at the
host level remember
host level firewalling is super awesome
and super handy
here's an example command from a linux
iptables
to rate limit icmp down to five uh five
frames a second all right
we can also limit the types of icmp
messages there are a bunch of different
types of icmp messages
we can limit which ones are allowed so
one of the things that we do at the
networking gear level
is we limit which icmp messages we
accept
and handle right this is something you
can also do at the firewall
and at the host level right but the
other thing to consider
is that i that icmp messages tend to
help improve the usability of the
network
right and the user experience of the
network did you know that a lot of
browsers when you try to navigate to a
website
which isn't allowed that when the icmp
reject message comes that it will
present the user with a message that
says hey
you're not allowed to do that right
that's independent of the web
the web application itself right
also like i said there are a number of
times where i've
pinged something thought it was down and
went running
one time i thought the whole set of dns
servers were down went running down the
hallways
to see yelling the dns servers down the
dns server's down
to only have the windows administrator
pop his head out and be like i'm logged
into them they're just fine
turned out to be another issue but i
couldn't ping them so as far as i'm
concerned they were down
right so just a reminder that i the
security through obscurity
is yuck it is not worth it um
an open port to the internet is an open
port to the internet
so changing port numbers not that great
right once somebody finds it it will be
found and it will be pounded
right um the other thing i've also heard
people doing is
not putting things in dns this is
not helpful right when you're unable to
find something in dns unable to type
something on a command line it becomes
harder to
troubleshoot and understand what's going
on right so i say put everything in i
allow all the icmp put everything in dns
and keep ssh on port
22 forever and ever
the next topic is one uh that i
i have a lot of feelings about it's one
where a lot of people deploy it in the
hopes of
um building a little security blanket
around them but in reality they're just
decreasing the usability of the network
and dramatically
making it harder to manage their network
and what i'm speaking of is nat
that is used all over the place and i'm
here to tell you that nat
is not a security barrier really it's
not
there's a lot of not happening
on firewalls and things of that nature
but i'm talking about natting itself
for those who don't know nat stands for
network address translation
and there are a couple of different
flavors of it um but it generally allows
us to translate one ip address
and port set into another the most
common use
is in your house you're probably using
that as you
as we speak um to watch this video
because it allows
multiple like multiple devices inside of
your home network to talk to the
internet via your
your isp right but let's focus
on the the enterprise space right and
that's typically used
to to handle two problems one being
not having enough ip addresses um and
two
security the first one not having enough
ip addresses
um is solved by technologies like ipv6
if we could get ipv6 deployed in more
places and get rid of nat that'd be
great
but that's a whole different talk i want
to focus on the security aspects for now
one of the big things that i find
in the conversations i have is a lot of
people
love private ip addressing they think
that
it will prevent people from contacting
or being able to get to a device or a
resource
but in reality private ip addressing
isn't a magic security blanket
right a lot of people put stuff behind
gnats and magically think that
everything
is hunky-dory right just because it has
a private ip address it's all of a
sudden magically protected
and typically i start to see people
getting lacks about their security
posture
i've seen a couple of times where things
have been compromised
after being moved into a nat and having
private ip addresses
than when they had public ip addresses
right
but like i said i've seen a number of
administrators start to get lacks about
their security posture because
everything has a private ip address
right so let's take a look
at um some some network diagrams real
quick right
so here i have a fairly standardish
corporate network
right we have the internet we have some
clients um and then we this this
nat box um i'm gonna say is the edge of
our data like a data center type thing
right we have some web services we have
some network services and we have some
database services
right but the first thing that nat does
is it creates this inside outside
barrier right now you're probably
thinking to yourself
oh well firewalls do the same thing yes
but firewalls do not separate out ip
addressing
we have one ip addressing scheme inside
the firewall or inside the nat box and
one ip addressing scheme outside the the
the nat box and this leads to all sorts
of problems because
everything now has two ip addresses
right you've got the internal
private ip address and you have the
external public ip address
so that www.tallwireless.com if you're
outside the inter if you're outside the
core outside of the nat
then it might have a one ip address but
when you're inside you have to hit it on
a different ip address where it might
not work
right along with that we have to open up
um ports via destination nat to allow
outside cert
outside clients and devices to talk to
things inside
right that open again an open ip an open
port on the internet is an open port on
the internet
right we now have a we have now have a
lot of nat to track we have to track all
those translations
we now have to have ways of querying it
we have to have ways of
monitoring it to make sure we don't run
out of resources
and if there's a security incident it's
just one more thing that has to be
investigated
to understand what's going on right
now you can get along with doing
what's called hairpin adding or hairpin
routing
which allows a service to
access an internal service to access
an external services ip address
and then have it matted back inside so
for instance
the database server could talk to the
web server on its public ip address
but this is just more configuration that
has to be tracked managed
taken care of and well we
i don't like that simplification
all about simplification right now a lot
of people do their natting
on a firewall right so if you have a
firewall and you're doing that
you now have to open up the ports to
allow
the fire you have to set the firewall
rules to allow that traffic in
right so we've created this complex
space that we now have to manage and
track
and reason about which isn't great fun
all while having two separate names
ip addresses for everything now
i don't like this and i really don't
like the idea
of having um a device
have to contact something based on where
it is in the network i should be able to
just
access the web service and be done with
it
so let's let's let's review real quick
not bad
right there's more to keep track of this
could be all of the destination ads you
need to keep track of the firewall rules
translation records
and there could be even more there's now
more to clean up during commit
decommissioning there's more to clean
there's more to set up during
commissioning right and there's
something could totally be left behind
it also increases our our auditing space
right there's so much more that needs to
be verified
checked to ensure that everything is uh
way it should be
and my personal thing that i hate about
nat is that there's a
two of everything right a classic
example that i've used in the past
is at a previous organization i worked
at you have
a dns server right you have one set of
ip addresses to access the dns server
when you're
inside the network and you have another
set
when you're outside but they all go to
the same server but it means they now
have to track
in the configuration for all of my
switches routers
servers where i am in the network to
decide which
uh to decide which ip address um that
we're going to
right and like i said network position
means something am i
inside the inside the net am i outside
the nat
am i having to go through multiple gnats
because that that that happens a lot
right and the other big thing
is every time i've had to solve some
major problem regarding nat
i've had to deploy more than that more
things to track
more things to keep aware of right so
overall
we have a huge decrease in manageability
and a huge decrease in usability
without actually increasing anything in
our security space right
and then like the icmp we're just
complicating everything without
actually having any security gains
because it's all just security theater
and we don't like security at least i
don't like security theater
so let's take a look at another way of
looking at this right
so in this diagram
in this diagram we have the similar
network but let's make the assumption
that
everything here is public ip addressing
right
so the first thing is is all the web
services all the network services all
the database services
all have public ip addresses right and
i'll touch about
the amount of ip addressing in just a
second but roll with me here for a
second
right now like the nat you still have to
open up
ports for the firewall rules easy peasy
that simple right on top of that to get
that
actual feeling of not being able to get
in
we can set up a default deny on the
inbound everything coming in will be
dropped unless you explicitly allow it
right this gives you the same comfort
feeling
right with actual security posture
of having a private ip address having a
public ip address with a firewall in
front of you that says default deny
is better than having a nat in front of
you
with no firewall and no default deny
right but in this whole thing we've
simplified the architecture
we're doing native routing between our
data center services and our corporate
network so now everything is this
everything there's no insight outside
and so we can just access everything
directly by the same ip address the same
name and it's much easier to manage
across the board
right but it also simplifies our
auditing space it's easier to reason
about inside
of our heads when we're trying to figure
out problems and do troubleshooting
and when it comes to maintenance it's so
much easier to figure out what needs to
be done
and be able to build an infrastructure
that is more
tolerant and resilient to handling
maintenances
right plus we get an increase in
usability and an increase in
manageability
right now you might be thinking in your
head
wow that could be a lot of ip addresses
a lot of public ipv4 addresses and well
i will agree that it could be a lot
right
i also know that there are a number of
organizations that
just can't afford that and don't have
the staffing
to manage it or the network connectivity
to manage it right
i will admit that i've come from a
number of organizations
that have a large number of public ipv4
addresses
available to them and this has not been
a problem
although they keep using they've kept
using natting to solve the problems
instead of the giant pool of public ip
addresses
right but i'm i'm very aware that public
i.p addressing
is a rarity in today's world
right i think we've almost run out in
terms of non-allocated ip space
but because of this there are ways to
approach this in a hybrid manner that
allow
for reachability and security
and getting the best of both worlds
while
still keeping the manageability and the
complexity down
now i will say you can go native full
ipv6
there are so many addresses in there but
again
another talk so let's look at that
diagram
one more time so in this in this model
again
right let's just say for instance our
database services we give them all
private i p addressing
right so now the web services can talk
to the database services
great clients in the corporate network
don't need to talk to the database
services they only need to talk to web
and network services so we'll give those
public ip addresses
and we're going to natively route all of
that stuff
internal to the data center out into the
the corporate network
right so if there is a reason to contact
the database services they can plus we
can put an
a source nat on the firewall that will
allow the database services to access
the internet
to do things like updates and things of
that nature
right this is a model that i've seen
deployed and i actually deploy myself
because
i am also not a rich person and can't
afford the amount of public ip addresses
that i need
for my own personal infrastructure right
so this is a great model right so really
public ipa addressing it's awesome right
um you should i'm not saying go
run and rip out all of your
private ip addressing and go buy you
know
hundreds of public ip addresses at
really expensive prices
but i'm i'm i'm challenging you to think
about
how you think about private ip
addressing and public ip addressing in
relation to
the usability of your network and the
manageability of your network and what
does that
add to your network overhead in running
it
one of the final areas that i want to
talk about is
network layer encryption and how it
should not be relied upon
a lot of systems people and application
developers
rely on vpn and site-to-site
vpns for their encryption encryption
properties
but it ends up binding the network
architecture into a very specific path
now i'm not talking about client vpns at
this point at this point i'm talking
more about site-to-site vpns
um that are used as shortcut paths
um to uh you know connect say a data
center
two data centers together or to connect
uh the data center to the cloud
right but one of the things that happens
with these is that
people start becoming very dependent on
those shortcut paths and especially the
network
the encryption aspects of that vpn
connection
and it should be noted that vpns don't
necessarily have to be encrypted
there are a number of different vpn
technologies out there that don't
encrypt traffic at all
right one of the more common situations
that i
i've seen is when people start getting
into the cloud they want to build vpn
tunnels between their data center
and the cloud and these are ipsec
encrypted vpn tunnels and
there's nothing wrong with that but one
of the problems that happens is that
people start to treat that
as part of their data center and then
they start to rely on the fact that that
pathway is in fact encrypted
and what this does is it binds the
architecture to that pathway
so that uh once you need more
throughput or you have a lower latency
physical connection
you're now so bound to that encrypted
nature that you can't move
off of it and people get scared about
passing that traffic over other links
through other networks
without it being encrypted but at the
same time they don't want to
actually encrypt that traffic at the
application layer
so i'm here to say stop relying on
network layer encryption um and start
moving that up into the application
itself
there are a number of different
technologies um tls
even in the point in point to endpoint
ipsec
um are great technologies to use to
bring that encryption
up into the application or up into
higher levels of the network stack so
that we
as network engineers are able to just
move packets
and worry just about moving packets and
not have to worry about
traffic taking a very specific path
because that path happens to be
encrypted
that should all be handled inside of the
application lower traffic
right
so we've come to the end um there are a
couple of takeaways that i hope that
everyone gets away from gets from this
talk
um one icmp is good two security through
obscurity is bad
three not bad public ip good
and four bring that encryption up out of
the network and into the application
layer
there's another thing that i hope that
everyone walks away from
this talk with which is i hope that
everyone
starts to take a little more time to
think about
not just the security aspects but also
the manageability and usability of the
systems that they develop
it's really easy to get caught up in we
need to make this secure we need to make
this secure
and not think about the
use how are users going to use this what
are the workflows look like
how are we going to do maintenance on
this how are we going to
troubleshoot this how are we going to do
all of those things
right complexity can be managed but it
can also be shifted
and it can sometimes even be eliminated
if you just sit down and think about it
a little right um remember to take a
look at the whole system
um take a look at what uh your actions
today will have on future plans
and longer term strategies within the
organization
i'm aware you should also have some
longer term strategies
and bring in some networking subject
experts right to help with the network
design and the network
aspects of the the thing you're
designing right
um we might
have suggestions on how to achieve what
you're trying to accomplish
while not crippling the network or not
binding
the system into a specific design or
um setup that locks us into something
forever and ever
right and those network there's a lot of
network people who really love building
secure networks and systems
who also think about usability and
maintainability so
you know reach out to them right i want
to thank you all for your time and thank
you for listening
i'm on the blue team discord server
answering questions
if you want to reach out i am always
happy to talk about the intersection of
networking security management
and usability so please reach out i'm
not that intimidating
even though i am very tall and you can
find me on twitter under
uh tall wireless but i'm going to warn
you that i don't just post about
networking and security
um you're going to get a whole lot of
random so i hope everyone has a great
defcon and i will talk to you guys later