and thanks for having me i'm really
excited to present uh this year in the
cloud village i've presented in many
club villages and and defcon in general
but uh the cloud village is a new one
for me so super excited about this um as
jeff mentioned uh the title of this
presentation is identifying toxic
combinations of permissions in your
cloud infrastructure so
this is a mashup of a lot of research
we've done
over a few years now in doing risk
assessments for organizations and
largely focused around aws azure and gcp
so we'll give you some examples some
more stories things like that to really
help you think more about a lot of these
complex permissions in the cloud and
with that i'll go ahead and jump right
into it
so really won't talk too much about
myself but rather instead i'll just tell
you a funny story how i got into cyber
security you know uh hanging out at def
con for many years that there's always
really fun and interesting stories about
how each and every person has gotten
into cyber security
for me i was actually working for one of
the stock markets and at that time
i was a unix system administrator and an
oracle dba
this is around circa 1993
and i was always one of those uh one of
those folks that's always kind of
tinkering around with stuff and
there was a tool from dan farmer you may
remember dan
in the old school days of defcon 1 and
beyond
and dan authored a lot of different
tools out there for doing
security assessments things like that he
had a tool called satan which was an
acronym for kind of a security
assessment tool set um for unix
platforms at that time so i downloaded
it ran it full bore on the network and
knocked over a server in the middle of
the trading day on the trading floor so
um obviously was not a good day for me
i was called into the csos office at
that time although they didn't call him
csos back then they called him vp of
information security
and he effectively said listen i'm here
to offer you one of two options you're
either fired or you can take a promotion
so i really couldn't go home until my
wife had been fired so i took the
promotion and asked questions later but
he effectively said listen you know i've
got some auditors on my team but nobody
really thinking about the network from a
hacker perspective and you know you
already manage the firewalls dns and
vine and a bunch of other stuff it'll be
your job to now protect the network and
build out a team doing that and that was
how i got into cyber security so i hope
you enjoyed that story
all right so in getting into the
presentation then um
in terms of you know presenting this
talk and a lot of this different types
of content you know over the last few
years you know really kind of begins
with really begging the question of you
know like what am i really looking for
here how do i identify toxic
combinations of permissions how do i
look at that from a proactive
alive and a reactive perspective too
furthermore you know i'm a big believer
like dan farmer in you know the best way
to really understand the risks and
vulnerabilities in your environment is
to do you know various forms of pen
testing and the security assessments
obviously in a planned and you know both
a black box you know type of scenario
and all of that but in understanding how
a lot of these
breaches occurred is really breaking
down the the ttps of those tactics
techniques and procedures to really
understand how they occurred to better
understand how to protect our own cloud
infrastructures
in addition um
we'll kind of map that to
what we've modeled out in terms of a
cyber kill chain but now for the cloud
and then we'll start to break this down
not only in terms of aws and azure but
also gcp and then we'll wrap up with
some additional resources
so at the end of the day you know what
am i really looking for there's a lot of
different aspects you can look for in
cloud infrastructure
specifically around identity in account
activity not only in terms of human
accounts but remember service accounts
outnumber human accounts today five to
one and that's going to increase over
the next few years they say by 2023
service accounts will outnumber human
accounts 20 to 1. so it speaks a lot to
the automation and at the pace at which
we're building in the cloud that there
is a plethora of automation going on and
that's going to obviously continue
there's also a lot of new services in
the cloud that you may or may not be you
know familiar with
i think a lot of people who have been
you know working in the cloud already
are familiar with the nomenclature
surrounding ec2 instances or vms within
aws
s3 buckets or storage within aws azure
blobs and container or storage within
azure but there's so many different
services coming out each and every day
keeping up at the pace of this is really
really difficult
also when i think back at my days
working at the stock market you know
circa 1993 we would have different teams
that would build out the network the
servers loading the operating system
loading the databases these things would
take a week two weeks to deploy end to
end and we'd have different teams deploy
that
in today's world many organizations give
developers access to the whole stack so
in 15 minutes or less they can deploy
all of that and begin their work but
also that power becomes you know
less and less focused on separation of
duties and how do you start to
distinguish what should i developers and
what should i not and how do i further
control that especially in a production
standpoint
also how you access the cloud
infrastructure has changed we still have
things like secure shell and rdp but now
in aws for example you have aws access
keys which give you access from the
command line interface so developers
administrators tend to use that for
managing and doing things within the aws
environment
and then as you start to think about the
risks and the cyber kill chain the
lateral movement privilege escalation
exfiltration all of these things
starting to understand of all of these
different permissions and policies and
more
how do i really start to map those out
from a cloud perspective versus how we
traditionally look at it from a network
perspective
so what we've learned in doing over 100
risk assessments in the cloud for
various organizations is that there are
tens of thousands of permissions in the
cloud aws by default if you don't modify
it for a user you're giving that user
over 10 000 permissions
same thing for azure as well over ten
thousand permissions and also for gcp
thousands and thousands of permissions
and again unless these are restricted
the uh you know the probability that a
user is going to have admin type
permissions where they could use that to
destroy the infrastructure
you're accidentally you know giving them
a lot of those permissions and roughly
50 of those permissions allow that
capability whether you realize it or not
and and furthermore you know since we
look at each and every identity in the
cloud we find out that that on average
95 of the identities in the cloud have
these permissions yet the majority of
them don't need it nor should they have
it
and what's really alarming is that
you know what we've talked about up into
this moment is really permissions that
have been assigned but what are
permissions that our people are actually
using what does their activity really
expose well their activity exposes that
they're using less than one percent of
the permissions assigned so clearly
there's a huge leased privileges problem
and there's a huge issue with over
permissioned access in the cloud
the cloud security alliance did an
awesome
research paper around 11 prominent
breaches most of these organizations are
household names not for only us in the
cyber security space but just even to
your mom dad and other people that you
know
and one of the systemic threads across
all of this was really over permissioned
access
and really you know
really exposes the fact that least
privileges are more important than ever
because across all these different
breaches was a lot of permissions that
the user or service account had never
used yet were used by the attacker for
performing the breach
there are some really powerful commands
within the cloud
a single line or script can definitely
destroy a portion of if not all of your
cloud infrastructure
so as we start to map this out to a
cyber kill chain but now in the context
of cloud infrastructure
we can take all of those things that
we've mapped out traditionally in terms
of reconnaissance infiltration privilege
escalation lateral movement and
exfiltration then now look at it through
the lens of the cloud
a lot of organizations have shored up
their security in terms of the perimeter
so there's a lot of different you know
virtualized perimeter type products out
there that help you identify what may be
exposed to the outside world
but in terms of the permissions and you
start to take a look at these toxic
combinations
if they've got tens of thousands of
permissions
and someone's able to initially
penetrate my cloud infrastructure even
as a normal user are they going to have
a lot of excessive permissions that they
can exploit that that normal user never
actually uses and the short answer is
yes and that leads to many other things
within the cloud you also have this
concept of role chaining or
cross-account access you may have
multiple subscriptions or accounts or
projects in the cloud and cross-account
access for example in aws allows you to
set up roles and policies that such as
the sts assume role to allow you to
actually jump from account to account or
subscription to subscription and move
around in this cloud infrastructure and
as you might expect if misconfigured can
allow also an attacker to move around
and not only perform privilege
escalation but lateral movement
and this really starts to kind of bring
out the fact that role-based access
control how we traditionally approach
this is really kind of becoming old
school because it's really kind of based
on assumptions and you really need to
look more and more at the activity of
what's going on in the cloud to really
bring that second ingredient to the
table to better understand gee i've got
a lot of over permissioned folks in the
cloud i need to really prune this back
and remember the ability for read only
in the cloud read only to storage that
alone can allow exfiltration right so
even read-only access can be quite
dangerous
so if we start to break down some recent
breaches and then we'll start to kind of
dive deeper and deeper into this
you know there was one really prominent
breach with a financial services
institution that once their web
application firewall was breached via
server-side request forgery once access
to that i am role was obtained allowed
the attacker to start to exploit the
over-permissioned access all those toxic
combinations of permissions and we'll
break down those permissions for you a
little bit later in the presentation
that at a high level
allowed the attacker to uncover and map
out over 700 other s3 buckets that were
in the environment that had never been
used before by that particular role
and furthermore
with that read-only access determined
that one of those s3 buckets contained
more than 100 million customer records
and with that read-only access allowed
that attacker to download those records
if we start to kind of dive deeper into
this what is the cause for this well a
lot of it stems initially from default
policies so you may have your json
policy here with things like resource
star and these different permissions
that are actually allowing broad or
administrator access in the environment
but by looking at activity we can better
distinguish for example what permissions
have been assigned versus what's being
used this starts to help us uncover a
lot of these unnecessary permissions
being assigned to everybody in the cloud
if we dive deeper into this and we take
a look at this now from aws access keys
for example
in this particular example this was a
prominent security vendor i won't
mention the vendor by name but
one of the developers had
actually developed code embedded the aws
access keys
in that code and had posted it to github
and an attacker had found that these
were sitting out there and furthermore
uncovered that
a developer had actually made a backup
of the rds database or snapshot and used
those api or aws access keys to then
access that backup snapshot which
actually uncovered user accounts
password hashes and um being a longtime
employee of verisign way back i would
argue even the bigger issue being
exposing the certificates and their
private keys for every one of those web
application firewalls around the world
so
at the end of the day more than 13 000
uh customer certs private keys and the
password hashes were exposed
and aws you know and amazon in general
you know recommends as part of their
well architected framework that these
keys be rotated every 90 days but when
we do our risk assessments for
organizations we find many times these
keys are not being rotated one of the
key reasons for that is they end up
creating keys for everybody in the
environment but some of the people are
not even using them and so those keys
sit out there forever and unused
so this isn't a silver bullet i mean if
you you know rotate access keys it
doesn't prevent you know developers from
from posting them out you know on the
web accidentally embedded within code
but you know although it's not a silver
bullet um it is you know kind of one of
the pieces of the security hygiene so
ideally if you're rotating these keys
every 90 days maybe by the time a
developer posts it and someone else
discovers it's sitting out there you've
already rotated the key and maybe that
old key is unusable so it's just one of
many things and we'll talk more about
other things you can do for mitigating
these risks in the environment
all right i'm a big believer left-hand
farmer in really getting to the heart of
the matter and really providing some
real world examples we said that there's
10 000 permissions plus you know within
aws but what are a few examples of that
so one of those um
is how you define your virtual firewall
policies something called a security
group within aws
and if you give a user um all 10 000
permissions which we frequently find um
they have access to this uh permission
which is authorized group ingress
security group ingress for ec2
meaning they can change the firewall
policy for your vm
so think about this for a minute in an
old school perspective would you ever
give an average user or a developer the
ability to change firewall rules
obviously not right but by giving this
default permission to every user that's
effectively what you're doing
also if you start to think about this
from an insider threat
another dangerous permission is iem
create user this allows a user to create
another user id
it may be for another developer or it
may be a dummy account used for insider
threat
so when giving out this permission
you're inadvertently giving all users
the ability to create other user
accounts so from an administrative
standpoint would you ever consider that
in a normal environment
obviously not right there's a reason
there's administrators right but
effectively you're giving that
permission you know in that ability by
giving them that permission
we also talked about role changing
across account access and this is
related to the update assume rule policy
and the sts assume rule
by giving out this permission you can
allow developers and other people to
change how people can move around from
account to account or subscription to
subscription also quite dangerous
especially if they're going from a
development account to a production
account
so this can also be quite dangerous and
then furthermore the iem put user policy
allows them to modify policies to change
access to resources and other things so
again just four of these 10 000
permissions
what about gcp well there are similar um
policies and permissions and tasks you
know within gcp or google cloud so i've
mapped these out for you as well i won't
go through them in detail but by virtue
of the presentation
and probably the recording you know you
can always go back and review these but
again in terms of external access
insider threat lateral movement and
privilege escalation that map these out
for you as well if you have a google
cloud environment
so what about azure
so within azure um we're all you know
familiar with the the solarwinds breach
and one of the things that came out of
that was a lot of customers coming to us
you know as part of our risk assessment
saying hey i want to make sure i'm not
exposed within azure as well and
microsoft you know had a really great
article cve and everything around this
around tracking and monitoring role
assignments right and ability for people
to change role assignments so that's a
really important permission we consider
within azure that you should at least
monitor you know whether you've got some
kind of ids logs however you're
monitoring that with your sim really
really important amount of that you're
going to have people that are normally
using this on a regular basis but
tracking that and having an audit trail
of that is super important we'll talk
more about audit trails later
and then preventing it
get to least privileges for each and
every one of your users review what
permissions are being used based on
their activity and use that to get do
away with all the permissions they've
never used
most of these cloud infrastructure
environments provide privilege on demand
permissions on demand just in time
access so if you remove a permission
that someone uses once a year and they
suddenly need it that's exactly why just
in time permissions
exist allow them to go ahead and request
that permission and it gets added but
you know it is a balancing act but does
it make sense to leave those other 9000
permissions they've never used including
admin permissions sitting out there
probably not right so you have to kind
of weigh the cost of you know getting to
least privileges versus removing that
one permission they might use once a
year
all right so bringing this back to the
cyber kill chain
we talked about aspects of infiltration
privilege escalation lateral movement
and exfiltration and discussed under
each and every one of these some
examples of that
we also kind of gathered some key
metrics across these three
key platforms to kind of bring out where
we find the most prominent risks
so within aws we find that more than 65
percent of all enterprises have an ec2
instance or more with access to all s3
buckets whereas that ideally should
probably be a lot more restricted for
the the actual access do we want
somebody to gain access to the
environment by accident have access to
all s3 buckets or narrow that by really
restricting that broad access to what is
really ideally need needed for that
particular vm in that particular
workload and environment
in addition more than 50 percent of the
enterprises have identities with
privilege escalation so we talked about
briefly privilege escalation but they
have those super admin permissions and
roles
and again you know usually the number of
administrators compared to developers
and everybody else is usually less than
one or two percent so should 50 of the
people in the environment have those
admin permissions probably not
within azure um we find more than 85
percent of enterprises have over
permissive users and service principals
a lot of this stuff's actually been
orphaned and left behind after a project
so a lot of this stuff isn't even being
used anymore so again looking at the
activity will help you identify these
leftover service principles roles and
more across all these cloud
infrastructures to say gee a lot of this
stuff is old it hasn't been touched in a
year or two let's go ahead and
decommission it
and then more than 70 percent of the
subscriptions have identities both users
and service principles with over
permissive contributor roles remember
identities with a contributor role can
significantly increase the risk because
they have the ability to delete delete
critical infrastructure so again you
want to be careful with those
contributor roles and we find them
excessively used overly used
yet
or over-permissioned but under-used
within a lot of these environments
and then in google cloud we find more
than 85 percent have user-managed keys
for service accounts that are not
rotated we talked about that in the
context of an aws environment and more
than 80 have service accounts with over
permissive owner or editor roles uh
either directly attached or inherited
from a folder or organization so again
just speaks to the least privileges
all right so the other half of the
presentation how do i fix all of this
stuff
it's really really difficult to do this
manually when you have tens of thousands
of permissions and and
when you start to think about this if
you're multi-cloud or hybrid cloud now
your problem goes up exponentially
how you perform this and the experience
behind it really requires a good
understanding of the authorization model
for each cloud infrastructure
do you have different teams that manage
different cloud infrastructures most
likely right so how this is done may be
done by different teams but it needs to
be done in a cohesive way
and it does require deep visibility
right we're not just looking at what
permissions have been assigned but in
identifying not only those toxic
combinations of permissions what
permissions are people actually using
based on the activity
so really a lot of this can be pretty
daunting but there's a lot of tools and
approaches that you can use
within aws you've got cloudtrail access
analyzer access and advisor some people
using cloudwatch
within azure they've kind of merged
together microsoft graph and azure graph
and security center is actually pretty
awesome now and then within gcp you have
a variety of google cloud logs
there's a lot of different products both
commercial and open source
that are out there to help you analyze
these logs because they can be quite
extensive you could have easily have
millions of entries on a daily basis
so how do i analyze these logs let's
dive into the next rock layer here
within aws you can kind of log into
the cloud portal and in doing so you can
view each event within the history but
again if you've got a million events
you'll never make it through in one day
right
so
if you download the json for example you
can start to kind of comb through this
data and leverage some of these open
source tools or some of your own
automation to kind of go through these
but let's break down what are some of
the elements within
for example one of these events
in this first example we have one where
it lists out the a um the access key id
so this is revealing the authentication
method which means it's using aws access
keys not the username and password
credentials like you would within the
web portal
uh in addition keep in mind that by
default there's no multi-factor
authentication with access keys that's
why hackers love to attack using this
method so maintaining rotating these
keeping tight control around these
access keys is just like how you
probably handle secure shell today
also you can see what activities being
performed by the event name in this case
list buckets so based on a role policy
permissions in this case listing s3
buckets for your storage in the
environment and then furthermore the
source id
and the type of browser being used which
are kind of traditional things that many
of us may look at in other environments
but again you know are key elements
within one of these events that you may
find within cloudtra
this example is um
one that
instead implies that there's something
nefarious going on in the environment
here we have an event that looks similar
but is a little different than the other
in this case we have the same access key
id the same aws access key and list
buckets being performed but now it's
coming from a different ip and a
different browser
and while although this is a really
simple fundamental example it does start
to send some light on how you look at
this data and then furthermore how you
can identify anomalies
um aws authored a really cool um
pre-provisioning tool kind of creating a
clean room almost like you would uh in
forensics right so
all of us have seen csi shows where
there's a clean room they're all wearing
like white clothing and everything as
they take a look at evidence they've
gathered from the field
from a digital perspective right now
many organizations and and law
enforcement and others do this in terms
of mobile devices and laptops and things
like that and now in the cloud you can
create um a clean room and there's
templates sitting out there such as a
cloud formation template that'll help
you quickly create that and be prepared
if you need to do forensics in the cloud
the cloud security alliance authored a
really cool paper surrounding this that
gets into
how you can
fundamentally have a playbook around
your cleanroom forensics
and how you can take this along with a
cloud formation template spin that up
and actually perform this in an isolated
environment
as we jump a little bit over into azure
more
now within azure within the web
interface you have the ability to go
straight to the command line for bash or
powershell making it really really
convenient
and here we can actually see what each
of these two environments look like
whether it be bash or powershell
and a lot of organizations and
administrators and developers and other
people will use this but again be
careful of some of the you know
potential dangers here in terms of
security hygiene
in terms of um your azure cloud shell
bash or powershell
it can keep some files within your
storage account within your subscription
and if you make any kind of mistakes and
you're using the powershell commandlets
and there's some error logs they may end
up in the dot azure folder in your img
or image file system
and one of those as highlighted by
netspy
is a new um
azure vm commandlet that can be
vulnerable so it will log their
credentials
when creating new virtual machines
in their example they showed how
by looking at one of these files that
you could find
different things like an administrator
username and password so again you want
to be careful with the security hygiene
surrounding this
the the risk
is you know if uh if someone gained
access to a normal user account could
they then open up um for example the
powershell go into this look at the logs
and see if they could find a privileged
account like an administrator account
security center within azure has
improved dramatically and there's a lot
of good data here
this will depend on the type of
subscription you have but at the bottom
right you can see that there's a section
for most prevalent alerts
and this will tell you some of the most
prevalent alerts or risks going on
within the environment and you can drill
deeper into that this will expose for
you
a particular
event like this one here the failed
secure shell brute force attack going on
when it occurred what frequency is it
repetitive or not
and you can drill into one of those
individually to see what accounts
they're targeting
and furthermore from what ip and other
things that may help in terms of your
analysis
in this presentation we're really not
talking about office 365 but just be
careful that traditionally within the
exchange online environments that a user
by default any user may be given
powershell access now this is for the
default um
user
and it's just for their user account not
like an administrator with more broad
power shell access
but again if that's not needed for the
user you may want to turn off powershell
for most of your exchange users your
exchange online users that is
if they're not using it because if any
person's exchange online account is um
you know breached um you could
effectively maybe use powershell to do
automation to further target other
accounts download distribution lists
delete sent emails mark emails as unread
things like that and you want to look
for evidence of that within your unified
audit logs keep in mind that the unified
audit logs may only be
saved for 7 or 30 days depending on your
level of subscription so if you need to
go back six months you may not have that
data so some great powershell scripts
available from microsoft to actually
extend your unified audit logs save them
in a blob container and definitely
consider start doing that moving forward
we mentioned briefly around anomalous
behavior
and this can really be leveraged very
well in the cloud to identify a number
of things
for example when a critical resource is
accessed for the first time you're going
to have a lot of users by default with a
lot of excessive permissions and
excessive access to resources they've
never used so identifying when a user is
using a new resource can become really
important in in terms of
identifying initial iocs or indicators
of compromise
also if they're using that permission
for the first time signing in from
different or multiple ip addresses a
strange time of the day or multiple
attempts to gain unauthorized access
in terms of mitigation strategies
ideally you're also going to find you
probably have a lot of inactive
identities these may still be people
that work for the organization but have
never used their account within adbas
azure or gcp
this is low hanging fruit in terms of
mitigation and locking things down
either disable or remove those inactive
identities
to cut down on all of that excessive
access and permissions
and then you can set a policy it may be
90 days or something different in terms
of high risk permissions does dave the
developer need you know all of those you
know 5 000 you know permissions of the
10 000 that give him or her you know
access to any of those you know uh super
admin kind of permissions probably not
so you can remove that and again you can
base that on activity you've never used
them dave i'm going to go ahead and get
rid of them
and then just really starting to get
more and more to least privileges and
then leverage your just in time access
if somebody says hey i need access for
something new well that's what it's
designed for
and then in terms of your your hygiene
um for cloud permissions activities that
is uh you know again taking a look at
the high risk permissions if you're not
sure what they are because there's 10
000 permissions overall there's some
great tools that will help you identify
that what we know in doing our risk
assessments is that on average for
example in aws 5000 of the 10 000 are
considered high risk or admin kind of
permissions so roughly half which is a
lot
determine what high risk permissions
have been assigned to identify what and
where
and then furthermore um revise those to
remove the unnecessary permissions how
do you do that again we usually point
organizations to looking at their
activity saying like listen they haven't
used you know 9 500 of these permissions
so you can probably get rid of all of
those and then leave them with the 500
permissions they normally use
and then the anomaly in outlier
detection and rinse and repeat
so just winding things down
some great uh additional resources scott
piper uh runs summit route dot com and a
blog great stuff around avs security
rhino security um has uh um some great
reverse engineering type blogs the aws
forensic readiness one that i mentioned
earlier there's the link there and for
microsoft they have the azure hacker
hunting series
so last slide
stephen schmidt in a letter to a senator
following a prominent breach said this
even if a customer misconfigures a
resource if the customer properly
implements a lease privilege policy
there's relatively little an actor has
access to once they are authenticated
significantly diminishing the customer's
risk what we're really talking about
here is really that ongoing cycle around
least privileges to get rid of a lot of
those excessive permissions and toxic
combinations
and with that thank you so much i really
appreciate it i