hello and welcome
my name is ron soto
and i'm here today to speak to you
about detection challenges
in cloud-connected
credential abuse attacks so let's get
started
my name is ron soto
i've been in the industry for around
10-15 years
i used to work that
for lexic
currently working
as a principal security research
engineer splunk
and like i said before i used to work at
burlex then i worked at caspita
a little bit of time and akamai
i co-founded had miami
in pacific hackers i also created my own
ctfs
and some of them you may have heard them
one of them was command and control
and the other one is
um no quarter cdf
so
let's
let's talk a little bit
about this how the cloud permeates
or permeates inside the perimeter
i think it's
pretty clear at this point in many
environments
even in small businesses
where
i myself i used to be a system engineer
have my own little it company
and i used to actually install microsoft
smb server i remember setting up
exchanges uh which
sounds crazy but yeah i used to set up
exchange servers
in
offices of 12 to 15 to 20 people
or even crazier things such as a
blackberry
enterprise server as small businesses
so because as
basically that was the the way things
were supposed to go and we didn't have
the evolution of the cloud nowadays
you really don't need a domain
controller for example in small
businesses
and many large enterprises you you still
do
although we see things such as opta
taking over a lot of the
the the market
but basically as you can see here
uh even microsoft uh has
made a huge move into the cloud so we
have
office 365 exchange online you no longer
need to have
a
microsoft smp server
or an exchange server
or
if you don't want microsoft
you can have pretty much
most of it
uh by using google apps and then you can
have email presentation messenger
telephony
video conference you name it so
these are some of the examples of how
the cloud
now gets into
inside the perimeter many of those all
servers many of those uh infrastructure
which uh some of them referred now as
mutable uh infrastructure have gone away
the uh
infrastructure at the cloud level is
referred as immutable
where usually the
providers
basically apply
the setup
and
updates and patches to it um you can
also see uh in this slide that i
actually placed a graph
where uh
other examples
of things that we
use almost on a daily basis such as
code repositories for example like
github
or gitlab
or we have things
that we use for uh deployment of
technology or development of software
such as circleci
or any other
technology
that is
applied to the same
end goal
so here's an example
of
basically
how
the cloud
in
the perimeter
can basically
merge together we're going to talk about
the
what we call a converge parameter
and in this case we're looking at two
couple of examples in this example we're
looking at
amazon transit gateway
or a shirt being it as you can see here
we basically are
seeing environments where
this type of service
basically
um
allows
the cloud real estate
to be part of your perimeter
and for all intents and purposes for
your users this is
invisible they may be placing codes
saving code moving files
doing all kinds of operations
logging into a server by ssh
copying files across the one
pushing updates and
basically
uh they won't be able to realize that
some of those instances that they are
interacting with
are actually not part of
your formal perimeter they are in the
least uh
property or lease or rented property if
you want to call it
from the cloud providers
so
in this case uh two clear examples of
how this can happen
this can also happen in an informal way
which i will cover later on in this
presentation
[Music]
for example
i call this the
the whole potato why because there's
always seems to be a no
not a clear idea it's very clear for the
providers but for the customers
uh it's almost like a hot potato uh is
it mine is it not did i pass it around
um
the truth is that many of these
providers do have
a line they they draw into
what your responsibility is as a
customer that have
um
plenty of
uh
cloud real estate and what they provided
you and depending on the service model
we know some of the service models uh or
the most popular surface models from the
cloud providers are things such as
software as a service
platform as a service and infrastructure
as a service
in contrast with on-premise which is
basically when you own buy
everything you have it in a way
where
um
you can do whatever you want but once
you start mixing up these things you
have to be alert
and be conscious that some of this stuff
you might just not leave it to the
provider there seems to be this false
notion
that because for example we're talking
about like i said before a mutable
uh uh infrastructure where many times
the cloud providers do
actually a great job
updating and hardening their resources
that still does not guarantee
the fact that this
infrastructure can be vulnerable or can
be attack i'll give you an example
many years ago there was something
called
hard blade
and i was working on a company that
basically we're looking at
30 percent of the internet at times and
what happens is
then the amount of servers even with
environment i repeat
even with embargo because we were
within an organization where there were
embargoes for disclosure of such
vulnerabilities and this in particular
was uh under embargo even with the town
of embargo the amount of service we had
to patch it was almost impossible to
finish it
in time
and the reason why i'm saying this is
because these things happen and will
continue to happen so when you have
a
number of different technologies you may
have
containers that run alpine for example
you may have servers around ubuntu you
may have
a
windows servers out there as well
which you can set up at amazon or on
azure
definitely
it adds to to the mix it adds to the
the attack surface
uh considerations and in scenarios like
this um
even as these things are mutable even as
the cloud providers do their best to
protect and harden them
you're still responsible for it that
your data is yours applications are
running in there that may be attacked as
well because there's there they became
vulnerable
uh because of the disclosure or even in
zero day these are your responsibilities
uh and these are uh uh
some of the
considerations that we're gonna see
today as i uh walk you through some of
the attacks
that i was able to research
during this year and how to address them
so coming back to this right
we are talking about a
converged perimeter converge perimeter
is
basically
the cloud real estate that you're using
leasing or whatever form or commercial
use
that you're you're doing or you're
giving it plus your perimeter perimeter
technically ends either with
the border
or your when
or your internet gateway or routers
and then once you have things such as
the examples i just gave you then you
have a converge parameter you basically
if we if we're looking back at
this figure where you have
uh real estate from a provider plus your
perimeter
and your users are
formally or even informally
accessing cloud resources then what this
means is basically that this is part of
your perimeter it's a converged
perimeter even though you are leasing
or
renting
cloud real estate
so
the reason why i named it like this is
because we
are going to look at certain type of
attacks
which is basically an abuse
of
reuse credentials which is how microsoft
has looked at it and the reason why this
happens
uh and and
hence
the title of this slide is because
basically when you have
as i am bringing you up
in my past lines disparate technologies
right all kinds of operating system
versions all kinds of applications
you need passage
through this
uh entities
for many reasons functionality
continuous delivery continuous
integration
deployment of technology
you need mechanisms or
in this case
tokens or credentials or secrets that
allow you to establish
authentication
authorization
um and some sort of a trust
in in this infrastructures
that are now more complicated because
you have a converged perimeter and you
have to interact with all kinds of
desperate technologies so
yeah
it is a feature these secrets these
tokens or certificates for example
you may have
federation services which we're going to
look at
later on
you can use also certificates for
for
authentication
are things that
are used to facilitate the operation in
passage many times of
data many times of code
many times of
deployment of technology so uh here's
where we're focusing at this point
because obviously
uh these are the the key elements that
we're gonna be looking at
because
by obtaining this type of secrets token
certificates
attackers can do a lot
so
coming back from
what i just said
when you are in a federated environment
and you use
this type of
technologies obviously as you can see
here is a graph where basically we are
looking at this part of technologies but
this disparate technologies can indeed
communicate
by using
let's say you could use a federation for
example where you use either oauth
tokens or you can use saml
which is a
a mechanism
of basically
uh authenticating between entities
and what this shows here is that this is
the realities of what we're
facing today when we're looking at the
interaction of cloud technologies
and all kinds of
operating systems mobile systems
remote offices you name it you need
something that will allow the
interaction operation and passage
and most of all
you need
the users at the end of the day the
operators
to
use efficiently technology
if we were to put
or force for example
multi-factor authentication for example
between many of these internal systems
an external assistance every time this
basically will become
suboptimal and and
and basically all the benefits
that we use we we take from things such
as continuous delivery continuous
integration or even deployment of
technology that that basically knows it
that basically
uh was slower to a point where
you had to consider is this even worth
it
so let's look at some key points of
federated environments since we're going
to be talking about abuse
of uh cloud-connected federated
environments uh things
that we need to look at are for example
uh formal connections
uh of perimeter
and cloud real estate resources what i
call the converge perimeter
uh they help this this this the reason
and
uh to bring it up uh in a way uh
what why is it that we're looking at
federal environments and why are they
important well they they they allow the
increase of cloud utilization
so as we said before move on move on uh
on-premise resources to the cloud
uh the increased uh this uh utilization
also provides the ability to increase
uh the resource availability or even
geographical reach we're talking about
what they call silicon valley
elastic technologies you can expand or
shrink as you need it
you had to have a local data center or
you can have the the minimum you can
have inside the perimeter but take
advantage of this and this obviously
it's not the same thing to use
or
to create environments that you may need
at the moment and then pay for usage
that is a wonderful thing that has
helped a lot of companies
and like i said before
uh we're looking at standards that
allows the passenger of data
identification authentication that's why
we're looking at things such as tokens
certificates
even passwords
and api keys of course
formal federations
are usually the ones where there is a uh
to put it in a certain way a formal
technology in place that i identifies
and authenticates
entities between each other
uh and that implements obviously a
stricter
uh control access of this resources
and facilitates the use of it like a
single sign-on for example
uh at the at the at the uh the gui level
and at the user level but you can also
have what i call informal federations
the formal federations are places where
because of the usage of so many cloud
technologies
in
the state of address
credentials
secrets
certificates
api keys
that are
inside
your perimeter for example by developers
even though you have a formal federation
an attacker
if such attacker access
those resources they can people into the
cloud and even in some cases depending
on how your setup is they can even
people from the cloud
to your perimeter and that's
what we're trying to focus here today
that we had to understand
the risks
of the expansion
the the
the subject of the converged perimeter
so the converge perimeter risks
scenarios some of them
obviously we have
credential leakage in public
repositories
we have for example use of vulnerable
components from cloud like for example
open source libraries or containers
usually download it
embedded
they go through uh either for deployment
uh because they're hosting a database or
they're hosting an application that's
needed or in some cases in development
where uh they have the perfect
environment to host a specific
application or
a specific api that's needed they don't
get scanned they get downloaded from
public places and
there's that is a big risk
you can also
take a look at this posture of cloud
apps
and infrastructure
that may lead to
internal access
and
i will give you an example of a campaign
uh several companies actually rather
that we have observed where where
specifically this scenario
interesting enough provided
from cloud
to internal access the reuse
of federated credentials
attacks such as the golden saml
discovered by cyber arc
or things such as uh off
talking hijacks uh or past the cookie
for example we'll see some examples of
it with actual examples
and then
we can see
in the case of
federations people from cloud providers
from one cloud provider to the other i
actually seen that i researched it
and then from there
try to
move into
the converged perimeter the internal
resources
so here's some of the examples
of uh
cloud connected credential
um here's a a an attack
which is not that difficult
what you see on the top left is mimi
cats
that was run against a windows host
and then on the right you see
for
purposes of approval concert you see how
i was
not logged in i downloaded or rather
discover or reveal
mimikatz against a windows host that was
able to get the
specific cookie
then
via the the same browser
functionality inserted a cookie and
there i was i was able to basically
login back
this is not new
but it still works
i did it recently and this is something
that can be done by somebody that
has the possibility of
access
to in this case an employee so
this is the first example
here's a second example which was part
of the research i did with uh
jingle one from netscope
and here uh jengkol actually did an
amazing job uh by uh
identifying uh
things
such as the uh off
uh talking hijack from gcp basically as
you can see in
uh this slide you can extract the token
and then once you strike the token
this can be done basically by either
access
to an endpoint or
a
physical possession
of a device that has obviously the sdk
uh or the uh
cli
for gcp and then all you have to do is
refresh the token
and just so you know mfa does not
protect against this because if the
token has been authenticated already you
can keep refreshing it for example
that's another case scenario
of
oauth tokens
hijack
here's a uh one that had a lot of uh
um uh
i guess uh chatter uh doing part because
of uh
its use in
in the
apparent use in the campaign uh during
solar winds uh according to sisa and uh
here's the
the i took this uh
graph from this uh signia advisory uh
cyber arc did also an amazing job and
one of the researchers uh actually
created a tool
that uh basically stretched
uh the saml assertion uh creates a
forged
saml assertion
and then passes it so once you do that
as you can see in this
graph this is a federated environment
and in this federated environment we
have basically likely on the main
controller and in this domain controller
we have active directory federation
services you set up a
uh
federation
and then from there
you communicate and make it easy for
your users to basically use either
office 365
or exchange online
and what happens is if you're able to
basically
forge
and it's not easy not to say that it's
not easy to forge it
the providers have gotten very smart
they have become
especially very restricted on the
attributes because that's where you can
change things
such as the duration or
uh
the
the user
uh so
when you look at the seminal assertions
there are things such as attributes and
then these attributes are usually
checked
uh based on the identity provider
basically
uh
to prove uh in
a certain way
before
that they you're allowed access um
and the reason why this attack
was so successful is because if you
this can still happen by the way if you
are
taking saml assertions uh
not likely with this providers these
providers actually make it very
difficult for you to do a a
sort of a golden seminal attack but if i
know that there are other open source in
other
even internal cloud setups and you are
not
strict on the checking of
saml assertion attributes
this attack will work
and it's very dangerous because it
basically allows access
uh to every single resource that um
is dependent on
the interaction with such assertion
so here's another uh tool
that obviously had to do
although it's a tool that is ran
against um
a
likely
active directory environment however can
provide a lot and make work
in federated environments
less
coming back to the industry microsoft is
still the leader microsoft is
for the most part the
enterprise operating system uh active
directory and um
there are tools such as adfs dump which
are uh not only you have to compile them
every time you run them which makes it
incredibly difficult to detect
but
they are
great at obtaining
keys uh that will help you forge
uh certificates or um
even um
use these keys to
um
forge sessions or make requests for
further
uh identification and recon
of
the active directory
uh environment where you are so these
are things that
you had to consider because
very hard to detect and we do have
um a little bit of a
way of sort of
looking at this from a certain
perspective however there's not a silver
bullet against tools like this
so here's a a
like i told you before
uh some of the actual um
campaigns that we have seen uh whereas
um
you go uh basically full circle
um and how you go full circle well okay
so let's say you have exchange online
or you have
uh
yeah exchange let's start with exchanges
let's list let's say you have an
exchange online uh you can
many times we've seen
environments where there is no
multi-factor authenticator
authentication
so
credentials
actually remember we talk about
federated active directory right passage
authentication
identification
throughout many environments
so
we have for example a
exchange online right
we have tools that can we don't even
need tools we can actually if this is
for example was a
misconfigured exchange server you can
actually use powershell
to to elicit a response that will give
you
sort of a directory information
usernames in some cases even
depends on how it was configured
a
way
a bunch of information that may allow
you to target users or you can just
simply turn a board for brute forces and
let's say you're able to brute force it
and because this is obviously a
federated active directory environment
most likely the user repeats it's the
same password for other services so all
you need to do next is to find
the vpn
of this company or
rdps or servers that might be
uh exposed to the internet
so you are uh
technically able
to log in
because you have obtained
the password right so you have obtained
the credentials uh you were able to log
in in exchange online you were able
there are other tools
that you can use or even do it by a
powershell and obtain format information
uh depending on what level of privileges
you were able to obtain these user
credentials and this user in particular
and then from there
you can attack login and attack
extract for the credentials use things
such as um
imicates for example we looked at it in
the
past the cookie
example or adfs dump which we also
looked at
and then forged credentials and then
here's the full circle you go back to
the cloud now here in
gcb you're in
aws you're in an acer
and this is what i wanted to bring up to
your attention today this is happening
right now uh unfortunately
many times the expansion of the cloud
and the connection of such cloud
uh that has brought a lot of benefits
you know nothing better
that you know
the old times i want to my all wa i
remember i used to put all the uas
with
with no multi-factor authentication or
even vpn it was it was seen as how cool
we can access it and check our email we
don't have to go into the office right
that was that was actually the the
original purpose of this thing so well
there they
times have changed and these things can
no longer be exposed to the internet
without some sort of protections
or
we've seen actually cases
where
hardcore keys
api keys
passwords in
um a code uh had led to uh
breaking through uh either vbn
uh this happened this just happened by
the way this happens at colonial
pipeline they broke in via vpn they
didn't have multi-factor authentication
or you can even exploit the vpn uh like
it has happened sometimes
and
there are and i can tell you this
because i see it and i've seen it there
are tons of rdp servers that don't have
multi-factor authentication and many of
them
are federated
and with that then obviously you just
have to execute your tools the best to
your ability
and then if you wanted to be in the
perimeter
either because you access by a vpn
or by rdp then you can expand further
into whatever real estate
[Music]
cloud properties
this vulnerable environments have
which could be in any of the
main providers
that we all know
and were aware of so as you can see it's
almost like a circle you can go full
circle um and now um
let's say i was able to get in into
or even break and let's look at the
other side let's say i was able to break
into a certain a logging server that is
part of a connected formally connected
uh
environment by one of those services
that we just talked about
you will be able to basically
access service or or even
endpoints
that will be reachable
because of this extended
network
and converge perimeter
so
this is basically
something that complicates
the current situation that we have and
sometimes it gets even more complicated
with uh
where lines are not clear of ownership
between the providers
and the clients and now
we're seeing an increase in this type of
attacks that obviously require the use
of a
feather edit
cloud-connected environments which which
i would say are the the perfect
environment to do these types of attacks
all right
so with that in mind
how do we approach these attacks
is there anything we can do
yes we can do
uh we can do a lot we can approach
this attacks
and this is something that has
hopefully has started to happen
in many enterprises but because bringing
back what i said initially but because
there's this false notion that somehow
your provider is going to protect you
or you don't have to worry about it
it was not very clear and it didn't help
to be honest
and this was one of the things that i
realized as i was doing my research
the cloud providers they want you to
stay with them as far as analytics and
as far as uh um
uh even obtaining locks from them
is incredibly difficult uh it was uh it
has become easier lately but before it
was
like talking different
languages
schemas
and that poses a challenge for system
admins
for
aws administrators asian administrators
there is
companies are trying to avoid uh
marrying to a single uh vendor
so when every single vendor does not
want to play with the other all what
this does unfortunately i have to tell
you because i've seen it myself it's in
pairs and the other customers because
when you have a an environment that has
multiple vendors the best ways for all
of them to collaborate the the best way
for all of them is to provide
the easiest way to uh
disability into their environments and
that's what i'm trying to approach here
the approach that we did here is we
already know
we can observe
uh and analyze the endpoint right so
give you some of the things that i seen
uh and related or associated in this
attacks things such as the the
extraction the certificates or keys
tools like insert util.exe
on common processes running in servers
uh all of a sudden that may give away
for example the user adfs dump
uh
things such as registration or or
registry keys rather
used for privileged escalation or the
use of mimikats which
for the most part a lot of the vendors
can detect pretty good
and then
we can take a look at the cloud
providers
and in this uh
when we look at cloud providers uh
obviously we had to we had to focus on
the specific of every
technology right there are no universal
technologies for federations
some of the standards like the samoa and
the oath are being used
by
these providers however the products
themselves use it in different manners
that is why i'm putting some examples
there of some of the things that we
looked at and i'm about to show you
uh
things that are even more specific that
we believe
will help you detect these attacks
because it's challenging i can tell you
if we take away
one side
of this slide you will not be able to
detect this
you will not
like the pass
the pass the cookie for example or the
oauth i look at both loggings
from the longman perspective
it looks like anybody
and sometimes you can use for example
security groups
even using security security groups
depending on
the policies that you have
uh you may not be able to see it
and even worse if you don't have
visibility at the end point level so as
we have expanded
into a converged perimeter
we need to expand our visibility and
that includes
the cloud real estate so we have things
that
we need to look at and we need to get
alerts from it if you don't if you're
not getting alerts like for example if
you have an ultra 65 that has excessive
loading errors
then you're missing out you you you may
be under attack right now and you don't
know
but consider this if you have an old 365
from a federal server where where
passwords are repeated then the the
scenario that i just gave you it is
likely to occur it's even worse as we go
higher in privileges do you have
soup evidence you have admins per one or
admins per
land segments i seen that
layers of administrators
these are things that you had to
consider
that if they get compromised
they may indeed provide access
to all these resources
here's a
breakdown of
the actual uh detection searches that we
uh uh in my i developed this with my
with my teammates at the splunk threat
research team um and this is part of
what we do uh so
here's some of the ttps that we're
looking at endpoint for example they use
the searchutil.exe
which basically
allows you to see
the execution of it
which is usually very rare this is not
something that happens commonly um on
common process is an endpoint
which
may give you um
the presence of the execution tools such
as adfs
dump for example
registry keys to use
for privileged escalation
and persistence this is uh
one t1546
um
then we have things such as mimikats
which is we can detect pretty good this
is uh potential axis
uh
which is t103.01
this is obviously my third dtps
and we even have a no one that is
actually
mimicked by a powershell
in event code
4703 remember for you to have this
you need a system on policy
and you need a gpo
that will allow you to get the event id
that you need
most of these event ids are not audited
by default so you need visibility but
the visibility requires several steps
just like the cloud
you have to select a
sometimes you need to sort of sync your
log somewhere and then they need to get
picked up by some service that then
sends it somewhere else and uh this this
can be actually cumbersome
you have to do your homework at the
endpoint as well you have to have a
a strict
system policy with all these
consequences you know there will be a
lot of logs um they may have to be
rotated uh there's a lot like log life
cycle that you're gonna have to consider
and of course with uh evtx type of
auditing you had to know the right gpos
and uh uh apply them and and after that
obviously you need a place to retrieve
this it doesn't have to be a spoon it
can be eok
or any other
sim
or you can build your own sim right but
the bottom line is the events will be
the same
the logs will be the same
so let's move on and now let's take a
look
at the actual cloud
ttps so for example
we have things such as aws ml access by
provider user in principle
we have
this one as a giveaway aws saml of the
update entity provider is usually when
some of the
attributes are modified this is
something that is not very common and
that you should have an alert for
any needs obviously to be locked
properly
um
you have things such as uh excessive
single sign on logan errors
um
service principle in order 65 that's
also
an event in itself that needs to be
monitored
or the addition of a service principle
is also an event that needs to be
considered
an event to be monitored by itself
uh and of course uh o365 neo-federated
domain remember when i told you about
people think
from cloud to cloud i was able to do
that
by uh by basically adding a new
federated domain
so these are the type of
detections that will give you
a way to address
the abuse
of
federated
credential abuse
so here's a an example of a
[Music]
search i created uh at the
using spl code in splunk
and basically what this does is give you
obviously a deleted stuff
so um
or rather sanitize it
not to give away
uh
some of the
sensitive stuff in it but basically what
you get here is when somebody went and
said hey let me change the uh the samo
provider and that was me by the way
trying to replicate some of the attacks
and
this is a way to not only
investigate but in combination with all
of the other searches
at the end point
and at the cloud level then we can get a
very good disability not saying we will
detect a hundred percent of everything
but we get a good shot
uh of detecting this type of uh
attacks
the abuse of federated credentials in
cloud-connected environments
here's another example
you can find this if you're interested
if you're
uh interested in looking more the code
we have a github
where you can actually see the actual
code this can be translated into other
technologies and other languages
but again i'm just showing how this
translate into that technology just like
i showed you the attacks i wanted to
show you
the detections and the investigations
and with that in mind
i'd like to thank you
for coming today and watching this talk
and
we're going to open for q a remember you
can find me
at rodzotto.net my twitter is
rodso
and if you have any questions please let
me know you can
email me as well at rod rotsar.net
and i hope you enjoyed this presentation
thank you