thank you to all the cloud village team
for accepting our cfp
uh and allow us allowing us to speak
today
so let us introduce ourselves uh i'm
alex sierra i'm co-founder and cto of
tenchi security
uh i'm also accompanied here by uh leo
donald vaders
i have presented before as
uh jeff said in a few conferences
including last year on uh cloud village
on the topic on a presentation called
sas apocalypse called talking about
supply chain attacks in the cloud not
that supply chains have become a hot
thing
in infosec you know this year
but uh we haven't seen enough on the
cloud yet
um
leo do you wanna say hi for a little bit
hey hello everyone this is my very first
time
at defcon
and
this is actually six months ago i didn't
knew about this
i joined alexandre companies he told me
about the amazing word of cyber security
and here we are presenting
how about that uh this time last year he
didn't even know what defcon was now
he's presenting in cloud village
but let's talk about the subject at hand
why should you care this this talk uh
will cover
a little bit a specific uh issue
around doing serverless application
security in aws right and serverless is
a nice thing right serverless is all the
rage right now you see
getting a lot of good press
it's been widely adopted as far as as we
can see
and it's good right i believe it's
another step in the evolution of how we
build applications that we're always
looking for that next level the
abstraction
that already bundles many of the
concerns and abstracts away some of the
complexity of what's underneath
and you know if we didn't progress in
that sense toward higher level
abstractions i mean we would all be
writing distributed systems in assembly
right which i don't think would be fun
and so
the end goal here of course is to always
have
the developers work as much of their
time as possible writing business logic
and delivering value to their business
whatever that business is
uh and as little time as possible
focused on the nitty-gritty on you know
the operating systems the applications
the components and networks where their
applications are running right because
the business part is what's actually
paying for everyone's salary right
um
and so but the thing is uh in serverless
as any
as well as on any of the previous layers
of abstraction we used to do on
development
you always need to look at the
foundations of the infrastructure your
application is running on
and if before serverless you were
looking at servers and how they ran how
they're configured how they scaled how
would their availability look like
now you have to be concerned about the
serverless components and frameworks
that your application's been built on
top of and again worry about reliability
scalability performance and security
so if you're doing serverless in aws
you will be hard pressed to avoid
using amazon api gateway right or as aws
employees would say api gateway i think
right because they say army anyway
uh what api gateway does and you gonna
see it being used by pretty much any um
serverless framework that works on aws
like the serverless framework sim
chalice
you name it
uh it's a serverless scalable reverse
proxy service
uh it it's gonna receive http https
requests and it's gonna
route them to a specified backend most
typically when we're talking about
surface applications those back ends
will be lambda functions right you're
going to have a rest api say and you're
going to map each api endpoint to one or
more or or
actually you can map many endpoints to
one lambda but you're mapping endpoints
to lambdas
that implement the logic behind that
endpoint a client doesn't have to know
what your back end is and if you're
using lambdas again you're serverless
all the way through that that's what we
see being used the most often
and since you have this one point for
all this routing besides the actual
routing uh api gateway is a nice
integration point for a lot of
additional features like scanning and
caching
the server the service itself api
gateway as well as lend the scales on
its own based on demand
um
but you can also do caching this both
api gateway does its own caching but you
can also integrate it with cloudfront to
cache static content you can do
versioning of your apis really easily
you can do monitoring and of
observability so api gateway generates
their own logs but you can also
integrate that with services like x-ray
or cloud watch cloud watch logs things
like that
and as far as security goes api gateway
can implement input validation if you
specify you know the get parameters the
path parameters you can specify uh some
basic input validation is going to be
performed uh even before your lambdas
are called you can also use api gateways
integration point for aws web
application file or wafts
an api gateway most importantly for our
research
also allows you to do authentication and
authorization and abstract that away
from your code
a few key uh concepts around how api
gateway works
each api is an object that's not
identified by my id that's generated
automatically a bunch of
like an account id but it has letters as
well just not just numbers
you can define as an api gate to a user
stages for your api the way this is
typically used is if you're going to
have versioning it's going to be a stage
called v1 and then a stage called v2 and
v3 or you can have stages like
development or production or things like
that right
um and so
as far as authentication goes there are
a few ways to implement it uh you have a
ciam solution native to aws called
amazon cognito it's a customer identity
and access measurement solution
we're going to talk about it on the next
slide you have native capabilities on
api gateway to do open id connect or
oauth 2.0
you can also
god forbid
use iam policies and privileges to allow
uh entities or principles in your aws
accounts to call api endpoints on an api
gateway api
and
when you want to do something else
when you want to do something bespoke or
custom
uh you go to something that is called
lambda authorizers this is what you will
use when you are integrating let's say
with a third-party tool like off zero
jump cloud or things like that
so this is pretty frequent
before we go into lambda authorizes
which is the star of the the research
here let's
briefly credit my good friend andres
riancho from argentina now working as a
cso at wildlife studios uh he did a
research in 2019 specifically on cognito
security we won't delve deeply into that
here but just to give you an idea
so it's a cm solution uh from uh from
aws
uh but the way that a lot of the aws
documentation describes
uh developers should use it
is was not a really good architecture
the basically the idea was
you create users you assign them to
identity pools
they can authenticate using either login
password or social logins and once
they're validated against an identity
polls they can get a set of temporary
aws credentials that hopefully have been
scoped and have least privilege to just
access the things on aws they need to
access like calling lambdas or accessing
files on s3 buckets or things like that
right
um the problem is that the the premise
on the aws documentation was
that client-side components of your
application so your mobile app your
single page application
uh would do that
the the credentials would go back from
your api
into
the the the client and the client would
then call aws apis directly which is a
horrible idea it's like exposing your
database to the internet and every time
someone logs into your web application
you provide i don't know the single page
application of database credentials but
to just give them access to the right
tables and columns
it's just not the right level of
it there's an impedance mismatch right
there you want to abstract that away
through your api and have your api on
api gateway have the necessary
privileges to access any other aws
resources
and so what andres did when he found out
that was what people were doing
was he did a major search on the
internet including common crawl and
other
large data sets and he found a lot of
web applications and then some mobile
applications that had identity pools uh
ids hard coded into the apps to do
precisely that
moreover he found out that you don't
even need to be authenticated to get
some aws credentials when you create
you're using cod needle you can set
a level of
access even for anonymous users for
unauthenticated users so just using
those
he was able to gain access to over 13
000 s3 bucks that were not public
1200 dynamodb tables and 1500 lambda
functions
pretty amazing research check out this
link
check out his paper
it's really scary
now back to api gateway
let's talk a little bit about how lambda
authorizers work right so as we were
describing before
if you have an api on api gateway you
have a client that's going to do a
request and then api gateway has to make
a decision on whether this request is
authorized or not to route that to let's
say a lambda implementation then that
implements that api endpoint when you
use lambda authorizers the way this
happens is
before they call the lambda that
actually implements the api operation
they call a special lambda called the
lambda authorizer right that's also part
of your application it's a lambda
function that will get as a parameter
the authentication tokens that were
presented on that http request typically
the authorization http header right you
can configure it to be anything else you
want like a get parameter or another
header right
but typically that's that that's what's
used by default and that's what makes
the most sense for most people
and so what that lambda does is it gets
that information besides a lot of other
information about
other things on on that http request
it evaluates you know let's say that
authorization headers has a json web
token or has
some other form an api key you're going
to evaluate that using whatever logic
and code you want to write and you're
going to return
uh
information to api gateway that tells it
who this user is
any metadata that you want to have
passed to the back end of the api let's
say that the lambdas that implement the
api operations
and you're going to give that
api gateway what's called a policy
document
let's look that tells api gateway which
operations that person can and cannot
invoke so let's look at what that that
looks like this is straight from the aws
documentation basically
the lambda authorizer typically will
return a json object
that will identify the user right which
which principle it is
it's going to give a context with
metadata let's say which groups they
belong to which privileges they have in
your application or things like that
those things will be passed along as
additional parameters to the lambdas
that implement api operations on your
api
but also importantly there's this one
field called policy documents
that this is the part that api gateway
itself will process fully use that's
what it will use to decide which api
endpoints this user can call
and
shockingly
this is not in the form that anyone that
has ever used a reverse proxy a web
application viral would expect what you
would expect would be something like a
list of reg x's
of urls they can access or things like
that
instead
what the api gateway team decided to do
was to implement this in an iam aws iam
policy format
so basically they created an action
called execute api code invoke right and
then you have the traditional format of
allow deny and you have to specify a
resource
and if you look at how they encode that
resource here right it's an ran an
amazon resource notation as usual
uh with an execute api service right and
then you have the region as usual
because api gateway is a regional
service you have the account id and then
on the last on segment where the
resource would come in right they have
encoded separated by slashes the api id
the stage name that we talked about
before like v1 v2 or broad dev
the http verb so get put post delete etc
and the path the resource right so the
actual url path
of the api endpoint that was called
so basically what api gateway will do is
they will store that policy
and every time
they're gonna cache it you that's that's
the way again this is most often used
it's optional but it wouldn't make sense
to not use it
api gateway will cache that policy for
amount of time and every time someone
comes in with that same authorization
header it's gonna evaluate is the ap not
gonna call the lambda authorizer again
just gonna use the cached response to
say
is the end pointer calling match an
allow statement on this policy document
they're gonna build on iron based on the
request that's being made they're going
to compare that
with the policy that was returned by the
lambda authorizer right
so
this is
a controversial decision
i can't understand a little bit why they
did this right if you are developing a
new service at aws and you want to
achieve mvp status as much as possible
the iam policy engine it's just sitting
there right it's
tried it's trusted it's scalable it's
free for for users but it's not charged
music
that probably also has implications on
the price they had to to to charge for
api gateway right
uh but here's the thing first of all
this is a format that's not at all
familiar and intuitive to developers
it's much more so to ops into security
folk
than to developers right that people
that deal with iam
on a daily basis
uh and also it's not nearly granular
enough for the things you need to
represent
uh when you're trying to to to control
access to api endpoints right
so it's it's case sensitive that that's
not a problem
but
it doesn't have regular expression
matching
right so think about it think if you
have an
api path parameter that's part of your
path part of the url you're encoding ids
that are essentially external parameters
that instead of being provided to the
api in the form of
a get
parameter like after the question mark
it's being passed as part of the path as
as usual you know with rest apis
there's no good way
in aws iam policy to specify this url
and to be clear that you're just
allowing this type of value a specific
type of ad say a uuid or a number
on that middle of the url
because when you look at iam policy
syntax
it's very literal and all you have are
two wild cards you have a question mark
to signify just one character like the
dot on regular expressions
and you have the star
which greedily expands to anything on
that one iron segment
right if we recall how irons work they
have several segments separated by
colons right
and so if you have a
an asterisk anywhere inside this entire
last part
with api id stage http verb resource et
cetera you can expand to occupy the
entire thing it will not stop at slashes
it would not stop at anything except the
end of the string and this first colon
here right
so
this is a huge problem because it makes
it so that it's really hard
to avoid making mistakes and to apply
least privilege here
i'm going to let leo tell you a little
bit about a few examples that he built
that showcases this
leo
yeah so suppose those are the end points
for regular users of our application not
admins or staff users just regular users
that can create an account in your
website
as we can see we have
post get put and delete methods and all
of the routes has a prefix of dashboard
on it
so we do make total sense for a
developer that wants to be economical on
the way they write
api gateway allow policies to write
something like
we have there so our api id slash prod
for our stage name of production for
example
slash star to match all of the methods
suppose you get put and delete slash
dashboard for everything related to
dashboard that is what uh this
underprivileged users can do
slash
stars again to match for example
user
user with username user login and stuff
like that
so it's something for a developer to
write a policy like that it's very
simple and security and simplicity
security and simplicity goes together
right
wrong
actually when you write something like
that
please next
slide
uh we will out sorry would you allow
other routes as well so suppose we have
these other two rounds that are reserved
for admin users with elevated privileges
the first one for example admin store
order credit card dashboard that's going
to charge users in their credit card and
the next one admin dashboard create
admin users that's going to let you
create another admin users the first
star will expand not just in this post
method but also
in admin store order and credit card
so the user regular user that has this
policy will also be able to call this
method
and now so please next and also the last
star will also expand the two
after post method admin slash dashboard
and create admin user so as stars can be
anything there can also be nothing so
independent if you have
dashboard in your route in the beginning
the middle of the or the end of your
endpoint definition
a user with this kind of policy will be
able to call these endpoints
please
now we're going to take a look in a
video of a deployed stack with these
exactly same
uh
methods we have some user methods and
some admin methods we're going to see if
it really works or if it's historical
please can you play the video
first thing we're going to show you the
end points but it's pretty much the same
thing you just saw two admin routes and
six other user routes
and then we're going to take a look at
our authorization function
it's pretty standard authorization
function every serverless developer
should be very familiar with it first
things first we we verify we receive an
authorization token if we did so we use
a dwt generator token to verify that we
could use a third-party authentication
provider but for the sake of simplicity
we just do a verify and after we found
that the user is has the role of user we
generate this policy with star slash
dashboard slash star just like we
mentioned before and this first star is
where we have our fault
now we're going to start fetching uh or
having requests against these routes so
first thing we're going to get a
dashboard user with the username let's
see if that succeeds yeah it's a 6 200
okay as expected
then the other without the username in
the end slash dashboard slash user of
course it succeeds
and then the boot method after that
and it succeeds as well we're not going
to test every one of them because we
have very little time but you got the
idea all the user methods are going to
succeed
just as expected but what about admin
routes
the credit card and create admin stuff
we've made a post on admin store order
credit card slash dashboard
we will answer this need with the same
token with the same rule of the regular
user not admin
and the same thing about admin dashboard
create admin user proving that
independent if it's in the beginning the
middle or the end if you have star slash
dashboards like star
it will match
see thank you
thank you leo uh and so it's important
to note that if you had a path parameter
and you were able to choose as an
attacker to have the word dashboard
there let's say there were entities you
could create that would be later
represented in some other url the simple
fact that it had dashboard there
that one
arn would allow you to call that
endpoint so you can see how it's pretty
easy to mess it up uh using stars right
uh and
more importantly
uh it seems like even the team working
at api gateway aws or at least a team
documenting
and doing other tasks for the api
gateway
seemed to have an incorrect
understanding of how the stars expanded
on iam policies right it seemed like
they themselves
were not fully sure how this worked uh
which is a shame because developers that
were diligent that read the
documentation for api gateway they would
have been misled by this documentation
so this is one example from the old
version of that documentation has since
been fixed we're going to talk about
this in a little bit but it says what
about you want to create a policy that
allows you to access any method in the
htcp method
any resource on any api but on the test
stage
right now with the way that the
granulatory that the policy document has
it's impossible to create a policy that
does that
but the api gateway team said you can
just say star slash test slash star on
that last arn segment and you're going
to achieve that
that's incorrect you aren't allowing
anything on the test stage but you're
also allowing much more if you look at
those four numbered examples here on the
bottom right
so what you intended was number one but
you were also allowing numbers two three
and four
right so it seems and you're going to be
seeing that
more and more uh on the official aws
content that the team writing this
seemed to think that stars stopped
expanding at slashes
which is not this is not how they work
right
you can see this is explicitly said on
that's later on that same documentation
page uh the api id can be replaced by an
asterisk for all apis that's not
what it does at all it's gonna
be able to be replaced by everything
else and also include the stage name
http verbs etc so you can see they do
that with every single field
whereas as if you could use asterisk to
replace just that one field
right again as we just saw that that's
not how this works
right
also aws provided
what they call lambda authorizer
blueprints they have which is pretty a
pretty awesome idea by the way kudos to
aws for doing this they have on the aws
labs github account they wrote like a
skeleton
a blueprint of how an and lambda
authorizer should work on a variety of
languages so you know
javascript with node
python
go rust et cetera
and then developers can take a shortcut
instead of copying and pasting from
stack overflow they can copy and paste
from aws's own github repo
the problem is the repo had that same
incorrect assumption
you can see they had comments saying you
could replace any one of those things by
an asterisk and and that wouldn't affect
anything else
you had default values for api ids or
stages or methods being asterisk on some
of the language versions
right and again that's leading your
developers towards making a mistake
and even the aws console reinforces that
incorrect assumption
for example this is the console for api
gateway so if you look at a particular
method or a
better request
you can see that it's creating that same
arn format it's giving you the api id
and then as an asterisk for the stage
and then get and then a path right it's
using just
replacing just the stage by an asterisk
when it's showing you
what this
arn for this particular operation would
look like
um
and the same in the lambda console if
you go to a lambda that's being used as
the implementation of an api endpoint
it's going to tell you there's a trigger
from api gateway and it's going to give
you again something that looks like that
invoke api arn format
and it's going to be
api id and then the method
and the stage are going to be asterisks
so
all around ews the epi gateway team the
ws console team the team that wrote the
blueprints
seem to be working under this incorrect
assumption
that
stars expanded differently
than what they actually do and that has
to mislead developers right so let's
look at at a little uh
at actual examples of places where this
came to pass
leo
yeah now looking at real world scenarios
it's not so easy to find too many
serverless applications that are open
source but we find one that is pretty
interesting in this case
it is an application that makes
difference between guests and admin
users and because there are no such way
to represent using regular expression
for example the parameter paths of the
api these developers just wrote
something like get slash problems slash
stars let's submissions and for example
pose slash problems slash star slash
submit
thinking that it will allow for example
just using the ideas of uh these
problems that are
getting the submissions posting and get
the submissions but in reality he is
allowed much more rounds for example if
he has an admin route that has something
related with uh submit in the end or in
any part of it
like we have using this example get
problems full bar submissions or post
problems full bar submit
uh this the guest user with this policy
will also be able to call uh this device
so this is one of the examples we found
no there are not so many uh open source
service projects but
this is one a very interesting one
and also as a developer we everyone in
their life say
well it works on my machine
and this is exactly the case with when
we are developing serverless apis using
serverless framework
as npm says this service offline plugin
has around 130 134 000 downloads per
week so it's not something not used it's
very useful and it's what i use it all
my life to developing uh serverless apis
and it has a difference between the way
it be it comports between your machine
and when it's deployed to aws in this
example we are fetching a post we are
doing a post on
a route that is dashboard
our stage dev and slash dashboard and
that's it
in the server it's offline plugin that
we're going to test in your machine
it uh says it's forbidden for 103
forbidden you can't do this but when we
deploy the same stack the same code to
aws
it allows us to call this method
and it's very bad because
probably this serverless offline plugins
is being used in ci city validations and
manual validations and there are leading
to developers believe they are safe
in reality when they are not safe at all
this is a consequence of im policies
being very complex and much more
familiar for security and knobs guys
than developers and as developers right
tools for developers we end up in a
situation like that
we can't reproduce the very same
situation we have when deployed in our
local machine
yeah so it's sad even developers that
try to be do the right thing and test
their application thoroughly to make
sure things that should be blocked are
being blocked if they're using really
well known and good and awesome reliable
tools like serverless offline they're
being heard by this we submitted this as
an issue as you can see uh we're not
sure if they have fixed the the problem
yet
uh but we hope they do soon and if we
have the time we'll even try to write a
pull request uh to to assist in this uh
so once we found this this was on april
uh of this year uh we started so what we
did was responsible disclosure we
reached out to aws we reported the
issues we found in the documentation on
the console on the um
the aws console
we had a variety of conversations with
them going back and forth
they were
very responsive they were really
professional they were really concerned
on doing the right thing i have nothing
but good words to say to them and
especially
i'd like to thank mark ryland for the
the role he played
in intermediating helping mediate this
conversation right
the only thing i would say was
a little bit less than stellar is the
fact that they made a decision not to
update the aws console they continue to
use stars as a placeholder for any api
stage or any api id
on the lambda console and the api
gateway console which i i i wish they
hadn't i wish they fixed it
but uh they didn't in any case
the documentation for api gateway for
iem and the lambda authorizer blueprints
was was fixed and leo even helped them
on the lambda blueprint stuff by
submitting a pull request
and so this was very collaborative and
we thank them for
for being so nice about us
and
the other thing we tried to do was
thinking look we have something here
that is getting external input
and based on this input it's generating
a policy that's going to be executed by
api gateway
so if we replace those words with a sql
command that's going to be run by a
database
we started thinking hmm maybe we can do
an iem policy injection attack can i
send some form when there's a lender
authorizer that uses part of the
request even if it's just the
authorization header or maybe a field
inside a json web token to build
uh the document the policy document
right
can we
escape the quote you know
and insert an entire new statement on
that policy that allows for asterisk
that allows for everything
and it turns out we can't we cannot uh
aws made a choice that lambda
authorizers cannot return strings it
fails if you try to return a string with
that response format it has to be a
native language object like an object in
javascript or a dict in python
and so there's very little opportunity
to escape the string and try to change
the structure of the document so that
kind of injection attack
uh was pretty thoroughly foiled
we don't know if it was just a happy
coincidence on how lamb does work and
their wrappers do the hard work of
converting things to json uh in any case
it it did the trick
but what about inside that resource
string not escaping it
what about we have a resource that's in
a part of the policy that's going to use
external data let's say we get a
user id inside a
json web token and we are able to change
it to an asterisk okay that might work
that might actually work
but no one's going to do that right
oops
uh it turns out the very
aws blog entry where they announced
the introduction of
lambda authorizers had exactly that
problem they were
deciding how to build that arn
by adding
slash users slash the content of a field
on a jwt on a json web token and of
course
json web tokens are signed if you can
tamper with json web tokens you're
already you know on the second or third
stage of an attack chain
but still uh you know
saying uh algorithm equals none on jwts
is not exactly no
uh and there might be other ways in
which you can subvert or create uh jwt
is under your control if you can
or if you can find other ways to inject
data that you think are going to end up
into a policy document
try changing them to an asterisk or
appending an asterisk after them and see
how that changes the authorization
decisions of their api
this might actually work
right
so we're going to close up
with a few recommendations right on on
things that aws can do and that everyone
else can do
starting with aws
the first thing they could do that's i
mean they're pretty tied down with their
decision to use the iam policy format
right it would be a big change for them
to add an additional format based on reg
access or even harder i think to add rig
access to iam
right
but one quick thing they could do is add
conditions uh currently that
execute api invoke action does not allow
for any conditions it's just standard
part of how iam policies are
evaluated if you had conditions for the
api id the stage the method and the
resource separately
you could do that thing where i want to
allow just the test stage for any api on
any method or any resource it would look
like the right side of the slide
it is currently impossible to do this or
anything without conditions that
replicates exactly that
right
but if they added conditions
this would already make it so much more
granular like you wouldn't believe right
uh please please reconsider the decision
not to update the aws console change
those placeholders you know curly
brackets api id curly brackets or
anything other than asterisks you don't
want do not want to
lead developers into using asterisks
when they do write the arns themselves
so don't do that
and
just the conditions and just exchange
here still doesn't fix the problem of
how to deal with path parameters
so that can only work if they extend
and add more functionality to iam as it
exists today and my understanding again
either more functions that match regular
expressions on conditions or if they add
a new wildcard that doesn't expand
beyond slashes right that's not the
question mark and the star right one of
those two things has to happen right
and finally it's pretty it seems to me i
know nothing about how aws works
internally but it seems to me like they
they need to have more senior security
people on those famous two pizza teams
when they're designing how products work
because this is the sort of thing that a
security person probably would have got
early on when they were designing the
service
right for aws customers if you're using
stars on those uh invoke api arn's
only use it at the very end as a slash
star at the very end to allow a prefix
for your path
anything other than this and you're
treading on on dangerous ground right
you're in quicksand
you got one minute okay you might be
able to use deny statements to limit the
impact of some of those expansions so
look into that
uh do not use lambda authorizers as the
only way to decide authorization since
specifically you cannot handle path
parameters well you might need to
recheck at the lambdas that actually
implement the api endpoints if the user
is authorized or not right
if you used any lambda authorizer
blueprint code update it based on the
new version
and
that's it
those are the things we're recommending
let's see if you guys have any questions
that we might be able to answer hope you
enjoyed it