hi everyone
thank you very much for tuning in today
today's talk is about attacking
terraform environments by the end of
this talk you learned ways to attack and
secure your terraform environments
this talk would cover a lot of things
that probably not known before
one thing to say before we start their
phone by default is not as secure as you
would think you have to put a lot of
work to secure it and what i'm aiming
for is by this
by the end of this talk is to learn
about all the things that you can know
so that you can secure your terraform
environments and let's start
a little bit about me my name is mazin
ahmed i'm an application security and
offensive security engineer i founded
fullhand it's a security startup that
solves the attack surface visibility
problem for companies and
organizations and also i'm an occasional
bug bounty hunter i have
been acknowledged by facebook twitter
linkedin zoom oracle and many other
companies
and of course i'm in love with cloud
security anything related to cloud
security and of course uh
different things related to security
are my passion and that's why i put the
effort into having this research
made the agenda for today
is first we we're gonna go over things
uh a background on terraform
what styrofoam how does it work
and why do we need to learn about all of
this why is it important
attack victors and scenarios attack
victors and scenarios
and of course the most important part is
the demo which would be really cool
the recommendations at the end
and then would have room for questions
so first of all
what's infrastructure as code
back in the years
when
cloud was starting was becoming a thing
if someone would like to deploy
something they would go to the console
take their time in deploying their
resources
from the console
and if they would like to do this again
and again and maintain that they would
have to log in every single time to do
that which is a bit of a painful thing
and does not really scale that's why
infrastructure as code became a popular
thing in the devops and difficult
communities
why because you would be able to write
all of your resources and the cloud as
code and then you would be able to
deploy it
many times at the same time you would
allow room for peers with you so that
the people in your team and different
teams will be able to review it
it's amazing
so yep it allows that parts and of
course one thing that you can say that
once you have this being set up you
would be able to run it through
ci security scanning for compliance and
you can automate many things without
really having to have someone to review
everything
um uh being deployed
and of course when you deploy this and
using get you would be able to have an
archive of everything being deployed
it's just an amazing technology that
everyone loves
hopefully
and
what's their phone
jira form is an open source
infrastructure as code software made by
hashicorp
around 10 years ago
and uh
users define their code in a language
called the scl or hashicorp language
and then deployments and
and the entire infrastructure management
happens through terraform whenever you
write code you would pass it to a
telephone build a binary and then it
will
read the terraform state a state of the
current infrastructure and then it would
decide what changes should be done
why i'm talking about true form today
instead of all of apollo me or different
uh or cloud formation of different um
cloud orchestrator because therefore is
the most popular iec orchestrated in the
planet
this year a couple of months back they
have released terraform 1.0
the it has been a big thing for everyone
that is working on with terraform
and of course
it passes
100 million downloads
crazy numbers
another thing that is cool about
terraform once you have it is
you can write the same infrastructure
code
and
have it integrated with different
providers so you can write code that
allows you to deploy to aws or azure or
gcp or anything else
and
it's not only these
six technologies and six providers that
are there
they support more than a thousand
provider as of today
how
how you would be able to do this is as a
practitioner or a user you would write
your code then once it's ready it will
go through terraform plan and telephone
plan is like the way that telephone
would be planning your deployment
and show you
what would be changing
and then it would go through the apply
and once you click on run apply
it would be deploying
directly to your cloud provider
the thing is
when you are running apply and plan on
apply this means that
you are
checking that that drifts between the
current code and what is there on the
provider and if you are applying uh
changes this means that you have right
access on different things on your cloud
provider and who would have this it
would be terraform
so setting up their phone with grants at
high permissions that allows it to do
everything that you need to have it to
do
while it sounds curry
it's the default thing we have to just
set it up in a way that could be secure
and
not rely on the default things and try
to
think better about it
as i said
with great power comes great
responsibility
a single mistake here can lead to the
entire
compromise of
the cloud provider or the cloud account
so imagine then your entire aws account
being held for ransom someone would uh
like if someone compromised that they
would
go
into your club provider they would be
backing up all of your sv buckets
they would uh
incr or may uh delete everything they
would have it somewhere else
and then they would ask you for ransom
imagine this
ransom isn't the thing that you would
see only on your
uh internal network or
or in your laptops it's something that
can be done also in the cloud it's not
it's still not a thing today but i can
imagine that this would be happening in
the upcoming years
and as i said about the
terraform
states
the way that perform works is it would
be
storing all of the changes and all of
the additions that it made
in your cloud provider
in a file that acts as a database called
telephone state
every time you do a change it would
check it against
that file and then would update it
whenever there is anything
there are many ways to
have your terraform state being deployed
one thing is you can have it locally
second way that is
i think the most common way by
everyone that i know
is using a remote backend of
sv
so that you would specify a bucket you
would specify a key and then which
region would you like to have this
bucket at on and then terraform would be
automatically
creating this bucket for uh would be
automatically uh pushing the the
changes and the state file there and
whenever it updates it would uh
synchronize with that key
so attack number one or thing number one
if
you go if you have uh get uh get object
permission on the s3 bucket for solving
the state file then you would have
access to
secrets access keys and database
credentials so everything that is being
deployed
from
uh
for the organization
through terraform
would
have a copy of it
in the seat in the state file
so there's something like an rds
database would have their credentials
stored somewhere and that place would
have a copy in the state file same thing
if you create the an im key
access key then you would have it there
or any other secrets that you would
define
or the provider would be defining uh for
your phone
so whenever you get access to
uh to an organization and a
cloud account try to search for
if they have a sv bucket
that
looks like it's use uh it's storing
secrets for turf on it would have the
entire
instructions for you to go
further in the environment
another thing is
states on terraform enterprise terraform
enterprise has um terraform has
an enterprise offering called terraform
enterprise
and uh
the normal uh perform that people use is
a cli binary that peop that people would
be using for deploying uh infrastructure
but with three from enterprise you would
have
sso team management states uh
uh
says management in different ways and
configurations and
different ways for your integrations
and terraform to price is amazing but
still the same problem of storing the
sase files
is exist on terraform enterprise
and how it works is
you would
deploy a terraform enterprise as an
instance
it follows itself hosted model
also they provide a cloud hosted
cloud hosted model that they would be
hosting it within
an offering called terraform
and but let's focus on this on the
self-hosted one that you can host in
your
infrastructure
the states can be stored in many ways
postgresql aws rds
three mounted disks same instance desk
and
many more
if you get access to any of these
places these places would have access
to the state file and then you can copy
this state file inspected for secrets
use these secrets for
getting a step ahead
right another thing to say
of course we we talked about therefore
enterprise is deployed as an instance on
the aws account or in the cloud account
and
if you compromised and insta dfe
instance this is the best place for
attack cap resistance and there are many
reasons for that one
once it gets deployed
successfully
no one really touched it
another thing is upgrades and
maintenance really happened like
it's not a common thing that uh and a
devops engineer would go in and log and
assess it to the server to see different
things unless
something is broken then that's for our
there is an actual upgrade that would
happen this is the only time that
someone would go in and uh
and try to uh ssh to
this instance
so this is one thing to add and of
course it's uh
if because terraform enterprise
would have um
permissions to deploy to the cloud to
your cloud
environment
you would have the company would have to
provide it with proper
permissions to deploy to deploy online
so it would have probably admin access
to your aws accounts
or any
account that is being used for
if you get access to this uh
vm or this instance
then you would have high permissions in
doing whatever on the infrastructure you
can create
new users you can back their users you
can
read
buckets which is something that
is not really common but of course you
can do that
and
you can do everything
and one thing that is funny here is
let's say that
you are in engagement and the and the
security team detected that your
the database was compromised because of
the state files and all
they would ideally go and rotate the
database keys and
how would they rotate the keys they
would rotate it through terraform
that's a typical way
or one way actually but if they uh if
they updated it with terraform then you
would still have the latest keys being
synchronized
on the terraform instance which is
really cool
let me pause for water
and as i said
they're from enterprise
it has high permissions and it's not
something that you should really put on
the internet
but there are hundreds of organizations
around the world that have their
terraform enterprise exposed to the
public if there is a single zero day
here
i'm pretty sure this would be a
nightmare for
all these companies
and these are filters that you can use
for
just checking it out by yourself
okay we talked about how you can use
this in your advantage with uh when
compromising and an aws account that has
uh
that or an aws user that has access to
the sv bucket that stores the state
files
but let's say that you were not really
able to compromise the aws account
and you get access to a developer
machine and this developer have access
to there from enterprise or perform
cloud
what things can you do here
therefore
i have
exposed a full dedicated api that can be
used for automating functionalities and
tasks this can go
from any uh everywhere from updating
users to
provision
plans to
add comments and everything
and one thing that they have is an api
called state version output
and this state vision output api allows
you to download the state file that you
you uh that you would have uh
needed to compromise with an actual you
i am user or all of this
so now if if it wasn't really possible
to do this
you can just uh go to the api and ask it
for your for your state file and nicely
and it would respond
with the actual state file in plain text
which is nice
of course this will also contain
sensitive data with clear text
credentials and everything that could be
fun
okay going further
a couple months back
alex k released a blog post talking
about terraform plan rce
it's a it's a problem that reform has it
for over the years
but it wasn't really talked about
within the terraform communities
and we and this uh
this blog post has uh become a
viral
post about how this attack could be done
in uh a practical way
and they were talking about
how you can use
an ammunitious provider or an external
data source
uh that could allows you to
get
code execution on
the terraform enterprise or the
the
or the instance that is running
terraform
and
this can result in
uh the compromise of the entire
infrastructure when someone submits a pr
that is not approved
what you're going to do is
if you find let's say you said
you landed on a developer machine and
this developer machine has
i don't know github token that allows it
to so to push pr on the account
that is being in the
in the repository that hosts the
terraform code and this one is
integrated with terraform enterprise
like the notebook and like the normal
setup
what would happen here is
if the default if you as a developer
check out
the a feature branch and then push code
and
with
the malicious payload
and all the attack that uh about that
talks about the malicious provider that
you're going to talk about then you will
get
code execution on that server you would
get sell access
and this is really simple and of course
it's not really intended and this should
never happen in real life
but it has been
a fight between usability and security
here it's hard to find a way to secure
it at the same time it's hard to not
allow
something like external data sources to
be used
uh chris for alex k for having this
blog post written and shared to us
and this is an example of an actual uh
proof of concept
you if you'd like to test it out and you
are authorized to test it
go in
copy the code that i have here on my
screen
of course if the slides would be uh with
would be public on my website and my
blog
and just use this code and then push it
in as a feature brand
on
and get the repository that is uh
responsible for hosting the terraform
code
and then see the magic happens
you would get shield access on
that machine
of course this is a big problem here
but the thing is uh the other
uh platforms are also vulnerable it's
not only a problem for terraform
enterprise anything that relies on
terraform built would be vulnerable to
this
uh eric ostermann in 2018 tried to
introduce a fix and it was rejected the
fix was to use uh to
to restrict users from uh
querying guitar from plan or within
atlantis to only to only a subset that
is being specified and this would grant
security here and could be a temporary
fix but it does not really resolve the
the actual problem from terraform
ideally you should not really be able to
run code uh on terraform enterprises
uh enterprise instance and get
full access to there or get access into
the
the machine or the instance that is
running atlantis the student or even the
developer machine that runs telephone
plan
this is not the thing the right thing
all right moving ahead
we talked about the attack that uh you
can upload
you can upload an an external data
source
like this one
to gain access there but wouldn't it be
fun if we can do this in a way that is
not really detected
from the pr
let's talk about
the evil provider attack
publishing providers
with inter within hashicorps terraform
is automated this is made to encourage
publishers to push code to terraform
but the thing is
trust but verify
we cannot really just push code and have
it being uh
trusted
uh to be used by the community and by
terraform without really verifying its
content or even having
a way to detect attacks here so let's
say let's say for example
that one of these providers or one of
these 1000 providers that are being used
by people it's not only these six or
eight providers
are have been compromised and they have
been back there and we're going to talk
about how
and if this happens
and someone just say perform in it and
third from plan
what would happen is the attacker would
be compromising the company it would be
chaos
but
you know i don't want to have this being
as
a theoretical attack that does not
really happen in the real world and
someone is
discussing it in public
i would like to show an actual proof of
concept about this attack from zero to
the end
so let's start
first
let's try to set up a provider
uh i set up a new
github account i called it event reform
of course i didn't want to use my uh my
personal github account because i know
it may get suspended
we're going to talk about this one in a
bit
and
then i developed a sample provider and
this provider was pushed into event
terraform such therefore provider e
of course this is my picture
anyone can know that this
the code that i have
and then i backdoored this provider with
this code
this is a simple therefore
this is a simple
provider that would grant me
a
reverse reverse cp shield upon the
execution
to my server
and then i pushed it to
to the terraform registry and it was
accepted
it was approved
i had my category as cloud automations
because it automates my hacking here
and i was showing a bit what happened
once i had the provide they're being
approved by terraform i
i
i made a pr to push it uh to
somewhere
and it looked like this it's just is
the same way that we would be pushing
providers
in
our companies
and now
the fun part is the demo
popping uh shall enter from cloud hosted
by terraform
that's right hosted by hashicorp
this is app the telephone.io
you
right
and you would wonder from here is how is
it a detection here probably you know
in a well organized uh environment that
is running a good
edr solution or antivirus solution this
attack would be prevented or detected
but it wasn't the case i i put i pushed
the provider that i
built
into virustotal and
no provider really detected
and back to the attack that we showed
here
pushing the provider that grants me a
recipient
got
executed
correctly within hashicorp
infrastructure which is really bad
imagine
what could happen afterward i haven't
really exploited this
further i only stopped at this
part
and i tried my best to make it as
obvious
to the security team to see how the team
response would be like
and before i uh
i
i continue here i wanna say that this
attack is against the terms of use by
hashicorp
running this in an account my resulting
accounts person
so how was the security team response
within hours of my exploitation they
detected the evil provider attack and
contact me directly to chat about my
findings
and then we discussed various ideas and
thoughts to introduce a fix from for the
thing that was presented here
and hashicorp uh
for me that they are really thinking
about ways into having this uh being
mitigated in their uh for their
customers
for the time being there is no patch
there is no fix to introduce
and there is no way to prevent this
i appreciate the hashicorp's efforts in
handling and analyzing the research
one thing to add is i was planning to to
to make the responsible disclosure the
next day of my tests
and before even going to the next day
they were
the ones who were contacting me and i
really appreciate them being this
proactive
uh one thing to add here is we are also
thinking about uh starting their bug
bounty program at the the current time
they don't have a quantity program but
this is one thing that they also
are planning to have
now going to the recommendations
first
be careful there is
like you would have hard times in
maintaining a security from environment
so
being careful is my main recommendation
here
another thing is
i don't know why it sits like this
okay uh another thing is
when
using sv as a remote backend try to have
a bucket access policy to prevent other
users other than
there from instance to have access to
your instance
this way you can uh prevent that prevent
uh
like unauthorized access in certain ways
of course there is a bypass there are
bypasses for this uh
solution where if you have another user
that have access to modify the
the bucket access policy for this bucket
then they can delete the access pocket
policy and then they can access the
bucket here
but at least if you can use this as a as
a protection this would be a good way
another thing is to continually update
and review your terraform instance
it can be easily forgotten in the noise
and if there is an attacker
or an actor that
lands on your terraform instance as uh
like mentioned earlier this is a great
place for attack persistence
so it can be easily
abused badly here
and then it can be easily for defenders
to forget about it so
yeah
another thing is
be careful in permitting people to have
right access even feature like feature
brand taxes for your uh terraform
repository
because it can lead to direct code
execution uh and there is no way to pass
it
i'm
waiting for pass to happen
we uh hashicorp and i were talking about
uh way different ways to detect the
attack
meanwhile the actual uh
patch is there
but it's just ways to detect it there is
no way to prevent it at the moment
another thing is to maintain a terraform
instance in an isolated vpc that no one
really
actually said on the internet do not be
exposed to the internet this is bad
the next zero day that would happen
someone would scan the internet and
probably compromise your organization
because of this mistake
and it would be bad
and
i think this would be lastly state files
are sensitive like they contain a lot of
data from database passwords to ssl keys
uh sorry ssl certificates access keys
read them as sensitive data
and of course a good idea is to
set up ci check to detect uh rogue
terraform providers that this would aid
in the exploitation discovery whenever
someone is trying to exploit your
telephone setup
this is one thing to add as a defensive
measure
one thing that
has a corp
security team
asked me to add is the terraform cloud
security model
this is a document that talks about
different ways that
that you can use for threat modeling
your terraform cloud security while at
the same time it it can apply for your
term from enterprise setup
and now we are reaching to the end of
the talk
final thoughts terraform is amazing i
love your phone
it brings several security features when
implementing
iec with terraform
but still with great power comes great
responsibility protect your terraform
environment
stay safe and set up persistent monitor
like full hunt to discover shadow ide
missed configurations and mistakenly
exposed services of course it would not
really work out for all of the other
home attacks we talked here
but yeah just one thing to add here
and lastly
the questions