hi everyone
thank you very much for tuning in today
today's talk is about
attacking terraform environments by the
end of this talk
you learned ways to attack and secure
your terraform environments
this talk would cover a lot of things
that probably
not done before one thing to say before
we start
therefore by default is not as secure as
you would think
you have to put a lot of work to secure
it and
what i'm aiming for is by this at by the
end of this talk is to learn about
all the things that you can know
uh so that you can secure your terraform
environments
and let's start
a little bit about me my name is mazin
ahmed i'm an application security and
offensive security engineer
i founded full hand it's a security
startup that solves
the attack surface visibility problem
for companies and
organizations and also i'm an occasional
bug bounty hunter
i have been acknowledged by facebook
twitter
linkedin zoom oracle and many other
companies
and of course i'm in love with cloud
security
anything related to cloud security and
of course uh
different things related to security are
my passion
and that's why i put the effort into
having this research
made the agenda for today
is first we we're gonna go over things
uh a background on terraform what
styrofoam how does it work
and why do we need to learn about all of
this
why is it important attack victors and
scenarios
attack victors and scenarios and of
course the
most important part is the demo which
would be really
cool the recommendations at the end
and then would have room for questions
so first of all what's infrastructure as
good
back in the years when cloud was
starting
was becoming a thing if someone would
like to deploy something they would go
to the console
take their time in deploying their
resources
from the console and if they would like
to
do this again and again and maintain
that they would have to log
in every single time to do that which is
a bit of a painful
thing and does not really scale that's
why
infrastructure as code became a popular
thing
in the devops and lipstick of
communities
why because you would be able to write
all of your resources and the cloud as
code and then you would be able to
deploy it many times
at the same time you would allow room
for
peers if you so that the people in your
team and different teams will be able to
review it
it's amazing so yep it allows
that parts and of course uh one thing
that you can say that
once you have this being set up you
would be able to run it through
ci security scanning for compliance
and you can automate many things without
really
having to have someone to review
everything
um uh being deployed
and of course when you deploy this and
using get
you would be able to have an archive of
everything being deployed
it's just an amazing technology that
everyone loves
hopefully and uh what's their phone
jira form is an open source
infrastructure as code software made by
hashicorp around 10 years ago
and users define their code in a
language called the
hcl or hashicorp language
and then deployments and and the entire
infrastructure management happens
through terraform whenever you write
code
you would pass it to a telephone build a
binary and then it will
read the term from state a state of
the current infrastructure and then it
will decide what changes should be done
why i'm talking about true form today
instead of all of
apollo me or different uh or cloud
formation or different
uh cloud orchestrator because
turf is the most popular iec
orchestrated in the planet
this year a couple of months back they
have released
terraform 1.0 the
it has been a big thing for everyone
that is
working on with terraform and of course
it passes 100 million downloads
crazy numbers
another thing that is cool about
terraform once you have it
is you can write the same infrastructure
code
and have it integrated with different
providers so
you can write code that allows you to
deploy to
aws or azure or gcp or anything else
and it's not only these
six technologies and six providers that
are there
they support more than a thousand
provider
as of today how how you would be able to
do this
is as a practitioner or a user you would
write your code
then once it's ready it will go through
terraform plan
and telephone plan is like the way that
the form would be
planning your deployment and show you
what would be changing and then it would
go through
the apply and once you click on uh run
apply
it would be deploying directly
to your cloud provider the thing is
when you are running apply and plan on
apply
this means that you are checking that
that drifts between the current code and
what is there on the
provider and if you are applying uh
changes
this means that you have right access on
different things on your cloud provider
and who would have this it would be
terrifying
so setting up their phone with grants at
high permissions that allows it to do
everything that you
need to have it to do
while it sounds curry it's the default
thing
we have to just set it up in a way that
could be secure
and not rely on the default things
and try to think better about it
as i said with great power comes great
responsibility a single mistake here
can lead to the entire
compromise of the the cloud provider or
the cloud account
so imagine your entire aws account
being held for ransom someone would uh
like if someone compromised that they
would go into your club
provider they would be backing up all of
your sv buckets
they would uh incr or may uh delete
everything
they would have it somewhere else and
then
they would ask you for ransom imagine
this ransom isn't the thing that you
would see
only on your uh internal network or
or in your laptops is something that can
be done also in the cloud
it's not it's still not a thing today
but i can imagine that this would be
happening
in the upcoming years
and as i said about the terraform
states the way that perform works is
it would be storing all of the changes
and all of the additions that it made
in your cloud provider in a file
that acts as a database called terraform
state
every time you do a change it would
check it against
that file and then would update it
whenever there is anything
there are many ways to have your
terraform state being deployed
one thing is you can have it locally
second way that
is i think the most common way by
everyone that i know is using
a remote backend of sv
so that you would specify a bucket you
would specify a key
and then which region would you like to
have this bucket at
on and then terraform would be
automatically
uh creating this bucket for uh would be
automatically uh
pushing the the changes and the
state file there and whenever it updates
it would
synchronize with that key
so attack number one or thing number one
if you go if you have uh get
uh get object permission on the s3
bucket for
solving the state file then you would
have access to
[Music]
secrets access keys and database
credentials
so everything that is being deployed
from
uh for the organization through
terraform
uh have a copy of it in the sit
in the state file so there's something
like
an rds database would have their
credentials stored somewhere
and that place would have a copy in the
state file
same thing if you create the an im key
access key then you would have it there
or
any other secrets that you would define
or
the provider would be defining for your
phone so whenever you get access
to uh to an organization and a
cloud account try to search for
if they have a sv bucket that
looks like it's use uh it's storing
secrets for your phone
it would have the entire instructions
for you
to go further in the environment
another thing is states on
terraform enterprise terraform
enterprise has
terraform has an enterprise offering
called terraform enterprise
and the normal uh
perform that people use is a cli binary
that peop
that people would be using for deploying
uh
infrastructure but with terraform
enterprise you would have
sso team management states
says management in different ways and
configurations and
different ways for your integrations and
terraform to price is amazing but still
the same problem of storing the sales
files
is exist on terraform enterprise
and how it works is you would
deploy a terraform enterprise as an
instance
it follows itself hosted model also they
provide a cloud hosted
cloud hosted model that they would be
hosting it within
an offering called terraform cloud and
but
let's focus on this on the self-hosted
one that you can host in your
infrastructure the states can be stored
in many ways
postgresql aws rds
a3 mounted disk same instance desk
and many more
if you get access to any of these
places these places would have access
to the state file and then you can copy
this state file
inspected for secrets use these secrets
for
getting a step ahead
right another thing to say of course we
we talked about therefore enterprise is
deployed as an instance
on the aws account or in the cloud
account
and if you compromised and inst at efe
instance
this is the best place for attack
persistence and there are many reasons
for that one and once it gets deployed
successfully no one really touched it
another thing is upgrades and
maintenance really happened like
it's not a common thing that uh and a
devops engineer would go in
and log and assess it to the server to
see
different things unless something is
broken
then that's for our there is an actual
upgrade that would happen
this is the only time that someone would
go in and uh
and try to uh ssh to this
instance so this is one thing to add
and of course it's uh if
because terraform enterprise would have
um
permissions to deploy to the cloud to
your cloud uh
environment you would have the company
would have to
provide it with proper permissions
to deploy to deploy online so it would
have
probably admin access to your aws
accounts
or any account that is being used for
if you get access to this uh
vm or this instance then you would have
high permissions in doing whatever on
the infrastructure you can create
new users you can back their users you
can
read buckets which is
something that is not really common but
of course you can do that
and you can do everything
and one thing that is funny here is
let's say that
you are in engagement and the and the
security team detected that
your the database was compromised
because of
the state files and all they would
ideally go and rotate the
database keys and how would they
rotate the keys they would rotate it
through terraform
that's a typical way uh or one way
actually
but if they uh if they updated it with
terraform
then you would still have the latest
keys being
synchronized uh on the terraform
instance which is
really cool
let me pause for water
and as i said they're from enterprise
it has high permissions and it's not
something that you should really put on
the internet
but there are hundreds of organizations
around the world that have
their terraform enterprise exposed to
the public
if there is a single zero day here i'm
pretty sure this would be a nightmare
for all of these companies
and these are filters that you can use
for
just checking it out by yourself
okay we talked about how you can use
this in your advantage with
uh when compromising and an aws account
that has
uh that or an aws user that has
access to the sv bucket that stores
the state files but let's say
that you were not really able to
compromise the aws
account and you get access to a
developer machine and this developer
have
access to there from enterprise or
terraform cloud
what things can you do here
therefore i have
exposed a full dedicated api that can be
used for automating functionalities and
tasks
this can go from any uh everywhere from
updating users
to provision plans to
add comments and everything
one thing that they have is an api
called
state version output and this state
vision output api allows you to download
the state file
that you you that you would have
needed to compromise an actual you i am
user or
all of this so now if
it wasn't really possible to do this you
can just
go to the api and ask it for your
for your state file and nicely and it
would respond
with the actual state file in plain text
which is nice
of course this will also contain
sensitive data
with clear text credentials and
everything that could be
fun
going further a couple months back
alex k released a blog post talking
about terraform plan rce
it's a it's a problem that reform has it
for over the years
but it wasn't really talked about
within the terraform communities
and we and this uh this blog post
has uh become a viral
post about how this attack could be done
in
a practical way and they were talking
about
how you can use an a malicious provider
or an external data source uh
that could allows you to get
code execution on the terraform
enterprise or the
the or the instance that is running
terraform
and this can result in
uh the compromise of the entire
infrastructure when someone
submits a pr that is not approved
what you're going to do is if you find
let's say you said you landed on a
developer machine
and this developer machine has i don't
know github token
that allows it to sell to push pr
on the account that is being
in the repository that hosts
the terraform code and this one is
integrated with terraform enterprise
lincoln
and like the normal setup what would
happen here
is if the default if you as a developer
check out the a feature branch and then
push code and with the malicious payload
and all the attack that about that
talks about the malicious provider that
you're going to
talk about then you will get code
execution
on that server you would get shell
access
and this is really simple and of course
it's not really intended
and this should never happen in real
life
but it has been a fight
between usability and security here it's
hard
to find a way to secure it at the same
time it's hard to not
allow uh something like external data
sources to be used
uh chris for alex k for having this book
blog post written and shared to us
and this is an example of an actual uh
proof of concept
you if you'd like to test it out and you
are authorized to test it
go in copy the code that i have here on
my screen
of course the slides would be with would
be public on my website
and my blog and just
use this code and then push it in as a
feature branch
on and get the repository
that is uh responsible for hosting the
terraform code
and then see the magic happens you would
get
access on that machine
of course this is a big problem here but
the thing is
uh the other uh platforms
are also vulnerable it's not only a
problem for terraform enterprise
anything that realized on terraform
built would be vulnerable to this
uh eric ostermann in 2018 tried to
introduce a fix
and it was rejected the fix was to use
uh to
to restrict users from uh
querying guitar from them or within
atlantis
to only to only a subset that is being
specified
and this would grant security here and
could be a temporary fix
but it does not really resolve the
actual problem from terraform
ideally you should not really be able to
run code on terraform enterprises
uh enterprise instance and get full
access to there
or get access into the the machine or
the instance that is running atlantis
the student or even the developer
machine
that runs terraform plan this is not
the thing the right thing
all right moving ahead
we talked about the attack that uh you
can
upload you can upload
an external data source like this one
to jain access there but wouldn't it be
fun
if we can do this in a way that
is not really detected from the
pr let's talk about
the evil provider attack
publishing providers within turf within
hashicorps
terraform is automated this is made to
encourage
publishers to push code to terraform
uh but the thing is trust but verify
we cannot really just push code and have
it
being uh trusted
uh to be used by the community and by
terraform
without really verifying its content or
even having
a way to detect attacks here so let's
say let's say for example
that one of these providers or one of
these 1000 providers
that are being used by people it's not
only these six or
eight providers are have been
compromised and
they have been back there and we're
going to talk about how
and if this happens and someone
just say perform in it and third from
plan what would happen is the attacker
would be compromising the company it
would be
chaos
but you know i don't want to have this
being
as a theoretical attack that does not
really happen in real world
and someone is discussing it in public
i would like to show an actual proof of
concept about this attack
from zero to the end so let's start
first let's try to set up a provider
i set up a new github account i called
it event reform
of course i didn't want to use my uh my
personal
github account because i know it may get
suspended
we're going to talk about this one in a
bit
and then i developed
a sample provider and this provider
was pushed into event terraform such
different provider e
of course this is my picture anyone can
know
that this the code that i have
and then i backdoored this provider with
this code this is a simple therefore
this is a simple
provider that would grant me uh
reverse reverses official upon the
execution
to my server and then i pushed it to
the terraform registry and it was
accepted
it was approved i had my category as
cloud automations because it automates
my
hacking here and i was showing a bit
what happened
once i had the provide they're being
approved by terraform
i i i made a pr to push it
to somewhere and
it looked like this it's just this is
the same way that we would be pushing
providers
in uh our companies
and now the fun part is the demo
popping uh she'll enter from cloud
hosted by terraform that's right hosted
by
hashicorp
this is app to telephone that i owe
right and you would wonder from here
is how is it detection here probably in
a
a for in a well organized uh environment
that is running
a good edr solution or antivirus
solution
this attack would be prevented or
detected
but it wasn't the case i i put
i pushed the provider that i uh built
into virustotal and
no provider really detected that
and back to the attack that we showed
here
pushing the provider that grants me a
recipient
got executed uh correctly within
hashicorp
infrastructure which is really bad uh
imagine
what could happen afterward i haven't
really exploited this
further i only stopped at this part
and i tried my best to make it as
obvious
to the security team to see how the team
response would be like
and before i uh i
i continue here i wanna say that
this attack is against the terms of use
by hashicorp
running this in an account my resulting
accounts expression
so how was the security team response
within hours of my exploitation
they detected the evil provider attack
and
contact me directly to chat about my
findings
and then we discussed various ideas and
thoughts to introduce a fix
from for the thing that was presented
here
and uh hashicorp uh
informed me that they are really
thinking about ways into having this
uh being mitigated in their uh for their
customers
for the time being there is no patch
there is no fix
to introduce and there is no way to
prevent this
i appreciate the hashicorp's efforts in
handling and analyzing the
research one thing to add is i was
planning to
to to make the responsible disclosure
the next day
of my tests and before even
going to the next day they were the ones
who were contacting me and i really
appreciate
them being this proactive
uh one thing to add here is they are
also thinking about
starting their bug bounty program at
the current time they don't have a
quantity program but this is one thing
that they also
are planning to have
now going to the recommendations first
be careful there is
like you would have hard times in
maintaining a security from environment
so being careful is my main
recommendation here
another thing is
i don't know why is it like this
okay uh another thing is when
when using sv as a remote back-end try
to have
a bucket access policy to prevent other
users
other than their from instance to have
access to your
instance this way you can prevent that
prevent like unauthorized access
in certain ways of course there is a
bypass
there are bypasses for this uh solution
where if you have another user that have
access to modify the
the bucket access policy for this bucket
then they can delete the access pocket
policy and then they can access the
bucket here
but at least if you can use
this as a as a protection this would be
a good way
another thing is to continually update
and review your terraform instance
it can be easily forgotten in the noise
and if
there is an attacker or an actor that
lands on your terraform instance as uh
like mentioned earlier
this is a great place for attack
persistence so
it can be easily abused badly here
and then it can be easily for defenders
to
forget about it so yeah
another thing is be careful in
permitting people to have right access
even feature
like feature brand taxes for your uh
terraform repository
because it can lead to direct code
execution uh
and there is no way to pass it i'm
waiting for the past to happen we uh
hashicorp
and i were talking about the way
different ways to detect the attack
meanwhile the actual uh patch is there
but it's just ways to detect it there is
no way to prevent it at the moment
another thing is to maintain it from
instance
in an isolated vpc that no one really
actually said on the internet do not be
exposed to the internet this is bad
the next day zero day that would happen
someone would scan the internet and
probably compromise your organization
because of this mistake
and it would be bad
and i think this would be lastly state
files are sensitive
like they contain a lot of data from
database passwords
to ssl keys uh sorry ssl certificates
access keys read them as sensitive data
and of course a good idea is to set up
ci check to detect the rogue terraform
providers
that this would aid in the exploitation
discovery whenever
someone is trying to exploit your uh
telephone setup
this is one thing to add as a defensive
measure
one thing that has a corp security team
uh
asked me to add is the terraform cloud
security model
this is a document that talks about
different ways
that that you can use for threat
modeling your
terraform class security while at the
same time it
it can apply for your term from
enterprise setup
and now we are reaching to the end of
the talk
final thoughts terraform is amazing i
love your phone
it brings several security features when
implementing
iec with terraform but still with great
power
comes great responsibility protect your
terraform environment
stay safe and set up persistent monitor
like
full hunt to discover shadow ide missed
configurations
and mistakenly exposed services of
course it would not really work out
for all of the other home attacks we
talked here
uh but yeah just one thing to add here
and lastly the questions
thank you very much