uh thank you so much everyone uh for
joining me in this awesome session at
cloud village def con uh in a virtual
edition
so today i'll be focusing on talking on
kubernetes code it's a project meant to
be trying to teach community security uh
by using vulnerable by design labs
so today i will be like a big disclaimer
trying to be not showing a slides are
mostly focusing on the demo and how the
project can be set up and used as a
normal user and learn from it and go
back and try out your kubernetes skills
right
so without you let me
go head over to the project uh this is
the project uh you can find it in the
github.com
kubernetes code
uh as you see it's uh pretty much try to
design for intentionally use of like
learning towards kubernetes security uh
it doesn't matter if you are someone
beginner or trying to learn offensive
side or defensive side or even some kind
of tool which you are trying to learn
and see how it is effective right so we
kind of look at like some of the
scenarios and see how it looks like
right uh
yeah
before we diving into the good project
let me just zoom in a bit to see making
sure everything is fine so we kind of
have a ton of scenarios in this uh code
uh which is like uh basically trying to
take this naming from similar to web
code which is a standard ovals project
for years
and we kind of showcase a different
variety of scenarios on different areas
both offensive security and defensive
side of the kubernetes
so
let's see how it goes
bit about me uh myself okay i kind of
working towards kubernetes and
containers for quite some time uh nearly
past four years
apart from that i have been trying to
present my research and share with the
community around the world like black
eyed defcon use nix or really a bunch of
other conferences
uh i'm fortunate to find some kind of
vulnerabilities quite some time back uh
and contribute to the community
especially open source ecosystem and
also
part of a bunch of communities like all
the devops and all
and also written and reviewed bunch of
books and
yeah that's pretty much about me
regarding the project uh if you are
someone who are getting started with
kubernetes uh i would highly recommend
learning the basics before trying out
this project so it adds a ton of value
than learning directly the security
parts right
i feel that if you understand the
technology the most of the concept which
you learn from security uh can be
applied here right sometimes whatever
you say firewalls uh maybe in cloud
world they call it as like security
groups or nacl right something like that
in kubernetes maybe they call network
security policies right so this
terminology may differ a bit i would
highly recommend learn about the
technology and understand these concepts
so that you can make most out of it
than just having it
if you are someone new to kubernetes
it's an open source container
orchestration system mainly designed for
deploying your applications into the
cloud so that you can scale and manage
efficiently uh it was originally created
by a project uh from google called park
and now they donated this as a
kubernetes to cncf which is the cloud
native computing foundation
and i definitely highly recommend this
video i think i do bunch of trainings
and workshops so everyone i highly
recommend to go through this video it is
kind of children's way how you can
explain kubernetes to someone which is
just getting started right
and um yeah maybe i can just give a very
quick fast track one minute version of
overview which is definitely hard though
uh in kubernetes there is a concept like
master and note like where it's like
client server architecture type so the
master holds bunch of components which
means master node has something called
api server which is like a brain of
kubernetes it doesn't matter if you are
sending a request from external world or
internal components which are like
something like their own processes which
is
talking to the kubernetes right so it
has to go through the api server so
think of this as a quite critical
component in the kubernetes like a brain
right
once it's sent uh so the xcd is like a
database it's kind of key value store
where it stores all this data
and uh okay let's say as a developer our
devops team i sent like i wanted to
scale
like 10
10 engine exports in a kubernetes and
run it right so to answer this like
where i have these resources and
everything so scheduler is the job to
pick up like how much resources i have
in the notes where i should schedule and
all these things right
so controller manager is like kind of
the way how it's like uh
talk to this 8 cd and making sure these
are running always the declared or
desired state right so this is a very
high level i think there is ton of
things once you get into the details
similarly let's say once scheduler says
i wanted to apply this pod into some
node right so cubelet is another uh
process which sits on the node making
sure talking to the runtime it doesn't
matter it is docker run c cut or some
other container runtime so it is talking
to the container runtime which is in the
node uh to schedule the part which is
told by like api server right
and it kind of works across like sharing
this data here from to the api server as
a way right and another thing is like a
cube proxy which is like traditional ip
tables rules right so which kind of
connects these proc uh services across
these nodes to serve to the users and uh
applications and services right
and all your parts are deployed here
ford is like a minimum leased unit it
can have one or more containers right so
this is like a super quick one-on-one
view but i would highly recommend refer
to the documentation they have ton of
really detailed things and also if you
are someone trying to get started i
would highly recommend this kubernetes
by examples if you stuck something maybe
i wanted to just know what is exactly is
name spaces or volumes right maybe you
can quickly get your hands on on top of
it so these are quite learning material
if you are someone getting started okay
so let's say that you already be
comfortable with communities like now i
wanted to go and understand what kind of
security things i can uh identify issues
or like learn and secure the cluster
right
so i feel because i have been doing
trains for quite some time it's tough to
people to set up these environments so
what i have done is i try to create an
online playground which is just by
clicking in a url you can start a
browser environment where you can play
with this project and try to learn as a
concept rather than focusing on your
setup and understanding the the
complexity of the things right so you
can go ahead and start here like a click
on the launch which means that you will
start a kubernetes cluster for you
within the browser and uh once you run
these two commands these two comma kind
of sets up a project for you
right so whatever the guide you see it
is detailed in a way like everything is
documented what all things you can go
back and try definitely i don't have
enough time to cover all of these
scenarios uh but i try to document each
and every step by step so that you can
go back and apply this learning
knowledge right
so
while it is trying to set up like you
can see that we just set it up and we
hit the enter it will go ahead and set
up the some of the scenarios which we
are going to play around today or after
later as well and we can able to get the
access after this right
so for the sake of the time i try to set
it up already uh i can just type cube
cuddle get notes
uh you can see that uh there are a bunch
of nodes
sorry again
so you can see three nodes running uh
within the kubernetes cluster and uh
which means the three node parts of the
the
nodes are there in the kubernetes side
and you can also see bunch of pods which
means number of containers one or more
running right in some cases too
some cases one right
so
this is like a kind of trying to
replicate a real world cluster with
intentionally vulnerabilities to
showcase how kubernetes security we can
learn right
uh so
if you are someone like playing this uh
using katakura or definitely you can set
it up your own so you can also do like
uh making sure the parts are running
before you hitting the last script which
is nothing but trying to expose the all
vulnerable apps or services within your
local system so that it may not expose
to the internet someone like a real
attacker may not just exploit your
cluster right and also i would highly
recommend maybe i didn't go through the
disclaimer a bit early uh maybe i think
i might have put please do not try to
put or run your kubernetes code project
on your production workloads or in your
company workloads which basically has
intentional vulnerabilities so we it
doesn't come with any warranties or
guarantees so making sure try to run in
your safe environment and try to run in
a play environment where you can just
try it out things right
cool
once you set it up basically the project
uh then you kind of uh
written with a url basically where you
can access the kubernetes code right
like which is kind of like once you run
the script you end up seeing this
so this is how it looks like this is
kind of a home for the kubernetes code
where you can find out all the scenarios
which is available and how you can get
started with them to learn or practice
right so if you see here uh definitely
this guide is quite handy uh like this
is what we are trying to go through
today as well uh because i can't
remember everything for sure and there
are ton of other things also so you can
if you stuck somewhere when you are
trying certain scenario you can come
here and try to refer certain example or
a certain scenario uh more details right
so yeah let me just see uh making sure
i'll try to showcase you how you can
access this as well so once you hit the
bash script of access kubernetes code it
all exposed so now you can go ahead and
use this project and uh it is making
sure it's up and running right
so if you are someone trying to browse
that in other scenarios like let's say
ssrf in gate as well right rather than
browsing if you are using the katakura i
would highly recommend use this custom
port and mention this port number and
just use it right so that simple as uh
to just set up the project and get
started you are learning of community
security right so let me go through some
of the scenarios and showcase both
attackers our offensive side as well as
defenders and how we can learn security
of the system
and also see last part how vendors like
uh enterprise security or open source
projects can leverage this as a tool to
see how much gaps are improve their
projects as well right our products
to quickly showcase a simple attack
factor or scenario i think this is quite
commonly you see in day to day ca cd
system right docker in docker if you are
someone using any most of the bl system
like github ci or
pipeline bitbucket pipelines or github
actions and all you might end up seeing
something called a tint which means
docker and docker so if someone building
a docker command or like a docker
container within your build system they
might want some way to access your
docker socket or docker api so that they
can build that image like definitely
there are some alternatives keep coming
in the world so most people still till
to date i see in especially penn testing
engagements or in real world people try
to use this method to build that thing
so what i have trying to showcase is how
we can exploit this or gain access to
the underlying host system by escaping
from the container to the host system
right so to get started uh with this
scenario you can go ahead and click this
url so which has an intentionally
vulnerable uh
lab or like application so as for this
demo i don't want to focus on
application security issues so i'll try
to go ahead and showcase you that this
application is vulnerable to remote code
execution which means that it has like a
rc right command injection vulnerability
so from here which means that you have
to identify what is this it is running
in and how we can get escape out of this
container by looking at different paths
and aspects right
so now one thing as i look like maybe i
can look at like my process uh okay
where it is running exactly so you can
go ahead and run cad prox lfc group
it kind of tells that okay it is running
and kubernetes spots so it try to
showcase you the runtime sometimes and
also there is tools like am i contained
by jessica frazil which also kind of
showcase and give it a container uh
information so now once i identify
definitely you need to understand is
there anything which is running in the
file system or is there any something
which i can think of like something
interesting right
so as i already know this is
intentionally built and trying to put
some vulnerabilities i can see that
there is a darker socket mounted here
right it is in the custom docker docker
dot sock right so which means that
there is a docker socket which is passed
from the whole system to the container
right maybe i can just showcase in a
nicer way
maybe
so
so this is your application running uh
which is
in a pod or container i would say right
so which is running in a node which is
in the kubernetes node right so now the
docker is running here like which is
where run
a
docker dot song right
so now what we are trying to see is uh
this docker socket also passed to the
container in a different part which is
like custom
upper or top or soft so now we are
trying to leverage this docker socket
and talk to the host system because this
is mapped with the whole system and see
if we can escape out of this spot or
container and gain access to node right
like that is what we are trying to
exploit and uh see if you can gain
access right as i said you can
definitely go through this documentation
and you can see this how you can
leverage uh maybe we can use even curl
basically to talk to the thing or you
can even use the docker right so there
is no docker so
the one way in real word attackers can
leverage is definitely you can download
the docker because it's just a binary
you can double gate or like
and exploit this system or in another
way maybe you can even interact with the
docker by using call with the unique
socket by passing right
so for this uh this thing you can just
download this and uh make it available
into local system as binary now you can
see i tried to already put it here and
now
as i attacked if you wanted to interact
with the docker socket you need to
specify the iphone h flag so that you
can talk to this specific docker host
socket right so now you can see here uh
i can just say that and i can also query
images right
so
as you can see here uh you already got
bunch of docker images which are in the
host system right which is basically now
we are trying to query the node docker
container images uh without here
underlying in the node system right so
enough as an attacker if you wanted to
basically go exploit this and get a
shell back or something maybe you can
just run a container uh
volume on everything uh which is in the
whole system with either read earlier
or whatever the way you want or you can
even pass a flags like privileged equal
to true to get a privileged containers
maybe adding extra capabilities or
something right or you can just run at
your own container like i can create my
own container saying that it gets a
reverse shell back to me uh in this
container uh whatever the thing that's
it once you run basically get a shell
back from there you can traverse and
learn more attack surface and exploit
more
but uh for the sake of the demo i just
will try to stop here but uh definitely
in real world there is a lot of little
moment and attack attack exploration
from here right so this is a typical
example of how you can exploit docker in
docker sockets uh in a kubernetes
clusters or in general right
and there are ton of other
vulnerabilities definitely i i try to
not able to cover
but you can go back and definitely
explore like there is another
vulnerability where if they have extra
privileges right uh maybe you can just
see if they have so many privileges by
running capacity that's testing now you
can see bunch of other
capabilities it has already given right
maybe you can see that there is an extra
file system mounted or like a found in
the whole system right there is a host
system so you can definitely leverage
the traditional uh way of like a ch root
and slash system
uh
and whatever the command you want to run
right so now if you see here like we are
trying to basically uh leverage this and
try to run commands or systems uh within
the host system
so now whatever the commands you run
which are going to run within the whole
system by escaping out of the container
looks like there is no containers
running
another reason uh which is most probably
due to the reason
they are might be using container
because the underlying whole system is
changed but you can see docker images
works pretty fine
right so this is maybe another way to
escape and from there maybe you can use
and leverage this cube cuddle config
like which is available in most of the
container nodes right so which is in uh
utc vanities
and uh you can see cube like uh
cube config
so this is a cube light config which
means that it used to talk to the master
node like as you see uh when you
describe the architecture right so as
attacker you can also leverage this
config like uh cube catal
dash
cube dash
and uh pass the flag of uh whatever the
config which you found
uh
like
dot tube config and get parts right
so now we can use this to query if any
parts available within the cluster
and see if we can be able to use this
privileges or something to get the data
right
looks like yeah cool now we can see so
what we just did here is now we kind of
escaped from the whole system by
leveraging the whole system file path
mount and we use this vulnerability to
run here and also we leverage the cube
light tube config
to talk to the api server and from there
you can also leverage see if what kind
of privileges it has and from there you
can now can go on a bit more uh
authorization authentication exploits
right
so i can also query using what can i can
i delete nodes sorry
can i delete
notes
so this is kind of imperative command to
query to the api server asking whatever
the config i found cubelet using this
can i do something uh whatever operation
so you can see this
this configuration are the permission
which we have we can't be able to delete
nodes right maybe we can create fonts
right so we can use this extra
permissions to maybe create another part
within the another node maybe you can
hop into another node from there you
kind of do lateral movement on top of
each node and node and maybe gain more
and more data or containers access right
so this is how a typical maybe offensive
side you can definitely look and as i
said a bunch of other examples and
scenarios you can find like how we can
consume resources and memories when they
don't define resource limits and uh
quotas or
limit ranges
right
and if you look at like uh uh
example of defense right maybe i'm
trying to showcase a simple use case of
open source utility called like falco
which kind of help us to do monitoring
of runtime security right
so i wanted to deploy this simple uh
falco
deployment into our kubernetes cluster
and i will go ahead and update this repo
in my health
and i will install this health card into
the kubernetes cluster which is
available right
so now you can see this it got deployed
now you can see the parts running of the
falco in the light
which is basically creating and the
reason you may ask why these are running
three right uh if you look at like nodes
which we have three different nodes so
falco currently runs with a demand set
which means that if you have n number of
nodes so each node will have one part
running so that we wanted to collect all
the data and matrix and so that we can
monitor the each and every node level uh
the runtime and all these data right
containers so that's why it is kind of
running with a demand set so if you go
ahead and see the parts uh you can see
that they're running up and running so
now what we can do is uh falco has like
a bunch of uh rule set to detect certain
kind of security issues within the
kubernetes or in general linux right so
there are something like if someone
tried to browse some sensitive files
like cat edition shadow or something it
might trigger an alert right so what i'm
going to do is i'll go ahead and query
these logs uh from this falco right from
the all these spots and also try to
simulate some kind of attack or
vulnerability something which falco
triggers and see how we can detect these
as a defense point of view right
so let me go ahead and query these logs
uh
now you can see the logs coming so what
i'll do is i'll go ahead and create a
container
which kind of exists into your bash and
you can see i'm trying to use a hacker
container maybe i'll come a bit later
what exactly it is let me go ahead and
run this container
if you see me i just started a container
and exiting into the bash of that
container right now
uh okay i'll just quickly showcase this
example i think then i think i'll try to
cover the last section
yeah so now uh as you see uh whenever
some kind of new triggers happens within
the kubernetes cluster in terms of
security point of view you can see this
logs coming up right so let me go ahead
and do the simple example right edc
shadow whenever i read the file
sensitive because as you see falco has a
rule set for that it automatically
detects that which container it got
triggered and which file it has read
which container which node so it kind of
provides a detection or defensive
mechanism to understand identify like a
security point of view what kind of
things happening around the system and
all so we try to focus on different
scenarios how we can teach this defense
side as well uh in terms of kubernetes
security and how we can leverage these
things
and like network security policies and
all
as i am trying to come to the end uh as
i said uh please give it a try there are
ton of other uh scenarios which i
couldn't cover and also i tried to
showcase some uh vulnerability reports
from some of the tools which are open
source as well as the commercial uh
maybe not commercial uh to see how they
can detect these things as part of your
ci cd file system like privilege
escalation is allowed so maybe you want
to detect them so this is also quite uh
helpful when you are trying to learn and
how you can apply in your organization
at all
so with that i think uh thank you so
much once again for giving the
opportunity and uh i would highly
recommend uh give it a try and see how
it looks like and share the feedback
and also if you like the project and if
you use please share us some love by
giving your handle our name so that we
kind of may be helpful for us to take
feedback and improve this
as well