so yeah this um this workshop is going
to be about kubernetes security 101
uh some best practice to secure cluster
okay
so yeah just a little bit about myself
as jeff mentioned i work at trend micro
as an information security specialist
i'm part of the cloud and container
security research team
i'm also a member of the cncf
security tag team now to change name it
was called seek security now security
tag team
which is like the technical advisory
group for the cloud native computing
foundation right it's a volunteer work
that we do providing guidance and and
creating uh documentation
regards to uh cloud native tools supply
chain attacks we've published last year
our
cloud native security white paper and
recently our
supply chain security white paper so um
those are all available and free for
everyone to check it out
and
i just want to say before we start that
um i don't consider myself a kubernetes
security expert
as you're going to see in this workshop
kubernetes is a very complex
uh
technology i would say and
i've been studying for over one year one
year and a half now and still sometimes
it can be uh some things can be tricky
and and so yeah uh feel free to ask
questions if you have them the idea of
this workshop here is to be
a very hands-on so that you can do it
yourself as well and follow along so i'm
gonna do things very slow and and like
step by step right um
yeah and i also have a personal blog on
uh the link there katennasac.com
where i publish some
articles
at least once a month there is also a
list of all my previous talks uh before
the slides or videos if they were
recorded since uh since 2011 when i
started speaking at conferences about
application security and all that stuff
and there is also all my contact
information on my social media
and twitter github and linkedin it's all
magna logan so you can easily find me if
you want and feel free to add me and we
can chat more after this workshop
so
the agenda for today is
what is kubernetes right so i'm gonna
assume that you either heard about
kubernetes or at least uh
have
seen someone or an organization that
uses kubernetes but in this workshop i'm
going to assume that you've never used
kubernetes right so
um the idea is to start from scratch so
what we're going to do
um the way that this workshop works is
we're going to set up an environment on
aws that's why the there there was a
prerequisite of having an aws account
and we're going to use cloud9 which is
like a virtual uh developer environment
for aws so we're going to deploy a
cloud9 instance and use that to uh
deploy our cluster right
uh so
we're gonna understand what is
kubernetes the kubernetes architecture
right i'm gonna talk about all the small
components um
the the two main components the the
control plane and the worker nodes and
then the each little component that's
inside those those major ones
um
as joff said we're going to do some crap
modeling right i don't know if you heard
but
this year in april this year the miter
released the attack framework the attack
matrix for containers right and
we were part of this this work with
miter right so we partnered with mitre
when they uh when they did this uh when
they called the community for help and
and we provided some data regarding our
uh our honeypots we have some docker and
kubernetes honeypots that we use to
analyze uh attack data's on those
environments so we're going to talk
about that as well
um then later we're going to talk about
after we set up our environment on on
aws we set up a cluster we're going to
deploy
a application uh a container contourize
application there a vulnerable
application and so we're going to attack
it right so we're going to understand
what's going on uh um how can an
attacker compromise the cluster one of
the many ways that uh that's possible
and of course this is a misconfigured
cluster right so that you understand the
fundamentals of why it's important to
keep your kubernetes cluster uh well
configured and secured
and if we have time then we're going to
talk about defending kubernetes right so
what are the main uh issues on
that you can like what are the best
practices right
either regarding the api server and the
whole uh process of deploying in a
container uh generating a container
right from scratch and deploying it into
a kubernetes cluster and and there is
many things that we can talk about here
so yeah i hope you enjoy and and feel
free to ask questions i know that people
are monitoring those questions and and
they can send them over to me so if
you're struggling if you're not if
you're you're trying to follow along and
something's not working for you
don't worry try feel free to ask
questions and i can we can make this uh
very interactive
okay
so
before we start i just want to say that
i uh
since i started working with kubernetes
uh last year
i created this this kind of github
project the awesome kubernetes security
list
and um this project has
um has like a bunch of links and in
information that i use myself to learn
kubernetes and understand and has a lot
of stuff about kubernetes security so
let me share that here
with you so that you can take a look as
well so yeah first this uh um
i started with that as a like a kind of
my own thing when i was studying saving
links on a
text file uh for myself but i said no
maybe maybe some people other people can
benefit from that as well and
and that's why i did this this github
repo which has almost
almost a thousand stars already right
and i created that in october last year
so it it hasn't been even over a year
yet
and has a lot of information i know
there's some stuff that i need to update
and i'll update it soon so
so yeah feel free to start to fork it
and if you have any other links that are
either related to kubernetes or
kubernetes security you can submit a pr
to me and then i'll take a look and add
to the list as well so there's a lot of
information here
okay so i'll give an overview of about
what is kubernetes so that you can
understand and then we go to set up the
environment and now continue the slides
while the environment is setting up
because it takes a few minutes to set up
the environment on aws
so yeah let's let's do that first
so what is what is kubernetes right so
what is your understanding of kubernetes
if you have uh if you have seen it if
you have used it right
yes kubernetes is an open source system
for automating deployment scaling and
management of contourized applications
right and and but what does that mean
if you remember when docker was released
in 2013 right um there was a lot of uh
hype about using docker and using
containers and and
containerizing everything right
but quickly we learned that by doing
that
there was a lot of uh
it increases the overhead of management
of those applications right because when
docker was released there was no way for
you to
easily manage
multiple multiple containers running on
multiple servers right and
and so kubernetes was was created
as an idea to help manage those
containerized applications right so it's
a no
a container orchestrator as we say it
right
and um they just released the latest
version which was 1.22 right and it has
um before like uh until last year was
new versions were released four times a
year so kind of every quarter and i
think now it's three times a year so it
gets updated very quickly right so you
need to
follow along and keep yourself updated
so that you also can
keep your cluster updated right
um so some of uh quick facts and fun
facts about kubernetes here
it was created by google right in 2014
and it's based on uh on internal
projects from google which are called
borg and omega right which are closed
source right and they they decided to
google has been used containerized
applications for for uh many years now
and that's why they worked on this
project kubernetes means uh
it's a word from greek which means
helmsman right and helmsman is like the
the person who kind of drives the ship
the ship right a big ship and in kind of
a
allusion to docker which are the
containers right and the person kind of
steering the wheel of the ship there
that's that's the helmsman right
um
sometimes can be the captain sometimes
it's another person so that's why um
and that's another reason why the icon
for kubernetes is a helm right that's
what we call this kind of steering wheel
of a ship right and it's a helm
um
yeah uh another fun fact burku brunei's
is it's also called gates right kas and
the reason for that is just because
there is eight letters between k the k
and the yes of kubernetes right if you
count the letters there in between the
uber nettie it's eight letters so that's
why an abbreviation of that um that's
the why the way that they do it
um
another thing kubernetes developed in go
golang right it's as same as docker and
many other um
cloud native applications as well
so yeah
okay
let's move on
uh why kubernetes right so since since
it's important to understand what
kubernetes is it's also important to
understand why kubernetes like should
you use kubernetes in your organization
right you don't want to go
go to your company on on monday and say
no we should move everything to
kubernetes because kubernetes is the
next big thing right
um there is a lot of uh
a lot of management required to run
kubernetes clusters right and and as i
said like i think personally from my
perspective it's a very complex system
it's hard to understand and especially
if you don't have the the background of
what kubernetes
do in behind the curtains right so once
you understand that that gets easier to
to know what it's doing but but it's
like
you need to have that background right
so yeah just a kind of a
a comic here from dilbert right it's not
everything that needs to be okay
kubernetes right doesn't mean like that
it's
the you have uh
very few people that really really
understand kubernetes properly and
especially even kubernetes security and
and yeah it's if you want to
start moving your applications to your
kubernetes cluster then you're going to
require not just
uh budget right resources to do that but
also uh skilled people to maintain your
clusters right so
think about that before you go into okay
yeah let's let's move everything to
kubernetes it might not be uh the
solution and it might not be uh uh
required for your needs and your
organization needs
okay
uh what is the cncf right the cncf is
the cloud native computing foundation
it's the foundation uh uh it's a soviet
foundation of the linux foundation
it helps and maintain many different
projects from uh what they consider
cloud native right
so kubernetes is just one of them there
are many different others that you can
see here and some of them are even part
of kubernetes as lcd and core dns and
also helm to to to
package
kubernetes applications as well so
there's many different ones right so it
doesn't mean just kubernetes when we're
talking about cloud native uh cncf the
cloud native computing foundation and
cloud native security we're not just
talking about kubernetes right this is
the the url there at the bottom the l
dot cncf dot io if you go there it's
going to show you a lot of uh different
projects that are related to cloud
native right and just a quick thing here
about um
what what means to be a cloud native
right doesn't mean that the application
only runs in the cloud right and we're
going to talk probably going to talk
about that in the next few slides but
just
just to add to that
um being cloud native is is there's
different characteristics and
applications that they uh they have and
they're more they're they're they're
created to run in cloud-native cloud
environments but they can also run on
premises right there are there are
organizations and and many different
companies that run kubernetes on
premises right
okay
yeah so
that's the slide here as i said like
what what it means to be cloud native
the definition here it's from the cncf
itself and can be found on their github
and basically uh as they say to a cloud
to an application to be cloud native he
needs to have this characteristics on
the left your seven characteristics and
some examples here that you probably are
using or you probably heard about right
containers service meshes micro services
in general immutable infrastructure or
infrastructure as code and declarative
api so these are some examples of cloud
native applications
okay
okay so i'm going to stop here before i
go into the architecture and explain
all the details i'm going to share my on
my screen and and try to uh
have everyone follow along to set up
their environment okay
so let me stop sharing let's see if it
can
oops i can share again one second
okay
okay can you see my screen yeah good
so the uh one of the prerequisites of
the workshop
as we mentioned he's having an aws
account right so
go uh if you're not logged in on your
aws account please go ahead and log in
and and what we're gonna do as i said
like we're going to
deploy a
a cloud9 instance right which is this
service from aws and we're going to find
out very soon how it works it's
basically an online version of vs code
um
so if you use vs code before you're
gonna find it very easy to follow along
and from that right the idea of why why
i'm using this right the idea is that i
want to make sure that everyone has the
same environment right and so if you're
using i didn't want to like create a vm
and deploy and send it to you to
download and everything let's let's do
it online right and using cloud9 is a
good way to everyone is going to have
the same environment if we use the same
uh configuration here and then from
there
we we can set up the instance to deploy
the cluster uh afterwards right once we
have the cloud nightings set up and that
takes uh that's that very quickly but
then we configured the cluster on aws as
well
so
okay let's start here and i'm gonna
create one from scratch i have some
created here but i'm gonna create a new
one
so let's see cloud
out village right
workshop
so name the name you can can give any
name that you want right just so just
remember if you have different uh cloud9
instances
but yeah you can add a description as
well
here
uh you have the settings of the
environment of your cloud linings it's
going to deploy an ec2 right so the way
that cloud9 works it's deploying an ec2
but it's going to configure it that's
going to work as an online developer
developing platform right so it's
basically online vs code that's hosted
on aws so that you have uh uh
all your code there if you want you can
use that as well
um so yeah basically it's going to
create a new ec2 environment right the
idea here you can use t2 micro right
um which is free
uh free tier if you have free tier um
yeah just another note here before
before we move on
there might be some charges because not
because of this cloud9 but once we
deploy the cluster and eks and
everything
uh depending on how long you leave the
cluster online there might be some small
charges on on on your
credit card on your account right but
but at the end of the workshop i'm gonna
show how to remove everything so that
you don't leave anything uh running and
don't get charged a lot of money okay
sounds good
okay so here if you're out there
um basically we're going to create a new
c2 instance i i'm it doesn't matter like
i don't need that large instance for the
cloud 90s because i'm only dealing with
code and executing some commands there
so that's fine
the platform can be
amazon linux 2
and i can leave this everything default
here after the 30 minutes of idle time
if i'm not using the cloud9 instance
it's going to shut down
automatically here so i can leave
everything here
as it is right by default
and i'm on the right region yeah i'm on
usb s1 so i'm going to deploy it on usb
s1 the cluster i can deploy anywhere and
and we're going to see how to do that
but basically for the cloud9 instance
make sure that everyone on is on the
same uh uh same region and they have
this access to the same features right
that's that's the only uh the only
reason for that
okay here just uh um an overview of of
the settings that i set up basically
everything is default right now
uh we're going to create a new row for
for uh my instance later but let's do
let's leave it at that right now
okay so it's it's creating my
environment it might take a few seconds
to minutes right
and this is going to be like if you use
vs code before if you work with
coding in and programming in vs code
you're going to find it very interface
is very similar there is a few things
that we need to do here
um
once the environment is set up
and and so creating an iem uh row for
our instance and also disabling some
credentials so i'm gonna show here once
the cloud nine is is completed
and deployed right
um let me see
see if it's taking a lot of time
yeah let me go back
um yeah if i have um i don't know if we
have any questions so far
but yeah feel free to ask them if you're
stuck if you
if you're like you missed something uh
uh until now
feel free to uh send the questions over
through the the form the google form
okay no questions sounds good awesome
thank you
so yeah let's let's let me wait here
once okay now the client name cloud9
instance is set up
as you can see it's basically an ide
platform
online
i have a terminal here um i can add new
files and new terminals as well right
okay
so what we're going to do i mean
new terminal
okay
is the font good enough
uh for everyone to see
i don't need maybe
let's leave it like that okay
so this is basically the environment
that we're going to use
to
uh deploy our cluster from right
okay so a few things that we need to set
up here and i'm going to show you
is oops sorry
it's on this settings here on the top
right corner of my
uh interface there's
a gear icon called preferences right and
in this gear icon i need to go to
aws settings
and on the aws settings i'm going to
disable this aws
manage temporary credentials right all i
need to do is click here and disable
that
and that should be fine
okay
everyone got that preferences go to aws
settings and disable the aws manage
temporary credentials good
so after that
um
let's see
let's now we're going to
uh
let's let's let's create the now we're
going to go back to the aws console to
create
to create uh uh the iam road that i need
to give to this
uh ec2 instance because the cloud9 is an
easy to incest to be able to have
permissions to to deal to talk to ek
acid and create clusters
yes the workshop is going is is being
recorded and it's going to be shared
later as well
so yeah
fine
okay let me go back to
the aws console here let me open another
tab
and i think i need to share
stop sharing this one and share the
other one sorry about that
okay
so yeah on oops
i think i'm in the console yeah
on aws console what we're going to do is
create an im row and
please don't do this on your
production account because what we're
going to do is create an iam role with
uh
administrator permissions administrator
privileges
so that we can give um
can give the cloud9 instance permissions
to create the cluster and everything
right so that we don't have any issues
we don't face any issues we're going to
give this administration a row so i'm
going to iam and let me let me go back
so what i did was okay i went to iam
which is the
the identity and access management uh
service from aws
and basically what we're going to do is
create a role
for that
create row
i go here to rows on the left side and
create row
and here
i select aws service that's selected by
default ec2 okay
and then i click next in permissions
under permissions it's going to load and
i'm going to give this row administrator
access to my account right so
be very careful with that don't use that
in in a production environment all right
we don't want that because that's too
permissive right i don't want to give
that iam roll this is just for the
workshop so that we don't
don't have any issues deploying the
cluster
uh you can add tags but that's not
needed
and row name
c9
i'm gonna do cloud
cloud9
instance row okay
you can put any name just remember that
name because you need to attach that to
your cloud9 instance okay
so what i did was just repeating what is
what i did was i went to iam i went to
row and create a new row
selected ec2 and gave uh the policies
administrator access and created a name
for it right that's it
okay the role the role has been created
so the next thing that i need to do
is go back to
i'm going to the ec2 now
and attach that row to my easy tuning
that's my cloud982 instance right
so i'm going to ec2 here that's why i
tap ec2 and okay to see my machines
running i see that there is an instance
running make sure that you're in on the
same region that you deployed your
cloud9 instance
and basically
see there is this instance running here
that's that's why i i have the cloud9
running as well so select the instance
here the ec2 and
on actions
security
modify iim roll okay
that that's going to
uh we're going to attach that row that
we just created to the distances
so again select my instance go to
actions security and modify iam row
and here i have my row that i just
created the cloud9 instance row and i'm
going to give it i'm going to attach
that to the instance and i'm going to
save
okay
if you got
uh get this message so successfully
attach your role to the instance that's
that's great that's
all we need so far
um
is is anyone
uh behind does anyone have any questions
anyone struggling with something feel
free to ask okay
so now that i set this up
i need to
go back to the cloud9 and install a few
things on my cloud9 instance
to
be able to deploy the cluster right so
some things i'm going to install
is uh
cube ctl
some some other packages so we're going
to follow along let me share that with
you
let's see if i have it here
okay
yes let me share my s let me go back to
my cloud nines just because i'm not
sharing that
yeah
sorry for this switching here but
i don't wanna share any
any sensitive information right
yeah okay
so i'm here
one of the things that i need to do here
on the cloud nine i've attached remember
if you haven't done the aws settings
right disable that as well just let me
show it again
right the temporary credentials
and
yeah okay so you can see that there is a
kind of a directory structure here it
shows your environment
i can have many terminals and all that
stuff that's great
so
now we're going to
use this
this project on github and and
to download two files that that's all i
need to set up my cluster basically off
and i'm going to install uh some some
package and some tools but basically
those two files it's all i need to set
up my cluster um
so let me show you here we clone and
this url the kubernetes sac 101 workshop
if you download that
you can see that there is basically two
files there and i'm going to explain
what they are soon i'm going to
increase the font
let me close this i don't think we're
going to need this okay
here
and yeah
so you can see that there is two files
here two yml files right and if you
haven't played with yaml before that's
that's something that uh kubernetes uses
a lot
right to um
to manage uh to deploy applications and
objects on kubernetes right they're all
usually ml files that
that turn into json are sent to the
kubernetes api server and i'm going to
explain that soon
um
if you haven't cloned this so the
command is git clone let me post the
whole command there git clone
uh i'll share it with the um
the people participating and
yeah if you clone that if you have those
two files on your
on your cloud9 instance you're good okay
so far you're good let's let's let's
move along
so now
uh um this this cloud9 instance is brand
new right so i don't have if i try to
use cube ctl which i'm going to explain
some what it is but it's basically a cli
to
talk to your kubernetes cluster um
so i don't have that right now right
command not found that's not installed
so i need to install that
so the first command to install um
cubectl is this one here let me see
okay
let me post that on the chat
and so this is the first command here
and i'm going to share with you so
basically i'm downloading that
uh from aws and see if that's done
okay
so now i should have
uh oh i need to give it permissions of
course
the second command is to allow execution
permissions to this
binary right keep ctl and don't worry
i'm going to explain everything once we
once we start deploying the cluster
which is the the thing that takes uh
longer then we go back to the slides and
i'm going to explain some things that
i'm doing here if you may not uh if
you're not uh able to understand yet but
you you will be able to understand the
words
okay let's see
yeah so keep ctl is working
right we have that already now cube ctl
controls the kubernetes cluster manager
that's great
um other things that we can install
that's gonna help you um
with our
interaction with the cluster so let's
start a few a couple packages
uh um that can help you in and those
packages are jku and and get tags and
batch completion right so basically what
i'm going to do is sudo uh yum style
those packages right because they don't
have that on the
on the cloud9 instance by the phone
great
now
i'm going to
[Music]
configure
a batch completion to keep ctl so that
helps me when i'm
managing my cluster and using commands
with with cubectl
so basically cubectl completion bash
bash completion
and um
another one here
let me oops
let me post those two comments to the
team as well so they can share with you
yeah
okay so this is a true separate command
that you're doing and let me
let me clear here so that i move to the
top okay that's better
um okay
any any questions so far anyone
struggling any comments that you don't
understand
there's just a couple more things that
we're going to do now we're going to
install eks ctl right and and what
eksctl is
is a a
another binary to interact and and help
you deploy eks clusters right and what
is zks eks is the manage kubernetes
services from aws
right so it's
it's like a a kubernetes that's provided
to you by aws and aws manages that right
every major cloud provider has their own
managed kubernetes services you have eks
for a uh for aws you have aks for azure
you have gke from google and and many
others right
okay so we're going to install the eks
ctl uh binary because that's what you're
going to use to deploy the cluster once
the cluster is deployed
excuse me we just use
cubectl
basically
okay
so let me set that up
this is the command um i'm posting that
to the team so they can share with you
too
sorry uh
okay
moving that to
local being
and
also enabling
batch completion for eksctl
and
that's it
okay
asctl okay that's working too
great
so so far we've only uh downloaded a few
package installed cubectl and installed
eksctl that's what the two main binaries
that we're going to need
to uh uh deploy our cluster
right people may ask oh why why don't we
need to install docker right we're not
we're not running our cluster here on
this instance right we're running that
on
we're going to deploy two new ins
actually one new instance to uh uh to
run our cluster right and the way that
uh eks works
is that on
uh they they run they manage the control
plane right off of the kubernetes
cluster and we only need to deploy our
worker nodes and that will that's what
we're going to do we're going to deploy
async to instance that's going to work
as our worker node to run our
applications
okay
let's see
no questions so far
oops stick your download link for
sure
let's see
this is the whole
yeah this is the whole link yeah maybe
if it's
truncated
yeah that's
that's the whole link basically i'm
downloading the cube cdl binary if you
want download that from directly from
kubernetes uh uh the kubernetes website
which is kubernetes dot io there is also
a cube ctl there but i'm downloading
that from aws from
an s3 on aws that that they provide
right so that's
that's
better i think i don't think you're
going to have any issues if you download
from from the kubernetes.i o website
um but yeah if that doesn't work uh with
the link please please let me know
okay
so so far we
we configured
just to give you like a um
overview of what we've done if if you
just joined the workshop
we deployed the cloud9 instance we
configured an iem row for this instance
and then we downloaded the
two files from a github
uh that we're going to use to set up the
cluster and then we installed cubectl
and eksctl that's all we've did and
don't worry if you don't understand some
of those names and tools i'm going to
explain that as soon as we start
deploying our cluster and move back to
the slides and i just i just don't want
to waste
too much time um so i want to do the
setup first if
people haven't been able to do that and
and then we go to the slides while the
cluster is is being deployed
okay
let's see here um
so basically what we're going to do
is okay to create this cluster i'm going
to use let me show the files first let
me go back to the files
okay
so the first file that we're going to
use is uh that we're going to interact
with is this dks dash cluster.gamo all
right this is going to deploy my cluster
on
aws
so that's why we installed dks that's
going to help me and and here on this
fire you can see the command here
already right
um that's that's what we're going to do
basically run this command on the
terminal uh
of course pointing that to the file that
we have and basically uh i'm deploying a
cluster configuration right
and this is the name of my cluster this
is the region so here you may be able to
change your region if if eks is
available uh on the region that you are
in
right and
it's also a good idea because if we
deploy all i don't know how many people
are following along and applying
clusters as well but if we all deploy
our clusters on the same region we might
run out of resources so
the way that it works on aws they have a
limited resources for
availability zones right so inside the
region there's different
availability zones and we might come
we might have some air horse saying that
you don't have enough resources to
deploy that cluster in that region or on
that availability zone because eksctl
they're going to choose
from this from this uh um
this configuration here they are going
to choose which availability zone on us
is one region
they want to deploy a cluster we we
don't manage that of course we could we
could add that to the configuration file
but i don't want to add a lot of uh
overhead to the configuration so you
don't need to worry about that
and on this cluster i'm creating a
managed node group
right and i'm calling that manage node
group one and i'm deploying an instance
uh of the size t2 small
right and here i can say the minimal
size and the maximum size that means
that
my cluster uh can
have only one worker node one in one
instance and maximum
free right so if i need to increase
uh the number of worker nodes on my
cluster i can easily do that right and
if i like if i can change that to 10 it
doesn't matter what's going to matter
here is the desired capacity right
that's what what's the cluster is going
to start with right if i change that to
free for example then it's going to
deploy three different ec2 instances of
the same size in the node group right
and i don't want that i don't wanna i
don't want free i don't wanna we're only
going to use a small web application
so
i i don't want that i just want one so
that it doesn't i don't need to pay a
lot of money uh for for deploying this
cluster all right this is the volume
size right the hard drive and some
labels the worker uh labels the worker
role right and that basically that's it
right that's that's all that is on my
cluster
so here
i'm going to use this command let me go
to the
okay say workshop
here is my eks cluster so i just enter
the directory of the uh
of the guitar project that i just
downloaded again it's it's just
if you haven't
this is
say hello
do you like sure okay
um
yeah maybe maybe the chat let me let me
share the
the whole file here
one second some people are struggling
with the commands the first commands
and then give me a few seconds here
let me put that in a drive and um
making sure that
one second
hang on i mean
i should have done that before
sorry about that
[Music]
okay
okay let me share
on this file here i created google docs
and i will share the link of the google
docs of all the commands
and
um
and then we can share with everyone
participating so that they can check
let me just
select 101 workshop
that quickly
okay yeah i'm sharing on the google docs
um so if you can share that link with
everyone everyone can see that link
and these are the main commands that we
did to
um
set up cubectl
and eka city out on the cluster so if
you're having any trouble
um some of those are or two commands so
remember that and yeah step one step
three so follow that along and check out
the commands there if you're still
struggling after following that let me
know please let me know
um
so i'll i'll move on here and i'll
deploy my cluster and and while that's
that's going on as we going back to the
slides i can help anyone that's
struggling or answering any questions
right
so yeah as i said right i i want to
create my cluster and to do that i'm
going to use eksctl
uh to do that and the command here is
ekstl create cluster right and i'm
passing a file
which is the one this file here that i
have
so this is the command and i'll add that
to the the spreadsheet as well
so
um so that you can follow along if you
uh if you get stuck or something
so let me add that to the spreadsheet so
that everyone can see
a gift clone
right it's like 101 workshop
okay
i'm adding that to the
to the
sorry to the
google docs
just adding that there so that you can
see
okay let's see if that's going to work
yeah like demos or live workshops
crafters always are always hard
so here i started creating my cluster
and as you can see chaos ctrl is doing
its job it's creating uh subnets it
creates a vpc and then does everything
for me so that makes things easier right
and
um this is what takes a little bit
longer you might face some errors here
uh because of resources if we're doing
on on the same region if everyone's
doing on usc one uh what what you need
to do basically is either try on a
different region or try again but change
the name of your uh cluster so that it
doesn't uh
what what this is doing actually eka ctl
is using cloud formation to deploy your
cluster right as you can see here it's
waiting for the cloud formation stack
and all the stuff and so
if you try to deploy and it didn't work
then if you try to deploy again with the
same name it's going to complain to you
oh there's already some cloud formation
template created with the same name and
let's hope we don't have we don't face
the issues these issues i faced that
on the previous uh
edition of this workshop that i did
but let's hope not
it seems that no issues so far
um so yeah i'm gonna leave it running
here i'm going to go back to the the
slides so that we can continue the
just the explanation of the kubernetes
architecture
and then once the cluster is deployed we
come back here to uh uh to deploy the
applications and and now play along with
the uh attacking the application and all
that stuff
okay
so i mean
stop screen
and share the slides again
okay
so yeah we stopped
we stopped here
we stopped here right the kubernetes
architecture
and
so as we can see
if you if you're struggling feel free to
ask questions i'll i'll stop and answer
them
uh why i'm explaining no problem don't
worry
as we can see
there's two main major components here
to use the kubernetes control plane it
used to be called
master node but for
inclusive language we don't usually we
don't call it anymore even on the new
version of kubernetes it doesn't say i
think it does say but it's being
deprecated the name master node right so
i'm going to call it uh from now on
control plane
and so this is on the left side here
with with five five smaller components
on the right side of the slides we have
three worker nodes right
and and the worker nodes are the ones
that we use to deploy our applications
and and today in this workshop we're
deploying only one worker node only one
in c2 instance that's going to function
as as the main
uh the main node for deploying our
vulnerable web application we don't need
three but if we need it as i showed
there with the the mac size and the
desired capacity on on the yaml file we
could easily change that there right
okay
awesome awesome yeah yeah i should have
thought about the google docs um so yeah
on on this on let's look at the left
side first
the cube api server
um the cube api server here in the
middle right we see that that's kind of
a
big thing it's a main component one of
the main components of the control plane
every other component is talking to the
api server and you can see from the
arrows there it's not just talking with
the
uh components from the control plane but
it's also talking to uh the worker nodes
as well right
and the cube api server it's it's
basically a api server and rest api
server if you played with apis before
and
that receives and forwards all the
communication from all the components of
kubernetes right it has a um
basically everything that you
when you type from cube out it becomes
an api request or an hp request that
goes to the api server
of your cluster right
on the bottom here on the bottom
left
we have at cd
and at cd
is
is the main database the data storage of
your cluster right it's a key value
store where all the components of the
kubernetes kubernetes cluster is stored
there right and and so
everything gets saved there as a
as an object as a key value store and
then uh kubernetes tries to checks that
and tries to create whatever you told it
to create on the that's saved on that cd
tries to create that on your notes okay
and that's something very important here
because
the
the kubernetes works in the way that
kubernetes works it's called desired
state right so kubernetes is smart
enough right it's smart enough to see
that
it constantly it's constantly checking
if whatever i told it to create for
example nodes or application or services
whatever i told it to create there on
that cd
it checks
it checks on your nodes and see okay is
this application running already do i
have free replicas of that application
right
if i do okay good that's that's the
desired state
if i don't if whatever is is on that cd
whatever configuration is there doesn't
match what's running on the cluster then
okay i need to fix it right i need they
need to match right and so
so
if let's say i only have one replica of
my application and i need free
kubernetes is going to check that and
it's going to tell the cluster to create
two more uh uh
applications two more containers right
from
actually spots but it's going to create
those to match those in cd so i need to
have three replicas running on my
cluster and it doesn't matter on on on
which worker node depending on your
configuration as well but i need to have
that matching right so it's constantly
checking if you check the logs of
kubernetes it's constantly there's many
different uh components that's
constantly checking to see the desired
state if whatever is on lcd should be
reflected
on
on your your whole cluster on your
worker notes right
so you can see here that the api server
and lcd there are two major components
of your kubernetes clustering that's why
you need to protect those and we're
going to talk about uh security the
security part of it and the best
practices as well but be very careful
with those components right if an
attacker has access to your ad cd and
they can change objects there right
basically whatever they change to
kubernetes is going to just follow along
just kubernetes is going okay if you
want me to create a deploy a malicious
container i'm going to deploy a
malicious container right it's it's not
it's not checking who who put that
information there it's only checking if
the information lcd is matching whatever
is created on the cluster okay
and
on the top left here we have the cube
controller manager right
and the cube controller manages exactly
this object that control that has a
bunch of different controllers the pod
controller the service controller and
other controllers that keeps talking to
the api server and checking on that cd
is it matching is it is it oh do you
have the right number of pods oh we have
five here but but we need ten right and
so it's constantly checking that then
each specific controller is doing that
the cloud controller manager on the top
right of the control plane is the one
that talks to cloud providers right
that's the reason why kubernetes can uh
work with different cloud providers you
have the cloud controller manager if you
want to use like disk volumes or load
balancers right kubernetes doesn't have
those objects doesn't have uh those
services so you need to use the ones
provided from your cloud provider and
we're going to do that on on eks today
we're going to deploy a load balancer to
expose our application to the internet
okay
good
and on the bottom right uh is the cube
scheduler it keeps scheduling exactly
the one that tells uh uh the worker
nodes to deploy the pods right to ins
and start the containers right it talks
to the cubelet on each worker node to
schedule
a pod it can be scheduled like instantly
right in it can be uh uh to cubelet talk
to the kubernetes runtime engine create
those spots
okay so those are the five main
components of the control plane right
now
let's talk to let's let's take a look at
the
the worker notes
here on the
on the right side
there are three main components on each
worker node
i have a cubelet i have a cube proxy
and i have the container runtime engine
okay
so
basically the cubelet is the agent that
runs on each node
that talks to the api server and
starts any any containers and also gets
some
statistics and health information about
those those containers right and i'm i'm
i'm talking i'm saying containers here
but it's actually pods but i'm going to
explain that once we move to the next
slides that the smallest unit of
kubernetes is actually pods right um
there is also the q proxy component
that's uh that handles all communication
networking communication inside the
cluster but also externally right with
outside the cluster as well
and the cube
the container runtime engine
which
by default was docker
and
you know there was some
a lot of discussions about when uh when
they're talking the docker was going to
be deplocated in kubernetes and all that
stuff i don't know if you heard about
this this story
but you can still use your docker
containers on kubernetes nothing is
going to change it's just that
um
the the the reason being that kubernetes
to
to be able to run docker kubernetes
developed a kind of uh another component
called the docker shim
to be able because docker doesn't follow
the
oci the open container interface
and kubernetes was created to to run any
oci
uh uh standard containers right
okay
so you can deploy that the con the
cubelet is going to talk with the
kubernetes runtime engine and that's
what's going to really create your uh
your containers and run that on your
nodes okay
any questions about this architecture
here so far
so we're going to interact right what
we're doing what we're doing here today
on this workshop we're using the cloud9
instance to talk to the control plane
right we're using that and tell we're
going to tell the control plane to
create some objects for us right we we
just
with the eka ctl command that we just
used that's creating the cluster let me
just quickly see that yeah still
creating um
with the
kstel command that we use to create the
cluster um
it's kubernetes is doing that and the
way that it works on eks
we don't have access to this control
plane right
because
uh aws handles that for us and and
that's a good thing because you also
don't need to worry about the security
aspect of the control plane all you need
to do you talk to the api endpoint of
your control plane and you tell it to
create
uh create applications deploy containers
and all that stuff
if you're still having uh issues
deploying the cluster if you're having
uh if it's saying that's not available
it doesn't have any resources on that
availability zone
try to change uh
change regions right if it's still not
working under different names if it's
still getting errors
i'm gonna show once my cluster is
completed i'm gonna show you the
the outputs of the console and see if
that you're getting that but try to
change different uh
regions if not working on the region
sure
uh the oci is the open container
initiative right it's like a standard
for uh
standard for creating containers right
and the
docker
didn't doesn't apply it doesn't like
it's not in compliance with that
standard as far as i know please please
correct me if i'm wrong
and and so
kubernetes
couldn't couldn't run
docker containers by default
they had to create another software
that's called the docker shin
to be able to do that
so the problem is that
this this component that
allows kubernetes to run docker
containers
is
is like
it's maintained by the kubernetes team
uh yeah
sure sure okay sounds good
um so
what's what what's going on is they
don't they didn't want to maintain that
component because that component was as
they call it the docker shim
s-h-i-m it is is a kind of a like
uh
it's not a good practice what they did
right because kubernetes should be able
to run oci standard uh containers by
default but unfortunately docker is not
uh compliant with that
so they said oh i know we're not going
to maintain the docker stream anymore
and the docker shim was going to be
deprecated
right so that's why this caused a lot of
stir on the on the
internet and on twitter a few months
back
because people thought oh maybe like
kubernetes is deprecating docker i'm not
going to be used to use be able to use
my docker containers and compress
anymore what's going on right because
pretty much i would say 90 to 95 or 99
of uh kubernetes clusters are running
docker containers and not the other ones
right you can have different runtime
engines you can have the container uh
the container d cryo or podman it can
there's different ones but since docker
was the default one people usually use
that right and and because docker is is
very well known and people have uh
have used docker before even before
kubernetes that's that's what they use
right
so i think that's that's that's it with
the oci there is a blog post i think
let me see docker kubernetes
advocated there's a there was a blog
post on the kubernetes website that
talks about that
whole
that whole issue
let's see
okay
yes let me post that thing here
and dr sheen faq
okay
let's move on if everybody got the kind
of the overview of the architecture here
let's move on with that
so as i said uh um before right
we're when we deploy a container on
kubernetes we're not deploying a
container itself we're deploying a pod
right so the pod is the smallest unit on
a kubernetes cluster right
the pod can have
one container inside that pod or
multiple containers right
so as the diagram here shows
um right you can have one two or many
right and and the reason being is that
the idea of containers is to run only
one process on that container right so
if i have my application running on my
container and i need something to let's
say collect the logs and ship those logs
to a central location
right i don't want to install that
application inside my application
container because that's going to break
the principle
and the idea of containers right so i
deploy another container on the same pod
usually on the same pod and that other
container talks to my first my
application container and collects the
logs from there to ship it to another
location so that container is usually
called a sidecar container right it can
be used to map to collect metrics logs
it can be used
as a security container as well some
tools some security tools that deploy
the word the way that they work they're
deployed as a side car as well
so there's different uh uh
different usage for that that sidecar
container right and since they're on the
same pod right they share the their uh
the name spaces there
um
so other objects that you can have on
kubernetes is a deployment right
you can you can deploy a pod by itself
but
you want to use the features provided by
kubernetes right and
the the way the deployment works it
gives kubernetes or your kubernetes
objects scaling
updates and rollback abilities right so
let's say if i need to
create another pod to handle traffic and
there is a lot of traffic on my
application i can easily do that with
deployments if i need to scale up or
scale balance more if i'm not using all
those containers and and i can save
money and reduce the number of
containers that are running then uh the
number of pods i can do that as well
with a deployment right deployment is
just an api a kubernetes api object
right it's not something that that
exists on on the the node itself it only
exists inside my cluster it's just a
configuration and the way that they use
there's many different objects that i'm
going to show on the next slide we're
not going to be able to cover all those
objects here but just to give you an
overview okay
and on top of that deployment
we have something called namespaces and
those are different than the the linux
kernel namespaces these are kubernetes
namespaces right and and the way that
the namespaces work here is that they
work like
they're just a logical separation of
different applications that i have on my
cluster right it's basically folders
inside my cluster right there is no kind
of a security boundary between those
namespaces right so i can have
a namespace for my
uh developer environment a namespace for
my keyway environment and a namespace
for my production environment i can also
have a namespace for my developer team
one and my name is space for developer
team two if they're both using the same
cluster right
so
and we're gonna talk about namespaces as
well here which is another part
on in another slide
and and on top of the namespaces there's
nodes right nodes is basically the the
server that's running my application so
it can be
a vm it can be an ec2 instance it can be
also a far gate on aws or other kind of
serverless
applications or service serverless
runtimes or workloads
so it can be anything it can also be
runny on prem and can be bare metal as
well there's there's also some uh
some companies that provide kubernetes
running on bare metal
okay
and let me go to the
api objects
so there's as i said there's different
api objects that you can create on your
cluster you have a pod as i mentioned
right there is replica set which is is
used by the deployment to replicate your
pods there is demonstrate stateful set
right the deployment itself as i
mentioned there's service that's used to
expose uh either like a port or or my uh
my application to the world they can
expose that through a load balancer um
as well
there's job and chrome job which are
used to uh uh just they use the job
container it's basically a pod or a
container that runs only once the job
and the chrome job it's basically a
container that runs according to the
chrome tab right i i
i apply the configuration there and it
can join it can run periodically like
every every other minute every other
hour whatever i choose and whatever i
configured on the crown tab
configuration of that container
um there is also config maps and secrets
my config maps and secrets are usually
used to store information right config
maps is for configuration information
that are not really sensitive
and secrets are supposed to store
secret information right sensitive
information
but
i'm just going to say here that
secrets are not really secret
in kubernetes because the way that
secret works by default is that all the
sensitive information is stored on that
cd
and it's stored on uh encoded as base64
right so you're probably aware that
basic 6664
can be easily decoded and so that's not
really protecting that information
and also
on xcd by default all the
all the information stored in that cd is
stored in plain text right it's not
encrypted right so there is another
issue there as well so there's different
uh ways that you can uh
use secrets on kubernetes you can maybe
use like a third-party solution like
hashicorp vault or use the
the one from aws as well it doesn't
matter so just be careful if you're
using secrets with kubernetes don't use
the default settings right apply there's
some uh configurations that you can
apply to at least encrypt your secrets
on that city
and yeah there is in-grass and many
different objects right and those
objects are all they all work
they can all be applied as a yemo file
and i tell that and apply that to my
cluster using tube ctl and and that
depending on what i'm creating on that
file that gets treated on the cluster
exactly because of the desired state
right i don't care how kubernetes is
going to do that i just tell it okay i
want this object created on my cluster
and and then okay apply that to to the
lcd once i send that information via
cube ctl that goes through the api
server and gets stored in that cd right
and now the the controller is checking
that oh something changed here and oh
they want me to create this uh this
object okay now deploy this object and
everything happens really quickly
okay
any questions so far
let me see here i think my cluster got
created so i'm just going to talk about
chip ctl and then we're going to go back
to the cluster if anyone's still having
uh
struggling or still having issues please
let me know
so basically the kip ctl as we
we did on the cloud9 instance right is
the cli tool that allows you to control
your kubernetes cluster right
it's it's a config file there is a
config file that
uh can be found on your home directory
under slash dot cube and i can show you
what's where that's stored on the on our
cloud9 instance right
it's very similar to the docker cli
uh for docker containers let me see
another question here is an api object
like request parameters to the
appropriate api endpoints
request parameters
not sure i understand the question
api objects on kubernetes
they are
they're
they're just like
different different objects they have
different
uh
how can i say functions right
so basically when you talk to kubernetes
and you talk to the api server there is
an endpoint for each each object right
so let's say my kubernetes cluster and
like slash api slash pod slash create
right so there's different endpoints
everything gets gets uh convert from the
api request and gets to the lcd
right
once that's stored on that cd
and depending on which type of object
that gets created on my cluster
right i'm not sure if you can try to
rephrase that question uh
please uh so that how can i try to
answer that better i'm not sure
if i understood correctly
but but yeah so the syntax of the cube
ctl right it's very similar to the
docker command the docker uh cli and and
the reason being as well that they did
that so
so that people that were using docker
already could easily shift and start
using kubernetes without facing too many
challenges right
so yeah that's it
see yeah that's before we move on let's
go back to our cluster and let me go to
uh the cloud9 instance here
okay
sure
okay
so you can see um
from the command that i did right create
cluster it did a bunch of things here
creating the cloud formation
and everything and should be uh
should be working right
so to make sure that everything is
working if as i said if you face any
errors if there was something going on
or there was no not enough resources
either either try to change the name of
your cluster and try again because it
could be just a temporary thing or the
availability zone that eks ctl chose for
you it wasn't uh there wasn't enough
resources and once you try again with a
different name it's going to try in a
different availability zone or you can
just change regions and and do the same
thing and and see if that works
unfortunately
i don't know if there is a way around
that if someone that works for aws can
tell me about it but as far as i know
there's
there's no way unless you create your
own
uh
subnets and vpcs and configure that that
you know that there are resources
available but that can be like a soft
limit
from
aws that i'm not very aware of
hey so let me just clear this
see okay good you guys see my screen
is there another question there
okay no questions
so basically
let me do cube ctl get
notes
okay
so i can see here right uh this is the
first command i'm getting all the nodes
that are part of my cluster right and
there's only one node which is this
instance here right that's running
and it's selling the kubernetes version
here
um
120
and
yeah it's ready right so the instance is
ready if i go to the aws console i can
see another instance created there as
well besides my cloud nine instance now
right
um
let's see
here on uh
on the
home directory slash dot cube
there are some stuff that was created
the config file right
if i cut that there is some
some information about my cluster how to
access my keys and other stuff right i'm
gonna not gonna show
everything but basically that's it it's
there you can see that there is uh some
certificates the ip of my cluster and
all that stuff
one thing that you need to be aware of
on eks
clusters is that by default
once i create my cluster
the api endpoint
the api endpoint for my cube api server
that's public by default right i know
that's a long url and it's it's hard to
guess
but there are attackers people on the
internet scanning uh the internet for
looking for those api endpoints
so if you don't need that
exposed to the internet be be be aware
of that and make sure that you configure
that to be private and we're going to
see that very soon
okay so now that my cluster is created
we're going to deploy the other objects
uh actually the objects off of my
kubernetes cluster right now my
kubernetes doesn't have anything right
so if i do keep ctl that
pods
no resources found
uh is some resources right from they are
the default resources of kubernetes uh
of the eks cluster right there are some
stuff core dns aws cube proxy running
but there is no no application run right
you can see that they're all running on
the cube system namespace which is the
main namespace of my uh control plane
components right
let me see if you've got pods
uh n s or or namespaces right so these
can do like this namespace or and as as
a shorter version
right and as i said namespaces are kind
of like
basically folders for organizing stuff
on your kubernetes cluster right and by
default i usually have those four
uh name spaces when i create my cluster
either on the manage or unmanage version
okay
so now let's create the objects right
and let me go to this file here and i'm
going to describe everything that's
there before we apply this uh
this object there
so i'm creating a new namespace on the
cluster object right you can see here
and this is our different every
the way that yammer works right every
three dashes here it it's it's basically
a separate file or separate object but i
put everything together right so that's
easy to
uh to understand
so i have one object being created here
that's a namespace called
web app right so basically i'm creating
a folder to deploy my web application
this is an our back configuration and
creating a cluster role and a cluster
role binding telling that the
um the service account and i'm going to
talk about our back and services account
later but basically telling the service
account that i'm gonna use to deploy my
web application my my my my pods running
on this web app are running as add me
right and and so that's a bad thing and
i'm i'm purposely doing a miss
configuration on my cluster but that's
for the exercise that's for the workshop
right by default you don't want to do
that you don't want to do that
um there is also
a deployment right and i'm using
deployments to
deploy my my web application i'm using
i'm creating like a drupal deployment
here because i'm we're deploying a
drupal web application and you can see
here down below that the specs of the
containers that i'm using i'm
downloading drupal 8.5.0
and since there's no specification of
which container registering i'm
downloading from
that's getting that from docker here
right so once i deploy that i apply that
object to my cluster
kubernetes knows that okay i don't have
that image here or or whatever whatever
you want so
i'm going to download the by default it
uses docker hub to look for images right
unless you want to apply of course
that's not a good practice there's not
something that you want to do because if
someone compromises the docker hub or or
puts a malicious image there then you're
downloading an image to your cluster
right so the best practice is to have
your private container registry and
store images your
approved images there for you to
download on your kubernetes cluster
sounds good
so and there is also
a a service i'm creating a service here
which i'm going to use services or
objects that are that i can use to
expose uh
my containers right and i can i can
expose just like there's different types
of service
the type that i'm creating here is a
load balancer which is going to create
an aws load balancer and i expose my
drupal web application right which is
running on port 80 and it's going to
expose that on a port 80 of the load
balancer as well
so as i said there's different types of
services there is cluster ipe there is
no port there is a load balancer and i
think one more
yeah i forgot the other one and no
problem i'll i'll oh i'll remember
and this is basically i'm adding
i'm adding just a kind of a flag here
that i use this is i use this this kind
of
setup to deploy a ctf challenge right so
i have a flag here i'm creating a secret
and i'm storing that secret as a
a flag
named ctf on the cube system namespace
and this is the the the result of the
flag here right the the the value of the
flag and it's basically base64 encoded
it's just a flag that we're going to
practice and try to get that once we
compromise the cluster okay
sounds good so
i i hope you understood all these
objects and we're going to apply that to
my
cluster and you're going to see that
that's going to be very uh
quick
let me go back to the
environment
let me go back to that uh project that
we downloaded from github
and that's where my cluster underline
objects is
um yeah just keep in mind that for the
first file
i don't know why i did this but the
first one i used dash and the second one
i used underlined so let's see if the
command is correct yeah it's using
underline so that's fine
so once i do this kubernetes apply
dash f right so dash f means that i'm
passing
feeding it a file and so it's going to
look whatever is in that file and apply
that to the cube the kubernetes cluster
let me just apply that
and you can see that all the objects
that were on that file were created
right the namespace web app
the service account the the deployment
the service and even the secret
everything is created now so if i do now
keep ctl get
namespaces
you can see that there is a web app and
space is there on the bottom right
if i do tip ctl
get um
pods
and web apps so now that i'm i just want
some information from a specific
namespace i'm using the the parameter
dash n to tell kubernetes i only want
the pods from this namespace right
see there is one pod running what
happens if i get this like cube ctl get
pods
no resource is found because if i don't
tell kubernetes the cube ctl a specific
namespace
it's going to look into the default one
the default one is this one here at the
top and it doesn't have anything there
that's why it's not gonna find it
right so you need this that's something
that people that are starting to work
with kubernetes may
may forget and may think oh i just
created this where is it right if you
don't specify anything
then it's all the objects that you
create are going to create on the
default namespace but as you can see
from the uh object that we use here i
created this new namespace and i told to
create all these objects on the web app
namespace right except this one which is
on the cube system okay
see
okay let's see another question
[Music]
where are these api objects going to do
these get sent to etcd
and then some pooling services check for
new additions resources using apis yes
yes so the api objects they go to at cd
as a like
a key value store right so there is this
stored on that cd
and
the controllers
the the cube controller manager with
their controllers they check at cd and
see what change do i need to create this
object oh oh yeah this i need to create
a service or this load balancer or this
pod and then it talks to other objects
like the if it's a pod talks to the cube
scheduler and tells okay create the spot
on my cluster i don't have that on my
cluster right now and i need etsy is
telling me lcd is the source of truth
right sd is telling me that i need this
cluster these object there so create
that for me so some objects in
kubernetes are are like
how how can i say they they have a uh
physical representation right so for
example uh uh uh pod
it creates
it it creates itself there on on the
when it's created on the cluster
i can see it i can see that there is a
pod and there is like something running
there is a container and all that stuff
some other objects are just kind of
uh how can i say they're kind of they're
more abstract
they exist there the existing kubernetes
and stuff but there is no kind of
physical representation of that object
besides the configuration at cd
right so it's just like uh uh some stuff
that kubernetes do to to to
kind of encapsulate that object in some
specific configurations like such as
deployment stateful set
it's just a way for you to
uh to help you deploy uh containers in
like different with different
characteristics right basically that
so yeah i hope that answers your
question if not yeah let me know
okay
so now i have a web application running
and we expose that web application
to the internet v via the load balancer
right
so now
i can get the service
of my web application
right and you see that there is a
service called uh drupal dash svc
load balancer running here and this is
what we need right the external ip
so this is the ip of my web application
that should be running there so if i
click on that if i copy that url and
open on my browser and i'm going to show
you right away
let me share the screen again
to
change
change here to my other tab
okay
here
oops
yeah so you can see here that there is a
drupal
uh
web application set up on on this url
right and i need to install it i need to
configure and that's what we're going to
do right and uh um but yeah basically
okay it's running i'm exposing
this drupal container to the internet
with with only that cluster only that
container right and with the service
load balancer of course if you go to the
aws console where where the load
balancers are you're going to see that
there is a load balancer created there
as well i'll show that later but yeah
so that's what's being used my container
is running inside my cluster my
kubernetes cluster i'm using the load
balancer to expose that and i'm
accessing that through that long url so
don't use the same url that i'm using
because your url's probably different
than mine right depending on on where
you deploy your cluster
um
so let me install here i hope that
nobody messes with my
drupal
what i want to do here is choose the
language use the standard one right we
need to configure the web application
and need it running so that we can
exploit it
i'm going to use sqlite which is the
database here
so that i don't i don't need to set
anything
and
yeah it's going to install
and there's some just a few minor uh
configurations
there but basically we all need to do
that if we want to exploit and and play
along with our
uh
[Music]
exploiting the application right
so dodge coins i can set anything here
like this this doesn't matter this
configuration is just to complete the
the configuration of my website but
but yeah cloud village
thatcom.com i'm not gonna use any of
that so don't worry about remembering
the
the the username or password or or
or anything because we're not gonna use
that we all we only want to do that to
set up our
um
our drupal web application
so i'm gonna use just a strong password
just in case that anybody tries to
mess with my uh application
but yeah that's it
okay
so
[Music]
yeah okay it's fine
so yeah now i see the drupal web
application is running right and you can
see that's working i'm logged in as
admin i can post i can like it can do
anything it's it's a regular drupal web
application it's just using an outdated
version because that outdated version
has a vulnerability right and now we're
going to uh
now we're going to exploit that
vulnerability
okay
no more questions
so yeah let me go back now to
to the slides i'm going to talk a little
bit about how we're going to exploit it
and the vulnerabilities that we're going
to do and then we go back to this web
application
true
okay
see change the slides you're good
so what we're using today uh um
and and as we mentioned in the beginning
right we're using trap modeling to
to attack this this web to see what
vulnerabilities are in the application
and then poc right so that's one of the
things that i do during my daily job
right so analyzing new technologies
especially cloud and container
technologies um
check the possibilities of of
any kind of like a hypothetical attack
and create a threat model for it and
then try to poc it right try to create a
proof of concept for that as well
um one of the things that we like to use
is the miter attack framework right
and as i mentioned the meta tag
framework is is a a very well known
framework on
the
among the infosec community right it's
like a
knowledge base of tactics and techniques
based on real world scenarios right so
you have the the minor attack uh for
enterprise you have the mother attack
for linux for windows
and you have even one for cloud which is
now called ias infrastructure service
for for uh the all the cloud major cloud
providers
but we didn't have one for um for
containers and kubernetes until
this year right
so
what uh
what we did was uh there was a um
one matrix released by microsoft
in april last year april 2020
that they they kind of use the miter
attack framework structure of tactics
and techniques this this matrix approach
to
uh to create their own their own matrix
like based on the data that they were
seeing on azure on their aks clusters
they created okay so these are the
techniques that attackers are exploiting
in kubernetes environment right
and so but that's not an official
miter matrix right a minor project
so in
in december december last year the the
mitre team for uh cloud and containers
they released like they publish a blog
post
asking
uh for help for from the community
to collect data about real-world
scenarios and and what attackers were
doing on those environments right so
either uh previous attacks that happen
real world attacks that happens to
organizations such as tesla and um i
think there's i don't recall others now
but yeah there's many others eks as well
they used as a baseline so
they're they're trying to gather real
world data and real-world information or
or honeypot information from what
attackers were doing there so since we
do this uh our research on on a daily
basis we had a lot of data that we reach
out to mitre to provide them with that
data to help build this matrix so the
the work started uh basically last year
beginning of this year
and in april this year they released the
official minor attack for containers and
and i i kind of
add and kubernetes as well because
that's not that's not what how mitre
calls it but there are some techniques
here that are very specific to
uh kubernetes environments right or
orchestrated container orchestration
environments but yeah basically this is
the miter the official miter attack for
containers that was released in uh april
this year
so you can see that there is the the
common tactics there and there is those
those techniques there are various some
of those already existed on the miter
framework but they didn't have the
context of
container environments right so they add
that reference or the technique
specifically or that right actor
and and some of the techniques are brand
new they didn't exist before and so
that's that was a kind of a cool project
that i did that i helped build with uh
with meyer and there is the link there
with the matrix but it's all official
made matrix uh for the mighty attack
framework for uh containers is this one
and
and yeah there's different techniques
here um
different actors actively exploiting
those environments
on on on one honey pot
that we deployed um it took less than 24
hours for the attackers to compromise
our environment and we didn't we didn't
like publicize that we didn't we just
deployed the kubernetes environment with
a vulnerable web application and in less
than 24 hours the attackers had um
compromised the cluster right and even
broke out of the cluster and compromised
the cloud environment that that we set
up the cluster in
and they started deploying
big instances to mine cryptocurrencies
right that and that's usually the end
goal here with docker and kubernetes
environments when attackers compromise
those environments
oh like 90 or 95 percent of the times
the end goal is to
either
um compromise the the already the
containers that are already running or
deploy new containers to mine uh monero
cryptocurrency right that's that's
usually what happens most of the times
okay so what i did with the track
modeling part was even before this was
done this diagram here was done even
before the miter framework was released
it was done last year
and what i did was exactly create this
model diagram
based based on the the
microsoft attack made the microsoft
kubernetes track matrix
and and so
i created like a scenario where okay
which are the steps that
an attacker can do to compromise a web
application running on a vulnerable web
application running on a kubernetes
cluster
excuse me
so yeah here we have the initial access
right the web app running in a pod
so we have that already we set up that
already on aws right and and the web app
has an application vulnerability and
you're going to see that really soon
we're going to talk about the
vulnerability that that
this drupal outdated drupal application
has
and the attacker exploits this rce right
they exploit this this vulnerability and
then they get a shell inside the pod
inside the container right and from
there they can do other things right and
so this is the kind of the diagram that
i created uh um to help me
understand what what an attacker was
able to do and and that's when i did the
poc to validate that
and of course then
then that's where this kind of this
workshop came from right from this
diagram there as well
so i can i uh i don't know if you can
see the whole diagram if it's if it's
too uh too small but i can share just a
diagram link later with you
so yeah attacking kubernetes if we have
set up this
like i call the website dodge coins
right if you have the drupal set up now
you should see kind of a similar screen
a screen to that to that screen
okay
so what we're going to do
right what are we going to do today is
exploit this
yes yes i can share the attack scenario
diagram
um i don't have the link handy right now
um yeah someone is asking if i have this
attack scenario diagram uploaded
anywhere sorry um but yeah i do have it
i just don't have the link handy
with like a high higher resolution
version but i can share that together
with the slides no problem
so so yeah the initial access here right
so that's when we start now attacking
kubernetes and i know it took i'm sorry
it took too long to get here but i
wanted to everyone uh follow along and
be everybody on the same page to set up
the environment and so that you can
understand the basics of kubernetes so
now that we can start attacking it right
and that's that's what's important here
like unless you understand the basics
now it's just practicing different
scenarios and different attacks and
maybe you can do that on your own later
um
so
two main uh entry points of a kubernetes
cluster right
one of the entry points here is a
vulnerable web application right and
we're going to see
why why is that
this vulnerable web application has an
rce right a remote command execution uh
as you can see the cve from 2018 and
we're going to use that exploit code on
that github
to do that to exploit that vulnerability
right
other ways that
kubernetes clusters got compromised
before was the exposed dashboard so in
previous versions of kubernetes there
was a dashboard that was created by
default that allowed you to kind of
manage your cluster in a
graphic user interface way right
uh but that got deprecated and it's not
uh uh it's not deployed by default
anymore you can still do it but it's not
it's not applied anymore by default and
that that dashboard has some
invulnerability some issues that's how
one of the
uh one of the the cluster attacks that
happened a few years ago on a tesla
that's how they compromise their
environment they expose a dashboard
and and so that's not the case anymore
and see even the reason why on the if
you look at back at the kubernetes trap
matrix by microsoft
they kind of deprecated that that
technique right because
since it's not deployed by default let
me go back you can see here as the
initial access exposed dashboard is kind
of deprecated right there is other
options right on the initial access
there is using cloud credentials
compromise container registries and all
that stuff but i'm going to focus on the
application vulnerability here
um there is also the cube api server
right
and if the cube api server is exposed
and it's not properly configured right
you can even deploy pods through the the
api server the api endpoints right
so so yeah we it's the same thing with
the docker daemon api right if that's
exposed and there is no authentication
there is no
and the person which with access to that
can um
can have
we can have like uh uh access can reach
the api endpoint
then then they may be able to do some
things or get sensitive information
there right
uh the cube api server endpoint is
public by default right instant managed
services like eks
um sally asks you the question here do
you have any issues with aws having
vulnerabilities in your clusters
um
with this workshop right we're not
we're we're just deploying a vulnerable
cluster and we're compromising it
ourselves we're not stalling any malware
we're not running scans
on on aws and
yeah we
we
we had some issues in the beginning but
but like with our honeypots but we
changed the way that we deployed and and
also the environments
and and yeah uh i don't think we're
going to have issues with that
this scenario today
so yeah but if you do please let me know
and we're going to shut down everything
later so it's not going to be up for for
a whole lot
so that you don't get your your cluster
compromised by like someone else right
we don't want that
okay
so one of the main things
um
one of the main issues that i said is
the api endpoint right reaching the api
endpoint externally uh that can be a
problem you can see two examples here
accessing um on an unmanaged cluster the
api server of kubernetes runs on port 64
for free and on uh eks it runs on on
port 443 right and if i have the url of
that api endpoint i can just do a curve
request to to that url and i can get
some information about my uh my api
endpoint
so let's let's try to do that first
before we go into exploiting that let's
try to do that first let me go to the
um
yes call so let me share my screen here
sorry
keep forgetting that
so yeah i'm here on my aws console and i
went to eks the elastic kubernetes
service
and i'm gonna check my cluster here and
i can see that i have a cluster running
and it should be at least only one if if
it's a brand new account or something
like that
right and that's my am i the cluster
that we created for this workshop
um
so this is kind of the console of your
kind of your control plane right oh it's
telling me that i'm using an outdated
version there's a new version available
i can see my uh
can i see my configuration here right
so this is my api
endpoint
and that's
usually exposed by default let's see if
we can if we can open a new tab
you can see that yeah
okay let me show you here we can either
do a curl or or um
or just open a new tab in the browser
but yeah i'm gonna show you here
okay
so yeah you can see here that i got like
an er right
let's see api
version or just
version
oh yeah yeah
so so yeah i got just accessing the url
there i got an error right and i got
like this is kind of a json error and a
403 telling me forbidden user system
anonymous cannot get path slash right
and and okay this is okay i don't have
access to that and that's fine but you
can see that this is api response is
very specific to kubernetes right
and and you can already tell that
by the by the format that there is a
kubernetes cluster running here right
and if i go to slash version which is
usually open to anyone by default
i can see
the kubernetes version that's running i
can see
right the go version of my uh of my
kubernetes and the platform and all that
stuff so this is kind of a
uh information leak right um that it's
exposed by default and
in many eks clusters if you deploy the
cluster like we did by default with the
default settings it's going to expose
once someone knows this endpoint of
course it's a long url
but you can see that this is running on
eks right and basically what changes is
is this
this kind of uh sequence this string
here right
and so people can try to attackers can
try to
scan the web and scan for uh
for any kind of open clusters that may
be running a vulnerable version or or
are like exposed by default right so
they want to look for that so yeah be
careful about that
uh
let's see
what else can we do here
let's go back to the
let me go back to the
cloud9 instance now
we're going to
download
the
download the
the exploit that we're going to use and
and
and use it right basically that
so let me grab that again
uh one second
okay so this is the url
of the exploit
let me share here with everyone
uh let me share the screen to you
sorry
okay
so yeah this is the github of the
exploit uh for the cve
right it affects uh
drupal 8.5 so
8.50 that's the one that we're using
um just a couple things here um before
you just kind of
uh
clone this right if you did clone
and and download this to your uh cloud9
you need to have ruby running installed
it should have that on your cloud nine
uh but it might require it might require
some like dependencies
which is a gem and we're i'm going to do
that with you together if if that
happens we just install the dependence
and we can run right so all we need
right now is the the
um
the load balancer url
uh from where your application is
running so let's go back there
let me go to the cloud9
sorry for the switching screens
and that's that's safer
okay
so yeah i'm going to need i'm going to
need that url right so just save that
for later uh save it like on notepad or
something because you're going to need
that for exploiting the cluster right
and you can you could do that from
any other uh uh
machine i'm just going to do the
uh through the cloud9 because it has
rubbing installed already and so it's
going to be easier for me to do that
so
let's see
i'm going to add that command xl to the
google doc
and that and
each clone
let me just
get out of this
okay
so i downloaded the the
exploit which is which is written in
rugby from github and now i can see that
here right
if i take a look at the the description
there basically i just need to run the
uh exploit and set up a target right a
url of the vulnerable drupal application
right and that's the url of the load
balancer that we have already right
so let's try to run that and see if that
works if it doesn't then we install the
dependency on uh on on robbie and should
work
uh as intended
so
yeah
basically drupal got on the rb
and
the url of my
load balancer and let's hope that works
yeah it's missing
yeah
in high line see it's missing a
dependence here cannot load file
highlighting part
i think it's sudo jam install
timeline
yeah so another command let me add that
to the google docs
i had that already there
okay
so it should work now that's
fingers let me just
clear this
fingers crossed
oops
not yet
why did it work maybe
let me open the terminal again just in
case
let's try one more time
see if that works if it doesn't i'll do
from my own machine no problem
hmm
mine sounds good
i'm getting installed okay
and
do not work oh maybe i need okay
i need the
the path probably
yeah uh
let's try
let me show you from my machine and then
i'll figure out how to solve that for
everyone
okay
let me
open a terminal here
and i'll share my screen to wait a
second
let's do the
rv yeah that works let me share my
screen better
sorry about that
this is the problem with live
workshops live demos
i'm gonna share my terminal okay
so yeah i
i went to the drupalgetom folder where
where the exploit is right
and because i have the ruby and the
dependencies already installed that
worked already here and basically i did
drupal get on the rb
and uh
with the url of my load balancer right
where my application is running right so
i think there is some issues with the
with the path of robbie um
and on the
cloud9 and we can we can figure out that
soon so let me show you what's what's
going on here
so the exploit is running right is
looking for for the drupal stuff it's
it's collecting so you can can get
information about my uh
my web application where it's running
and all the stuff and and i get a shell
here right
so
let's see um
with mine
see it's running as the user wrapped up
data
from the web server or where these um
this container is running right
um
let's see
see
i can see from the files already that
right
if i've if you've known drupal before
and know the file structure of drupal
you can see that there is a drupal uh
application running
and
yeah so this is already i'm inside where
are where i am right now with the
exploitation of of the the web
application um in a shell
inside the pod that's running the drupal
container right
um so what else can we do here right
what else can we uh um
can we see
how do i know that this is a a
a cluster like this is running on a
cluster right
so some things that
can help you understand
um where is it running uh um the
environment right it's checking the
environment variables right
if i do and i saw will see a lot of
environment variables right i don't want
all of them
so
let me do
f
grab
h i
cube
let me do that
oh now i see there is a lot of
kubernetes environment variables and you
can see them all together here and i can
see
um the the ip address of my uh
my api endpoint my api server right you
can see the port that's running right
and so there's a lot of information here
so these already it's telling me
that this this is running on a
kubernetes cluster right
let me just add that
and just add that to the sorry to the
google docs
um
and
iq
uh what other
what other things that i can do here
right i'm inside the kubernetes spot
what can i do what else can i do right
it's not just like it's it's running
inside a container right
and and
um but what else what information can i
have
another thing is
that
kubernetes stores
their
service account tokens on each on every
pod
and that account token is used for the
pod to talk to talk back to the api
server right and so
if i go to any they are all on the
always on the same location on the same
uh directory
so if i go here to
[Music]
run
secrets
i think i need to increase the font a
little bit
secrets
kubernetes.i yo
service account
see if it's going to show me i don't
know if it's because of my shell no
see
[Music]
yeah my channel is not great i should
probably reply a new show okay
yeah now i can see it here
i can see there's free files here right
the certificate
the namespace which is basically where
where the the
vulnerable web applications running and
the token right and these can be used
for me to talk to the api server
there is another tool that can be used
as well
which is also it helps
getting information about the
the containerized environment
and this tool is called in my container
we post this
going to post that to the google docs
secrets
and
step nine we're going to
look at the mi container
so what we're going to do is use
download this tool
to the
um to the file that i'm running
and execute it right
so let me
go back here
one second
uh
where is it
okay
um
yes let me share that
so
let's see if that's going to work i have
the command somewhere here
yes uh
yeah here
basically
actually i want this one
um
yes i posted that to the google docs so
that everyone can follow
i'm going to try to do that through my
shell
it worked so basically i went to the
folder i used curl to download the
binary of my container from the github
right
and yeah change it uh
permissions there and execute it right
so you can see this is the output this
is the output of the mi container tool
and it shows that the container runtime
is cube
it shows it has namespaces pid namespace
through username space false it shows if
there is some
if there are any protections your um
linux security module is activated like
app armor or uh
sc linux or saka right so you can see
that the app armor is unconfined there
is some capabilities that you can see
here as well
and also syscalls that are blocked so i
know that as as well
um so this gives me a lot of information
already right so um
it's it's good for an attacker to
kind of do like a recon
of the internal environment right
i can even see if i can curl
i think i can even curl from here
yeah see the same thing that i uh
nopes not gonna work um so the same
thing that i did on accessing the tab
um the url i'm doing that from inside
the pod and since those are environment
variables it's going through it's
getting those and getting the kubernetes
the cluster information right
so that's interesting another thing that
i can do is
you probably heard about the
api metadata right the instance metadata
from uh
from
cloud providers right they all have one
aws has one google has one and azure as
well
so from a compromised pod if that's
enabled you can reach the api metadata
as well
and i'm not gonna show my
i'm not gonna show my api keys but i can
show you some stuff that works let's see
where is it
one
second and
yeah let me get okay let me grab that
first let
and find my
my cheat sheet here sorry about that
where is it okay
all right okay i'm not finding it so
yeah uh i'll show the the api metadata
soon don't worry let me just
another thing that you can do is
you can from the pod that you're
compromised right you can talk to the
api server too
and as i said here like i i just did the
curve to the api server but if i do it
like
just slash api
you can see that i got 403
uh error as well right so i got
forbidden because i'm not sending any
kind of
uh authentication on any kind of token
it's not working but as i showed that
there is some
credentials some service account
information the certificates and the
tokens inside the pod that i compromised
i can use that
to
feed
feed into my car request to talk to the
api server
let's see
let's go back there
so i'm going to create two environment
variables here let's see if they work
i'm going to create the token
environment variable that's
using
getting the token from that directory
and the namespace as well getting the
namespace from from that directory and
now
now i'm going to do occur request but
with that per request i'm sending
the header authorization there token to
[Music]
to this
um
to this api right api server
ah
didn't work
maybe so
maybe it didn't work my
no
yeah
so i need to copy probably copy that
fine
so fine
yeah i'm going to show my cooking that's
fine we're almost i think we're
almost at the end now
you know we're gonna have time to cover
everything
but yeah let me try to run that again
instead of token i'm going to
send that here
right let's see
unauthorized
something changed
it's not working
okay
no problem there
it's fine
let's go back to
slides and i'll figure out what's going
on
okay
yeah
so let's see where we are right now
on the
slides
okay
so
we we've done the reconnaissance we
hope this light's not showing yet
someone adds lines
can you see these lines on my screen on
the screen
okay thank you
so yeah we did we did some internal
reconnaissance we inspected the
kubernetes environment we inspected the
environment variables right and and the
service account token we even used a
tool from outside right externally on
github downloaded that tool and executed
yes yes probably yeah there is some
probably some issues i'll check it out
soon i just don't want to
get stuck because we're almost
almost out of time and i want to
complete that but i'll
i'll take a look
we have more time
um
yeah so we did an overview of that right
there is many stuff that you can do here
on uh on your container
there is also a possibility of doing
because the way that this uh cluster is
configured
there is also a possibility of of doing
even like a post
post exploitation or persistence on the
cluster via a privilege escalation right
so there is this technique uh posted by
duffy cooley who works at uh isovent now
he used to work at apple
where he created he posted this uh kind
of command using cubectl to deploy a
a container
with with like the privileged container
that has access to the worker node right
um
and and on the left the right side here
is the kind of the structure of the json
format that he's using to override uh
that container that he's deploying
basically
what what's happening here and and he's
using like um hosts pid equals true
right he's using the the privilege
container true as well as the security
context he's um
he's doing a lot of
uh like different techniques to access
the the worker notes right
and and and that's one of the ways to to
do that and how that happens because i'm
allowed with with that service account
that we created which is uh um
which is giving me a lot of permissions
um
i can i can do that i can deploy a new
port and that's one of the one of the
ways of escalating privileges in a
kubernetes cluster is deploying a
privilege pod on that cluster if i have
missions to do so right of course i
would need to download and install cube
ctl on that pod first to run it and do
the privilege escalation and if we have
time we can try to do that as well
um that can be a little
kind of
tricky on managed clusters because some
the way that
uh
they they handle those privilege
escalations
but but yeah let's see if that works
soon
um
let me talk about the defenses before
before we do some other attacks
otherwise we're not gonna have time
right
how can i protect my cluster from
attackers right isn't kubernetes secured
by default and where do i start right um
basically we can see that kubernetes is
not secured by default by by many
reasons right
so let's take a look at some different
uh things that you can do to protect
your cluster
um
one of the basic things right is taking
a look at when you installed your
cluster and this this applies only to
unmanaged cluster where you have access
to the
to the control plane
um but one of the things is it's just
integrity monitoring right understanding
what files are created when you install
kubernetes uh the recommended ownership
and permissions and those are from the
cis kubernetes benchmark
and if if you have the integrity
monitoring set up on that
node
anything that changes on those files
either permissions or or ownership then
you should be alerted right something
suspicious is happening because usually
those don't change very often and those
are the recommended permissions the cis
kubernetes benchmark if you haven't
heard about it is like a
very extensive documentation with best
practices for setting up your cluster
on uh on kubernetes environments and
setting up playing
it correctly
there is also another kind of
a recent document that was published by
the nsa
called the
kubernetes hardening guidance i think
it was released a couple days ago so
it's pretty recent
but
as far as what i looked from the
document has um
has some issues right uh they talk about
spot security policies which is
something that we're going to talk about
soon but those are are being deprecated
or deprecated already
um they they don't mention much about
the
encrypting at cd right which is the
database of kubernetes they only have a
small example at the end
and they're using the docker files with
the tag latest which i don't think it's
a good idea using the latest stack and
that's something that you learn when
using containers
so yeah i think the the cis benchmark
document is a better one to use for
deploying your your cluster securely
um there's also the same thing with the
worker nodes right it's just different
files that are installed on the worker
nodes with with different ownerships and
that's from the recommendation from the
cas benchmark 1.51 there might be there
might be a newer version
but yeah take a look at that on the cis
benchmarks website
uh i talked about cuba system uh
namespaces already and and how they can
be uh um dangerous
like
as i said there is no
security boundary between namespaces
right so the cube system namespace which
is the main namespace of your kubernetes
cluster and it's like a inception right
kubernetes uses kubernetes to run
kubernetes um so all the the main pods
of your control plane
are on that coupe system namespace and
so if an attacker has access to those
then then it's it's likely that they can
do a lot of damage right
so be careful with that cube system
namespace you don't want to give access
to that namespace to every user on your
cluster and
the idea is that you can use our back
which is row based access control to
protect that namespace from being
accessed
i already talked about the api server
how that shouldn't be exposed and how we
can get some information from from the
api server and see if there is a
cluster running
right
um
yeah there is some information that you
can get running some comments if you
have access right in this case we don't
have access to the control plane right
shell access so
we can't see the configuration of the
api server but but yeah you can do that
if you do if it's an unmanaged cluster
um the cubelet right we talked about the
cubelet on the
architecture diagram it's the agent that
runs on each node of your cluster right
make sure that all containers are
burning in the pod
and uh two main security settings for
the cubelet is restricting cubelet
permissions and rotating the kubelet
certificates there is another thing
that uh there was a blog post that i
published earlier this year
uh talking about how attackers are using
a cubelet exploit to compromise
different clusters um
let me see if i do yeah
so the way that it works just as a a
quick overview and i'll post a link for
the blog here
and
it's not like i'm not trying to
promote anything is just that's a
technical blog on what we've analyzed
from a very famous uh
actor
the way that they do is they compromise
your environment and once inside your
environment they scan the your whole
entire network
for uh kubernetes clusters running
inside your network right and and they
specifically target the cubelet right
with a cubelet exploit that's been known
uh uh for a while okay
so take a look at that blog post if you
if you uh if the um
if they can share the link but but yeah
there is something that's happening on
uh on the y in the y out as well
as i said the
kubernetes benchmark is a guidance for
establishing secure configuration parser
for kubernetes
there is over 120 security checks for
your kubernetes cluster and it has been
developed by uh
cloud native and kubernetes security
professionals with much more experience
than than i have so rory mccoon from
aqua security and reason
liz rice from isovalent and many other
contributors
right and there's also specific ones for
eks and gke and i think there is one for
aks if not already
so because the way that the managed
services work they they are a little bit
different a little bit different than
the standard the unmanaged kubernetes so
yeah it's better to take a look at the
specific documentation for that and
that's i think that's the best uh
guidance out there for protecting your
kubernetes clusters
let's see
um if you don't wanna
if if you don't wanna run yeah and if
you don't want to check your benchmark
uh
manually right because it's a long
document it has a lot of tracks that you
need to do and the way that the
benchmark works is that you have um
you have
this the setting that's recommended okay
here's the setting
then it has how to check if the set if
that specific security setting is
properly set up
and then if not set up it has another
command for you to run to change that
setting to the proper
like configuration right so the
benchmark is a pdf file it has a lot of
stuff and like many uh
best practices configurations right but
it's very hard to do that manually and
especially if you have multiple clusters
that's gonna be properly probably
unfeasible for you to do right so uh
aqua security which is also a company
that that works with uh container
security uh
products they they provided they created
this tool called cubebench which is open
source and it's free for everyone to to
use that they check whether kubernetes
deploys securely or not right you can
validate it against your cisco bernat's
benchmark right and it's developed in
gold so it's open source and let me post
the link here
on
from github
to bench
i think there is
a dash
yeah
okay
uh so you can use that right so i i just
have an example here i don't know i
don't think we're gonna have time to run
cube bench on our own cluster but it's
very easy to do that and you can see an
example of an output here showing that
my cluster my uh has a lot of uh
vulnerabilities or or maybe like it
failed a lot of best practices from the
cis benchmarks right a lot of fails and
warnings and whatnot and it checks
everything from the master node the api
server the lcd and worker nodes as well
so it's interesting too
if you if you have closer running on
your organization and you want to see
um their status and then how they look
if they were configured or not it's a
good tool to to use and get a quick
overview of the uh if they're compliant
with some best practices from the
benchmark
um other stuff that you can do is use
some image scanning right
before you deploy your containers on
your cluster the idea is to
scan them for vulnerabilities or or
outdated dependencies and there's
different image scanning tools uh uh out
there i'm not going to talk about them
there there's different uh open source
and and and
enterprise ones uh all those here are
either free or open source so you can
take a look at those um there's even
like a native docker scan command that
you can use now which is based off a
snake
but yeah there is different others
okay now
you scan the container image and you
deploy that image on
your cluster right but how do i know
that someone has compromised my cluster
some compromised that container that
part right
that where that's when the the cloud
native runtime protection comes in right
and and probably if we had like a
tool such as like falco which is an open
source to as well that belongs to the
cncf now it was donated by cystic
and
we it probably would detect some of the
attacks and and the uh
shell that we got inside the container
right
so what falco does is falco parses linux
kernel c skulls at runtime right it's
using a technology called ebpf
extended berkeley packet filter which is
kind of a
technology that many uh
container security
companies are using
to create their own uh their own tools
their container security tools right and
it has a lot of visibility on the system
so you don't need to install agents and
and like
mass
mess with the container deploys
sometimes deploy side cars and all that
stuff
so falco is a real engine right it has a
easy and powerful powerful view room
engine you can create your own your own
rules for your vulnerabilities uh it
generates alerts based on the threat
detected i'm even using the um the falco
sidekick t-shirt today that i got as a
as a contributor to the falco project
and it detects unexpected behavior on a
cluster right and and the sidekick is
another project that runs with falcon it
gives you better kind of visibility
on on on what falco's seen right on the
logs and everything
um
here we're almost out of time but yeah
there is other settings that you can
apply to your pods right of course cpu
and memory is limiting resources to
avoid denial of service that's something
that you can do you can apply a security
context as well there's a different set
of of
uh
of settings that you can apply on a
security contest
some of those are are here
and as i said the the linux uh security
modules right there the kernel features
second by armor and selinux can also be
used to apply that to your kubernetes be
careful because docker has some default
uh profiles for those for app armor for
example and tecomp but when you use that
on kubernetes kubernetes doesn't inherit
that from docker so you need to apply
that as well
and
yeah pod security policies is something
that got deprecated it's not being used
anymore
it was a way to apply the security
context as a whole on a cluster level
and there is new features now new tools
that you can use some alternatives such
as oppa gatekeeper or kyverno which are
open source tools as well that you can
create policies as code to say okay only
deploy this container if it doesn't have
any high or critical vulnerabilities
right
uh there is a uh um
a new pod security now i forgot to
update the slide but there is a new part
security it's not psp but there's a new
version that's being implemented by the
kubernetes see seek off and seek
security team right there is a it has
been approved already
um yeah i won't have time to talk over
our back and at cd network policy and
yeah let me finish with the basics and
i'll share those lives with you and if
when i talk about those other things our
back
uh network policies and audit logs we
can chat later i i'm available on on
social media twitter or linkedin
but basically what i wanted to give you
here is just an overview of kubernetes
an overview of the the security issues
and also an overview of how to secure
your cluster right it doesn't mean to be
it's not an extensive uh workshop it's
not an in-depth workshop so keep that in
mind
um the so i just want to leave you with
with a few a few basic rules with
kubernetes right uh first thing is
update your environment version early
and often as i said there is new
versions being released uh
very frequently now three times a year
and the version 1.22 is the most
up-to-date one uh recently that was
released recently
don't use the cluster admin user for
your daily work right
treat it like roots right so the reason
being why we got to uh exploit that
vulnerability and then uh get shell on
the quad and then do bad stuff on the
cluster was because the service account
of that pod had too many permissions
right you can even deploy new pods and
scale privileges and all that stuff
so be aware of that
if you can use manage kubernetes
services such as aks eks or dke because
they have better security defaults and
you don't need to worry about the
control plane because they care they
care they take care of that for you
and as i said check out the cis
kubernetes benchmark for more security
best practices
uh i hope you enjoyed this workshop
there's some references here and i can
share the uh the whole
command list that i use and you can try
to replicate that on your environment
and i'm sorry if some of the commands
didn't work but i'll take a look and
make sure that's fixed and before i
share with you so yeah thank you