uh so uh hello everyone uh welcome and
thank you for joining me
you're about to hear a story about uh
how a single mission condition in uh if
statement
enabled one of the easiest container
escape of recent years
so let's just get started
okay so who am i
my name is daniel prismant and i'm a
senior security researcher at the palo
alto networks
i usually do reverse engineering in
windows platform for years i joined the
containers focus team that mostly did
like open source vulnerability research
on cloud products
our team are full of security research
focused on finding vulnerabilities on
the cloud and i was the only windows guy
in the team so i decided to focus on
windows containers
okay so i'll go over the agenda real
quick we will start things off with the
demo to show you guys the impact of this
issue
i will then cover containers in general
and windows containers in particular i
will also go over some of the
fundamentals that one needs to know in
order to fully understand the issue i'll
be covered
i will cover the issue itself what made
it possible and after that we will cover
how cloud providers are affected by this
issue
and lastly we'll briefly talk about
malware that took advantage of this
issue to gain some
free processing power
and when we're finished i will try
answering some of the questions you
might have
okay
uh
okay let's let's get started with uh
with the demo
i will try to explain it while it runs
okay so first of all i'm deploying a
kubernetes yaml i took this yaml from
the kubernetes official website i didn't
change anything the only thing i'm doing
here is applying a deployment
when stating that the node i'm choosing
is windows and not linux
so we created the it created two
deployments for me
and i'm checking that they are actually
running and at this point i'm executing
into the container i will explain more
uh
in the next in the next slide
for now i'm mimicking an attacker
actually gaining
execution inside the container
so as you can see i'm inside the windows
container i move to c and i'm making a
directory for all my files for the
breakup
i prepared them beforehand i will
explain more in
the rest of the talk downloading all the
tools i created to to make breakout
basically it's a dll which carries the
logic itself and injector which inject
the dll into one of the
containers processes
as you can see i use the injector it
injected itself into the main container
process
and i also made a log file
which will log
what happened from inside that process i
created a symbolic link
to the host
how it may how it trustable i will
explain later
and for now i'm only doing a simple copy
file from the host using the symbolic
link i created
and i'm copying a file from the host to
the container so i will have access to
it from inside our container
the file i'm copying is the config file
of the node it's the config file that
the node
uses in order to to talk with the api
master server of kubernetes
as you can see it okay it contains all
all the certificates and keys
uh that the nodes that the nodes have
and i'm going to use
this configure to impersonate the node
in front of the kubernetes master api
server as you can see i'm pointing that
i want to use a cube config and not the
default config
and then i can actually query the api
master server and get information for
example i see all the running pods as i
did from outside the container
and i have i have a yaml here so a linux
container which opens a reverse shell
also this container is privileged and my
my goal here is to create a new
container privileged one a linux one so
we so kubernetes will create this
container
inside a linux machine and not windows
because i already have access to the
windows machine i want access to a linux
machine too so i'm creating a deployment
using the
the
the config file i stole from the node
and i created a new
privileged container escape
that container contains a reverse shell
so i'm gonna i'm going to connect to
that shell for my windows machine
using netcat
as you can see i'm inside the linux the
privileged linux container and i have
full access to the node 2 to the to the
linux node because the container is
privileged and at this point in this
cluster i had only two two nodes i had a
linux node in had a windows zone i used
the windows i used the windows note to
to break out and create
a privileged container so i had access
to both the windows using the
vulnerability and linux using using the
config file
okay
okay but let's explain let's explain the
demo a bit more
what does it all mean okay so i will go
over the demo step by step because i
didn't explain the issue yet but
nevertheless let's go over their main
idea here so i created a new deployment
and note i specifically stated in the
ammo that i want a windows node i use
the example yaml from kubernetes website
with default configuration of the
cluster if you google how to create a
windows windows container in kubernetes
that what that what will find
and i didn't change anything remember
that because it is important for later
so i created the i created the
deployment and issued an exit command
into it
this part might confuse you i use the
exit command to mimic an attacker
gaining execution into the cluster
so for example if your cluster
hosts a web server with vulnerability
and then attacker managed to use this
vulnerability to gain access to your
cluster we will end up in the same step
here because the attacker will gain
execution and i just use exactly i skip
the part when we're an attacker find a
vulnerability in your container and get
execution to the container
after that i use some symbolic link
tricks to gain access to the host file
system and in this case the host is the
kubernetes node
okay once i gain access to the nodes
file system i use the node kubernetes
config file to impersonate the node to
the master api server in order to in
order to create
a privileged linux container
because i created the linux container
the api server created this container on
a linux node
and my container packed a reverse shell
and once it was up i simply connected to
the shell gaining access to the linux
mode as well
and at this point
i had full access to both the linux and
the windows pallet of the cluster and i
could do with the cluster
anything i wanted i could create new
deployments to mine my cryptocurrency or
i could steal all the data in the data
cluster or anything else and the
possibilities from hero and
okay so what are containers and i know
this is the cloud event and most people
here probably know what the container is
but i will go over it quickly just in
case
a container is basically an operating
system based virtual machine meaning it
runs inside operating system with the
same kernel as the operating system it
uses operating system features to
isolate the virtual machine from the
rest of the system
unlike actual virtual machines which use
a completely separated operations
operating system and
based on hardware they they base their
virtualization hardware instead of
software
and containers can run anything but the
desired container must match the
operating system version so for example
you won't be able to run windows
containers on the linux machine
as you saw in the in the demo i had to
create a windows node in order to run
windows containers
and one of the most important features
of container is that they pack all the
necessary files to run the application
so for instance if you have a special
application with
special dependencies you can pack it all
in a lightweight container and send it
to the end user
okay so let's go over uh the differences
what is the difference between a
container and virtual machines real
quick the main difference is that
containers rely on operating system to
make it it's isolation while virtual
machines rely on hardware
so virtual machines virtualize
everything including the kernel while
containers run on the same kernel as
their host
and because of that containers are much
more portable and efficient
a container image can be as small as few
kilobytes while virtual machine images
will usually be at least few gigabytes
in size because they have to pack the
kernel too
all of that comes with a price of course
containers are much less secure than a
virtual machine if you look if you look
in google for container escape you will
probably find much more results than the
virtual machine escapes
okay so uh on the left side you can see
a virtual machine infrastructure with
the hypervisor managing all the machines
each machine has a separate operating
system
and on the right is an infrastructure of
a docking machine hosting a few
different applications in containers
each application is inside a separate
container but they are all running on
the same host operating system with the
same camera
okay so
what needs to be contained let's dive
a bit deeper into the internals
uh and note we are only talking about
linux for now and we'll talk about
linters yet later
well obviously you would want to limit
the containers access to a resource such
as cpu ram network bandwidth list
bandwidth and such
this is done using a feature called see
groups
and these features allow us to
limit resource
usage for a group of processors
obviously what you wouldn't want your
container to
to get all the cpu all the disk
from your host so you can limit that but
you would also want to limit the
visibility the containers have if we
would only limit resources nothing would
stop a malicious container from just
changing its own resource limitation
and for that we would also want to limit
the container's visibility
and to some of the host objects such as
processes network interfaces users
mounts such
things like that and this is done using
a feature called namespace
all of that is in linux
okay but this is talk about windows
containers so let's move forward to
windows
in order to get a good solution for
containers and windows uh the same
requirements i talked about in the last
slide need to be implemented here
and luckily windows had a solution for
that resource limitation for years it's
called job objects and they do pretty
much the same thing as c groups in linux
and so there's nothing too interesting
about them but it's it is important to
know that the feature existed for years
it's not new and it's not it it's like
the windows version of c groups
okay but what about visibility isolation
and until recently windows didn't
actually have a solution for this and
that's why windows containers
didn't exist until like a few years ago
uh but luckily a few years ago microsoft
came up with a feature called server
silo which directly provides the missing
features that were necessary in order to
create a container solution
server silos provide everything that
namespaces provide in linux
they isolate the object manager the
registry networking devices and
basically any named object that process
can access
and remember the name name the object
part that's the important thing
but also here the other type of windows
containers the hyper-v containers and
servo silo
on the right you can see
how a hyper recontainer architecture
looks
not much different than a regular
virtual machine actually each container
has its own lightweight kernel
basically it's the microsoft version of
vm it's not what we are here to talk
about today because it's a vm and on the
left side you can see the more
traditional meaning of containers one
kernel with few containers for different
applications
note the system processes are also
shared with the containers
and this is done using cell silos
as you can see all the containers are
using the same the same kernel system
same system processes
okay there is one more thing that need
to uh to know before understanding like
how it all works
[Music]
called root directory object
which is a key feature in windows
operating system without anything to do
with containers
uh
without getting too deep about this
mechanism it's suffice to say that all
application visible named objects such
as file registration events mutex rpcs
stuff like that anything that is named
are hosted in a root namespace which
allows applications to create locate
share these objects among themselves
take a look at this screenshot from the
win object application by sysinternals
it shows the root directory object
perfectly as you can see on the left
there are many directories under the
wood the root object and under global
question what question mark
there are tons of symbolic links
including the sea letter symbolic link
so yeah when you are accessing the c
drive in your windows machine it is in
fact a symbolic link to the actual
device
there are many more symbolic links over
there but the
the one most important to us is indeed
ceiling
and which points to the actual file
system device which in this case is how
is volume 3.
in this case what you can see
that would direct to the object of a
host it's not with a actual object of a
container or several style
it the wood directory object of the host
now note it can also be a virtual
machine but it's uh it's it has its own
camera so this screenshot is from a
machine with its own account and not a
container
okay
and this screenshot however you can see
what the containers look like in the
wood directory logic notice on the left
there is a stylus directory
every server silo container will have
its own subdirectory under this
directory
with the name being the server silo id
so in this particular case i had a
single container with a cell a silo id
of 804
and as you can see the silent directory
is almost identical to the host
directory
this is because windows tries to
virtualize a mini operating system as
accurately as possible
and so most of the symbolic links that
the host have the container have too
with different destinations of course
because we don't want the container to
point to the same devices as the host
we will discuss the server side of the
directory object further later
so let's move over to an actual example
of how a root directory object is used
in a simple create file code
the create file api receives the files
path and returns a handle to that file
the process can read write or do
basically any action with the handle
depending on the permissions the process
asked when calling the create file
function
and in this example we're going to
create file on a file in the c drive
named secret dot text
as i said earlier note that the c part
is just a symbol in clink as you can see
in the screen scheduled below you can
see the c it pointing to device hardest
country
um
okay
so uh c is first converted to something
that kernel can actually
query it is converted to the wood
directory object
form of this path
this is done in user mode before the
call arrives to the kernel
and after that in the kernel the kernel
query the global c path in the wood
directory object and receive the
destination of that symbolic link
we then queried that that destination
received from the symbolic link still
under the directory object
but this time the pulsing ends with
with an actual device
okay
[Music]
so at this point having an actual device
the kernel follows the request to the
device driver which in this case is the
file system driver and from here the
file system drivers take the
execution and that part is less
important for us because we're not here
to talk about
the file system driver and remember that
part where the kernel received the
symbolic link target and queried that
under the root directory object too
until it found an actual device it will
be important for us later
uh also remember that uh this query was
an example of a query from the host and
not from inside the container it is
slightly different when the calls come
from inside the container we'll cover
this in a minute
okay so before giving an example of a
file access form inside a container
let's discuss
how the system decides that the call
comes from inside the container actually
as you can see on the left
the it's a schedule for mida where i
search for the keyword silo server
there are plenty of functions that can
decide if a code comes from a server
side or not
and the kernel uses different functions
in different scenarios uh
this screenshot is from the enter
scanner binary so it's like the binary
that represent the kind of windows
um
in our specific case the kernel decides
a call comes from a container if it if
it decided it queries that the path we
saw before the global c path
relative to uh to the server side of
subdirectory inside a wood directory
object
and
this will happen to every axis of a
named object such as files so every
every access of a named object from
inside
server side meaning inside the container
will be queried relative to the silos
directory and then the silos id in this
case 804.
in this screenshot formida you can see
the branch in the kernel where it is
decided if path will be created to the
actual wood directory object
and not to the server style subdirectory
in the root directory object and as you
can see this is done using a ps get
terminal and silocontext
candle function which just returns the
context of the silo
okay
okay let's go over an example of
accessing a file from a container
as before we are accessing a file named
secret text under the cd drive so
exactly as before but this time from
inside a container
skipping the user mode part or the api
where the api adds the global part uh
before the c letter and jumping straight
to the camera part because we already
covered that
um
so the kernel calls
ps get permanent style context as you
saw
and you retrieve the style of context
and not null if you try to if the
knowledge you find the last picture it
will be queried
uh
towards the
the root director of the real one of the
host and not the style so it takes the
branch of the server silo in the kernel
code
and it created the relevant directory
under the subdirectory of the silo in
the wood directory object until it finds
the system voluntary
and
okay
but
this time
the system bowling gleam points to a
virtual hard disk device
under the side of subdirectory in the
directory object note the vhd before
before
in the name under the device before it
was just how this volume 3 in this in
this case is the vhd which represents
the virtual hovers
okay but take a look at the screenshot
and this is the interesting part
that virtual hard disk device isn't a
device at all it is a symbolic link too
which points to itself a symbolic link
and we end up in an infinite loop
because we have a symbolic link that
points to itself in the same name which
is to a symbolic link
uh
and this was the point at my research
when i realized there is something i'm
missing because the request to the file
was successful and i was able to read
and write to the file in my container so
it was obviously working but on the
other hand it looks like we should have
it stuck in an infinite loop
so what's going on here
uh
let's go over what the requirement is in
this case every container and it of
course is relevant to what i just said
about the infinite symbol
every container either in linux or in
windows needs to be able to communicate
with some of the host's devices it can
be a screen to show output or a network
device or the file system
so in the way windows works some of the
symbolic links under the server style
subdirectory
must point to a device in the host or
directory object otherwise the container
won't have any access to anything for
example the virtual file system is
eventually just a path in the host file
system
and the container must have access to
that path at least so it must have
access to the host file system device
and
we're getting closer to the actual
escape i promise
the way it is done in windows is that
processors with the right permissions
can set a symbol in clink as a global a
global symbolic well symbolic thing will
always be looked at relative to the
hosts or directory object regardless of
the process that is trying to pulse it
is inside a container
so when a container is created docker
sets some of the containers symbolic
links as global and that's why the
container can maintain some sort of
communication with the host
it
it pulses global symbolic links
and the result is devices on the host
and note that non-global symbolic
headaches like regular symbolics are
passed straight into the silos or
directory object as we discussed earlier
okay so in this screenshot again for
nida you can clearly see the condition
uh eax hold one of the symbolic links
parameters the one that holds the
symbolic link if it is global or not
and if it is the execution will take the
right branch and will retrieve the root
directory object from a global variable
in the kernel and if not
it will take the left branch and then we
will have to get
the style of context first as you can
see the left branch calls psg permanent
style of context so it makes sense
so if the symbolic link is global
the link will be passed from here
instead of from here
just just a visualization of that
okay let's move let's move to the house
how can we exploit the global symbolic
link feature to break out of the
container
so the function that is in charge of
making symbolic links global is the
undocumented empty set information
symbolic function
and after reversing it in order to
understand what parameter that function
expects i discovered that it requires
setcp privilege permissions in order to
make the link global
and sadly the regular container user
doesn't have those permissions so if you
are for example hosting uh
you're hosting a
web server and someone managed to to
abuse their vulnerability in it and
getting
container
and gain execution inside your container
that's process that web server process
in the context the attacker running in
we won't have that tcp privilege to
create a global symbol click
okay but lucky for us there is another
process in the container scope
meaning it is visible to the containers
user even without tcp tcp privileges
called cxxservice.exe
and it is the main process of the
container it's the process that spawns
all the other processes
and this process has in fact sapcb
coverage
and lucky for us again by default the
normal container user is administrator
so if you are running a web server i'm
coming back to the web server example if
you're running a web server the website
is running as administrator by default
take a look at this schedule from
and as you can see there is a check for
sctcb privilege before moving forward
with the execution in making a symbolic
and global and that screenshot is from
the anti-set information symbolic link
function that i mentioned earlier
so let's go over the escape plan
first we impersonate cxx service to gain
its
tcp privileges
there are numerous ways to do that
such as thread impersonation or dll
injection there are probably many other
ways that we didn't mention or didn't
research
but in the demo i use dll injector and
it did injection from there in the last
year when i researched it further i
moved to a thread impersonation
and there are probably many other ways
so after that we create a regular
symbolic link which at this point is not
global and points to our local
containerized cell right
then we call ntc information symbol in
qiling with our newly created and
symbolic links to make it global and
remember that we do that after we
already impersonate the cxx service
so at this point we have full access to
the host c drive
by using our local x drive because we
created the symbolic link x to c and
make it global so now it points to the
host
and
we are done here and the possibilities
are endless at this point we have full
access to the post
file system as you saw in the demo i was
able to copy
the config file of the node which is not
inside a container it's inside it's
inside the node
and i'm only talking here about the file
system but this can be used for the
registry itself and basically any named
object
the file system was just the easiest to
to export and demonstrate
uh okay
but how cloud providers are affected by
this
uh well the trivial thing is any
kubernetes cluster with a windows node
and attacker that gains access to the
container can just break out of the host
and possibly spread from there exactly
as i did in the demo
so it's depend on the
cluster how the cluster is configured
uh but said attacker will at least be
able to control to control every
container that the specific node is
compromised
that the com the specific node hosts so
even if the kubernetes cluster is
configured well and
unlike in my demo
one node can't access other nodes
the attacker will at least have access
to that node with all the containers
this snow host
okay so as i demonstrated in my demo an
attacker could possibly spread in the
rest of the cluster as well so not just
his node but other node as well
and another possibility in the whole
container as a service
service that's going on lately imagine a
cloud provider offering windows
containers as a service and not part of
an entire cluster
an attacker can host a malicious
container and break out of his own
container and gain access to other
customers private containers because
they are running on the same host
because the cloud provider won't create
a new host with a new camera for each
customer
so that's a possibility i didn't i
didn't check it yet i didn't exploit it
but
this can happen
uh okay let's move to an actual example
of how this vulnerability
could be used in the wild by attackers
uh
so a few months ago i discovered
cytoscape which is a malware that was
using the this vulnerability to gain
access to your kubernetes cluster
uh a target windows containers
specifically
in the in its case execution if it
detected that the container or the host
isn't a windows container it just
exited
and he used this exactly global symbolic
symbolic link issue to spread in the
rest of the kubernetes cluster it
basically opened the bag though to its
cnc and just waited for command
and as part of my research i actually
discovered it had active victims each
one being a kubernetes cluster with the
possibility of a huge amount of
processing power i didn't have any way
to actually check all right daniel uh
just just jumping in here you have five
minutes left and i actually have a
question for you if you'd like to answer
it
um about i'm about to finish just a
moment okay okay no problem
uh okay so as as i was saying the the
amount of processing power that the this
kubernetes clusters had is unknown it
could be huge clusters with huge amount
of processors
uh
okay but
but what can you do about it well there
are a few things first of all in order
for an attacker to even be in a position
to use this issue if else needs to gain
access to the to your container and most
of the time this happens to an outdated
application in your containers or a
misconfigured container manager
so keep your applications up to date
even if they are inside a container
this is relevant to both linux and
windows actually
and next this is the this is only
relevant to windows i advise you to run
your container as a container user
instead of administrator it will be much
harder to pull off something like that
without
administrator permissions and kubernetes
supports that as well there is a special
variable you can set in your yaml to run
your applications user even in
kubernetes and third update your windows
host this is mostly relevant if you are
running your cloud environment on your
own
and if you are using one of the cloud
providers they usually keep the host
updated
and last but not least configure
kubernetes properly for example there is
no reason the nodes user will be a
cluster administrator like in my demo
nodes shouldn't be able to create
deployments on other nodes
uh and real quick because we are running
off time a few weeks ago microsoft
actually patched
this anti-symbolic anti-semitic
information metabolic link function
and the patch is extremely easy to
understand and spread forward as you can
see
any call to anticipate information's
evolving from a thread inside a
container will be blocked with status
builder is not held at all
and this is done using the ps's current
red server silo function
which as its name suggests checks
whether the current thread is associated
with the process inside the server silo
on the right you can see how the
function looks after the patch there is
simply a call and there is a branch
there which checks if the calls come
from the cell asylum
um
so what's next i will probably start
with checking microsoft's patch maybe
the function the use
maybe ps get going threads in server
side is broken
and i can still break out i will
probably also look for other ways to
abuse symbolic links to break out of the
container and we'll also search for
other access points to the host that are
not specifically related to symbolic
links
and okay let's go over to questions