thanks
thanks and thanks
everyone for joining i am your valencia
i'm a security researcher at palo alto
networks uh well when me and the rest of
my team we look into the security of the
cloud and containers uh specifically and
i'm very excited today to be presenting
you husi
which is a tool i wrote it's actually a
container image that can be used to look
under the hood of container as a service
platforms so
let's start off with by going over the
agenda so we'll start off by talking
about what our container as a service
platforms are and why are they
interesting
we'll then talk about hussy what it does
and how it does it
then i'll show a quick demo and will go
over into the low levels of the
implementations
and finally i'll discuss some of the
findings that i was able to
see with hussy
and we'll wrap it up with some questions
so let's start with container as a
service so you probably heard about some
of the
containers and service platforms out
there you have aws far gate azure
container instances google cloud run and
then more recently ibm code engine
and those platforms are really a way to
run serverless containers the way for
you to take container
just upload it to the cloud and have it
process events scale up and scale down
according to demand and it's really a
way to you know run your services
without managing on or worrying about
the underlying infrastructures uh
managing the underlying servers or
managing the underlying nodes
which is a really great way uh
to you know
it's really a way to improve your ease
of use of those platforms
but
on the other hand without management you
really have no visibility into how your
containers run on the cloud
and
that brings the question can you really
trust container as a service platforms
with your containers
so
what could be an issue well container as
a service platform like you know every
public cloud offering
it runs your workloads alongside other
customers workloads right so you need to
trust
the platform to segregate your workloads
from other customers workloads because
there may be a malicious customer but
with container as a service offerings
you really have almost no visibility
into how your container run you just
upload your container image it receives
some you know http endpoint
but without beside that you don't know
you know how is the cloud provided how
is the cloud provider protecting
your workloads
and that's really what hussey
tries to do so the motivation is really
to gain visibility into how container as
a service platforms run our containers
and the way using russia does it it's
really a container image that upon
execution exfiltrates the underlying
host container runtime to a remote
server and it actually sends the actual
binary and you might see something
pretty interesting that you have a
container that reads a file from the
host the container runtime so we'll get
to how hussey does that in just a bit
but let's start off with by seeing a
demo
so
what i have here
is a file server
that will receive the runtime and i'm
going
to use ibm code engine which is a class
platform
so first i'm
creating a job template
and it uses the hussy container image as
you can see
and now
i'm going to deploy the juse container
to ibm cloud and have it run it in the
container as a service platform
so
this is a live demo so hopefully it will
work
okay so the container is just now
uploaded and you can see that the file
server
receives the cell some file
and we just wait up it finished and if
we take a look at the file that we got
we can see that we got
ransi
which is the industry standard container
runtime so this is the version that is
used
in ibm code engine
and
that's the actual binary
you can see that it's a go binary so
it's pretty large
so that's very very cool
in my opinion at least
and
let's continue
and talk about what we expect to find
with this tool just before we talk about
the implementation so like we saw with
ibm code engine
and we can expect a lot of rantsy right
ranci is the industry standard container
runtime so we can expect to see a lot of
it
we might see some old and vulnerable
rancid version
we might see some custom changes that
cloud provider made to valena to vanilla
rancid and we might even see other
runtimes maybe proprietary runtimes you
really can't know
and
let's continue we talk to talk about how
is who's implemented so hussy basically
follows two steps the first step is to
trick the runtime to execute itself
inside the container and the second step
is to have someone in the container
reads the random binary from proc pid
exit and send it to a remote server
so if that sounds a little complex right
now let's break it down step by step
so the first step is to trick the
runtime into executing it into executing
itself inside the container
so how do we do it well it's pretty
simple that's the docker file of the
container image all we do is set the
entry point to slash proc slash self
slash exit
and slash blocks
self exit
is a
unique
magic link
in linux
so
whatever process access that path you
will actually see a link
to the binary that is running so you can
see in the image below when elast tries
to access proxy factse he sees that proc
self exe points to ls but when the
rhytlic binary tries to access spokes
self exam proxy fx points to lead link
so when we tell the run time
to please execute prox effects as the
container entry point
we actually make it execute itself
inside our container
so
that's the first step right we we have a
way to trick the runtime into to execute
itself inside the container and the
second step is to have someone in the
container read that process that runtime
process
proc pid exact and send it to a remote
server
so
that's how it looks right we have the
runtime we tricked it to execute itself
inside the container and then our
process inside the container reads the
runtime executable
through slash clock slash runtime pid
slash excel and sends it to our remote
server right
so
it's actually not that simple
because containers can only have one
entry point they are made to run one
process
so if we set the entry point to proxy
effects and make the runtime execute
itself inside the container
we really have no way to spawn our
process that is supposed to send the
runtime right we have no way to spawn
the red process in this image
so
how can we overcome this
well the solution is to actually replace
the dynamic linker inside the container
image
so we assume the runtime is dynamically
linked and in linux when a dynamically
linked executable starts running the
kernel
loads the dynamic linker
to the process memory
of the dynamically linked executable
and
transfers execution to the dynamic
linker so the dynamic linker can load
the necessary libraries that the process
needs
so the kernel actually searches for the
dynamic linker
from the
in the root file system of the process
and because we trick the runtime to run
inside our container it will actually
look take the container dynamic linker
so
in order to make this happen we actually
only add one line to the
docker file
and that's the second line here we take
we create a fake dynamic link
and use it and replace the dynamic link
in the container with it
so
that's actually
a pretty nice trick but we need to a way
to actually create a dynamic linker and
when i was approaching that this task i
really had no idea how to do that
so
as we've said
the kernel loads dynamic clinical to
process memory to load uh shared
libraries right
so first of all
the dynamic network the dynamic linker
must be statically linked right it
cannot have dependencies on other
libraries because the dynamic link is
the one that is responsible
that is responsible for loading
libraries
second you know second of all it needs
to be written in c or maybe cpp i
haven't tried cpp because
languages with complex runtimes like
golang
they don't expect to run in the context
which
the dynamic linker runs
so they can cause some issues
and
at my first story i tried compiling with
the gillipc
and it turns out that the
a feature of glibc called fred local
storage
actually causes some problem but if you
compile with muscle lipsy then
everything works fine so that's how i
created a my custom dynamic linker
and to wrap things up that's how the
demo we saw worked so ibm
cloud took our who's the image the entry
point was set to slash proxy self slash
exit
and when
the container has one was run
our fake dynamic linker was loaded to
the runtime process and sent the runtime
binary to our remote server
so
we have a way
uh
to
read the binary the container runtime
binary from the host but there is also
an assumption in this
in this way which is that the run time
is dynamically linked right because if
the runtime is statically linked
the
linker isn't loaded into the memory
because it isn't needed so
in this scenario
we need some way to have someone in the
container read the runtime and send it
to us right
so i actually gave this some thought and
eventually the solution i came up with
is to read the runtime whenever is there
is an exec like a docker x0 or q-cup
cube catalex if you're familiar with it
whenever there is an exec into the jose
container
this flavor of hussie will send the
runtime and most containers of service
platforms they support a docker exec
like experience to the uploaded
container so if it's really not clear
yet let's see the docker file so
we now have our binary just run at the
entry point right we have a binary that
knows how to
upload the runtime
and
when we run the container that's what
you have right just that process running
inside the house container but then when
someone docker exit inside the container
we actually ask it to exec slash proxy
exam which makes the runtime process
appear inside the container right we
trick it to execute itself inside the
container
and then our upload runtime which runs
as pid 1
waits for the runtime to jump in and
whenever it jumps in it sends it to
remote server so
let's see a demo
so first off we need a file server right
and because the file server runs on our
local machine we need to see how local
container
see
the ip of the local machine and that's
really the idea of the default gateway
so i'm just going to grab
that and now we're going to
run our container
and that's the ip that you can see there
that we earlier saw is the ip of the
host from the context of the container
so again you can see this is the wait
for exact flavor of hussey
and
hussein now waits
runs as the pid 1 of the container and
waits for the run time to execute
so that's what we're going to do
and the name of the container
up so
and we're going to ask you to execute
blog self-exam
and you can see that
hussy
noticed the runtime
exiting into the container and uploaded
the runtime over here
to the file server
and if we try to see what file we
actually got
we can see that again
it's the runtime of the underlying host
which also in my
linux vm is run c
so
that's how you solve
study that's how i solve
statically linked runtimes uh having a
process in the container wait for docker
exec into the container
and that's really how hoosie is
implemented you have this flavor which
is for statically linked runtime
and you have
the dynamic lincoln flavor which is for
a dynamically linked runtimes and when i
took huci to a real
container as a service platforms i
actually saw both i saw
dynamically in current time and i also
saw a statically linked runtimes
so with that let's talk about
the findings so far so as i expected in
the beginning uh i saw a lot a lot of
ransi which is the really the industry
standard contender anton but i also saw
old and vulnerable rancid versions which
led to some
quite interesting research that they yet
can't disclose
uh well what about custom changes to
vanilla rancid
well
i really haven't had the time to take a
deeper look into that but from a shallow
look
i didn't see any custom changes but
i'm not
like i haven't looked into that
to the fullest because i was
more preoccupied with a
vulnerable rancid version
and finally i didn't see any other
runtime other than run c
so
what's next well you can use hussie to
poke at the underlying contender run
time of whatever container as a service
platforms you lack you can get
visibility into how your containers run
on the platform and maybe you'll find
some security issue with the runtime and
possibly get a bounty and help how they
contain other service platforms so right
after this talk i'm gonna make this link
publicly accessible
and i think i'm ready for
four questions