- Hi, and welcome to our talk.
My name is Guillaume Fournier,
I'm a security engineer at Datadog.
And today Sylvain and I are going to present the rootkits
that we implemented using eBPF.
If you don't know what eBPF is, don't worry,
we are going to present this technology
and tell you everything you need to know
in order to understand the talk.
So let's start with a few words about us.
We are the cloud workload security team.
We usually use eBPF for good.
And our goal is to detect threats at runtime.
Everything we do is added to the Datadog engine,
which is an open source project.
So feel free to check it out if you are interested.
That being said, for DEFCON,
we decided to use everything we knew about eBPF
to build the ultimate rootkits.
So, as I said before, we are going to start the talk
with a brief introduction to eBPF.
Then Sylvain will take it over
to talk about how we implemented obfuscation
and persistent access in our rootkits.
After that I will come back
to present the command-and-control feature
along with some data exfiltration examples,
and then I will talk about the network discovery,
and RASP bypass features of the rootkits.
And finally Sylvain will present a few detection
and mitigation strategies that you can follow
to detect rootkits such as ours.
All right, so let's start with eBPF.
EBPF stands for extended Berkeley Packet Filter.
It is a set of technologies that can run sandbox programs
in the Linux kernel without changing the kernel source code
or having to load kernel modules.
It was initially designed for nitro packet processing,
but many new use cases were progressively added.
So for example, we can now use eBPF
to do kernel performance tracing
along with network security and runtime security in general.
So how does it work?
So eBPF is simply a two-step process.
First, you have to load your eBPF programs
in the Linux kernel and then you need to tell the kernel
how to trigger your programs.
So let's have a look at the first step.
EBPF programs are written in C.
So it's not exactly C, it's more like a subset of C
because of many restrictions that eBPF has to follow.
But I'm gonna talk about this later.
So once you have your C program,
you can use LLVM to generate eBPF bytecode,
which you can then load into the kernel
using the BPF syscall.
EBPF programs are really made of two different things,
eBPF maps and the actual program.
So there are a lot of different types of eBPF maps,
but all you need to know
is that they are the only way to persist data
generated by your eBPF programs.
Similarly there are a lot of different program types
and each program type has its own use case.
However, regardless of the program type,
each program has to go through the same
following two phases.
So the first one is the verifier step.
So I will talk about this later, but for now,
just know that this ensures that your program is valid.
And second, your eBPF bytecode
will be converted into machine code
by a just-in-time compiler.
And when those two phases succeed,
your program is ready to be executed.
Step two is attaching eBPF programs.
So in other words, this is when you tell the kernel
how to trigger your program.
So there are many different program types
and I can't present them all,
but I'm just going to talk about four of them.
So for example, you can use kprobe
to trigger any BPF program
whenever a specific symbol in the kernel is called.
Tracepoints are similar to kprobes,
but the hook points on which they can be attached
have to be declared manually by the kernel developers.
Those two programs require an older syscall
in order to be attached.
And this is called as the perf_ event_ open syscall.
So the other two program types I wanted to talk about
are TC classifiers, so it's sched_cls and XDP programs.
So those program types can be used to do packet processing.
So whenever some network traffic is detected
at the host level or at a specific network interface level,
those two require the netlink to be attached.
And the only thing to remember here
is that each program type has its own set up
and does not require a different level of access.
Another very important fact about eBPF
is that the BPF maps can be shared
between different programs
regardless of their program types.
All right, so the eBPF verifier.
So the verifier is used to ensure that eBPF programs
will finish and won't crash.
To do so it's really just a list of rules
that the verifier checks
and your program has to comply with those rules.
So for example, your program has to finish,
it cannot be like an infinite loop.
So your program has to be a directed acyclic graph.
You can't have unreachable code,
you can't have unchecked dereferences.
Your stack size is limited and your overall program size
is also limited.
And finally, one of the most infamous features
of the verifier is it's very cryptic outputs.
So basically if your program doesn't pass the verifier step,
you will have a huge log of everything
that the verifier looked into
and eventually some kind of error telling you what happens.
But yeah, basically you are in
for a very painful beginning session.
Last but not least, eBPF comes with a list of helpers
that will help you access data
or executive operations
that you wouldn't be able to write natively.
For example, you have context helpers, you have map helpers,
a lot of things that you wouldn't be able to write in C
and that you would need external instrumentation to do.
In short, you have about 160 helpers.
And most of the heavy lifting of your eBPF programs
will be based on those helpers.
So that concludes this introduction to eBPF.
And I will hand it over to you, Sylvain,
so that you can kick off the presentation of the rootkits.
- Thank you, Guillaume.
Before we get into the details,
let's see why eBPF is an interesting technology
to write a rootkit.
First, the safety guarantee.
Broad eBPF mean that a bird in our rootkit
cannot crush the host.
(indistinct) will not cause any low message to be emitted.
The user therefore has no way to know
that something actually went wrong
and notice the presence of the rootkit.
As we saw earlier, the eBPF bytecode
is converted to native code.
And the number of instruction is limited,
which limits by extension, the performance impact
that our rootkit can have on the machine
that could otherwise be detected by the user.
On the commercial side,
the eBPF is used by an increasing number of vendors
in various use cases,
network monitoring security, for instance.
With eBPF becoming widespread,
returns of one product being abused,
too many issues programs also increases.
The safety guarantee we just told you about
should not give the security and (indistinct)
before filling of security.
There's a lot of activity around eBPF
and each new version of the Linux kernel
comes with a new set of eBPF helpers,
bringing new capabilities.
As we wanted our rootkit to run
on a widely used distribution on Linux
such as RHEL/CentOS Linux, or the latest Ubuntu LTS,
we used a limited number of helpers using RHEL/CentOS
or a feature like KRSI will have probably made
the development of the rootkit easier.
One of the primary tasks of a rootkit
is to ask itself, what does it mean in our case?
EBPF programs are bound to a running process.
If this process gets killed,
all the attached eBPF programs will be unloaded.
For that reason,
it is essential that we boast either our program
and protect it from being killed.
The eBPF programs and maps used by the rootkit
should also be hidden and we should forbid other programs
to gain access to them, so there are five descriptors.
So let's see a rootkit in action.
So let's start with kit, would give us its PID.
Then we can try your PS kernel
in order to see if we can detect it from the output.
We can try using its preface entry and nothing.
We can even try using your such file
or even a relative pass.
And we still have the same issue, no such file or directory.
And finally, we can try to send a signal
to see what happened, and we get no such process error.
Your prescription capabilities of our rootkit
mainly rely on the use of two BPF helpers.
So BPF provides user helper arose our BPF program
to ride into machinery of the process
that issues the syscall.
This can be used for instance to alter the other data
that is returned by your syscall.
It's also possible to alter the syscall arguments.
So it's one caveat with this helper,
the memory to be modified
has to be mapped into the kernel space.
Otherwise a minor or major pitfall will be triggered
causing the BPF provider user call to fail.
User BPF helper is to ease the BPF override return.
This one allows you to change the return
that you have a syscall and as an interesting property.
If this helper is used at the syscall exit,
it will simply change the return
that you will do at the syscall.
But if we use it at the entry of the syscall,
the execution of the syscall will be completely skipped.
It is important to note that this helper
can only be used at the entry of the syscall or at the exit.
So let's see how obfuscation of a file actually works.
At startup, the rootkit will populate the map
with the pass of it's PID folder.
Now the user space issues a file with syscall such as stat.
The syscall usually come in two forms,
one that accept the pass to the file as a string,
another one that accepts the VFS for the file,
but the user space program must have previous (indistinct)
using an open syscall.
So let's consider the former.
To properly identify the targeted file,
the rootkit needs to do an accurate resolution of the pass
as the pass specified could be a relative pass.
At the entry of the syscall,
that is not in context to the resolution.
So we need to go deeper in the kernel in our case,
in DBFS code.
So we are at the region,
but at this point we cannot block the syscall
as we are outside of the override of points
for the eBPF override return helper.
So the only thing that we can do
is to change the return value
so that the user space believes that the syscall failed.
We also need to scrub the content of the switcher
that could have been filled by the kernel.
Now let's consider the latter,
the ratio that accept the file as return.
We do the same passage version as before,
but instead of just pretending it was the descriptor file
that the syscall failed, we still have the file descriptor
that we can edit located into an eBPF map.
If the same processes issue the syscall
with the same file descriptor,
we can add the syscall entry return an error
and block the syscall.
In this situation, the user has no way to know
that the file descriptor exist.
And as we control the read syscall,
we can also hide all the references
to the file descriptor in proxy phase.
Blocking the syscall that accept the PID as an argument
is trivial if you're using BPF override return,
same for in kernel modules.
Now let's see your demo.
Let's demo the obfuscation of our BPF programs and maps.
So we still have the rootkit started
and we've released the maps and the programs,
thanks to the eBPF tool kernel online.
We can't see anything related to the rootkit.
Now if we (indistinct) kprobes and maps,
and we list, again, the programs,
we can see the program related to the binary,
but still nothing related to the rootkit.
Then we can even try checking the kprobes
and still nothing related to the rootkit.
The ID is to hook into the eBPF syscall itself,
two things are required here.
We want to avoid the real space
to utilize the two IDs of our own duties
using the BPF_prog_get_next_id command.
And we need to prevent the user space
from getting a file descriptor to the program or map
from the ID using BPF_prog_get_fd_by_id
basically to prevent ID guessing.
When these operations are handled by the rootkit,
the user space is not able to view
or modify our programs and match.
BPF provider user is a crucial part of our rootkit
but it comes with a pitfall.
While an eBPF program that makes use of it is loaded,
the journal printer pretty scary, but legitimate message
in the kernel ring buffer.
So let's see how the rootkit handle this.
First, let's check what is the current statues
of the kernel ring buffer using the kernel dnesg.
So nothing special here.
And we can even start a cache on the device
used by the ring buffer.
Then we start the rootkit.
So we can see that the cache terminated
with a pretty legit looking message.
We can recheck the kernel ringer buffer,
thanks to the message kernel online,
and we can only see legit-looking messages,
but I need more.
And then we can stop the rootkit and we can run again,
the message kernel online
in order to see what was overridden by the rootkit.
So let's see how it works.
In this diagram, Ronald is waiting for a new message
and is blocked on a red syscall.
Two BPF programs are loaded secondly by the rootkit.
The first one, which doesn't choose eBPF provide user
work on our red syscall and we make it written zero
and we'll write legit-looking message.
This will guarantee that the warning message
is related to the BPF ProLite user won't be read.
Then the second program using BPF ProLite user is loaded.
At this point, the red syscall can be unblocked
and we can override the content of the warning messages
with legit-looking messages.
Another important task for the rootkit
is to set up a persistent access
across reboots, for instance.
The rootkit can copy itself to a dedicated place
in either it's binary file with the same mechanism
that we already saw.
For the persistent access to the system,
we can use a generic method close to what we described
in the obfuscation path.
We can replace the content of some critical files
read by the root demands, such as crotab or even sshd.
Let's see an example targeting sshd
and using the reader override approach.
So the approach here is to open an sshd
to the authorized key files.
Only sshd should be impacted,
meaning the file will remain the same
for the user point of view.
And we want to have it available to the command-and-control.
So let's see this in action.
So let's take the authorized keys content first.
So we can see that only one key is present.
So let's start the connection
and it's in that password is required.
So now we're going to start the rootkit
and we are going to specify
that we want to inject an SSH key to the authorized keys
but only for SSH.
So we can try your connection again,
and it seems to be successful.
And now we can check what is the content
of the authorized key from the user point of view
and nothing changed apparently.
Persistent access to an application database
can also be set up using another type of eBPF program.
Uprobes eBPF programs attached to user space function.
In addition to being safer and easier to use than ptrace,
they offer a valuable advantage.
The kernel will automatically set up for us the hooks
on every instance of the program.
Let's see a uprobe demonstration using (indistinct)
So first, let's try to connect to (indistinct)
using the word bonsoir as password.
This one seems to be the good one, then trying hello,
and this one is rejected.
Now we start the rootkit
and we'll get the opposite treasured.
Now the value password is hello.
So the idea here is to hook
on the md5_cript_verify function
of a progressive creator
that check whether the user provided the right MD5
for its role, passwords, and the challenge
sent by the server.
Overwriting the expected hash contained in shadow_pass
with a known value makes the comparison succeeded
and give persistent access to the database to the attacher.
Now I will hand over to Guillaume that will show you
the command-and-control capabilities of the rootkit.
- Thank you, Sylvain.
Let's talk about the command-and-control feature
of the rootkits.
So what exactly do we want to do?
We want to be able to send commands to the rootkits
to exfiltrate data and to get remote access
to the infected hosts.
Unfortunately, there are a few eBPF-related challenges
that we need to face in order to implement those features.
First, you can't initiate a connection with eBPF.
Second, you can't open a port.
However, eBPF can hijack an existing connection.
So in order to show up this feature,
we have set up a very simple infrastructure on AWS.
A web app was installed on any C2 instance
and we used a Classic Load Balancer
to redirect HTTPS traffic to our instance over HTTP.
In other words, the TLS termination
is done at the Load Balancer level
and its eBPF requests are sent
to our instance and encrypted.
So our goal is to implement CNC
by hijacking the network traffic to our web app.
First, we need to figure out which eBPF program types
we're going to use in order to implement this feature.
Although eBPF provides a lot of options to choose from,
we decided to go with two eBPF program types,
XDP programs and TC classifier programs.
So both of those programs
are usually used to do deep packet inspection use cases.
And while XDP only works for ingress,
TC works on both ingress and egress traffic.
Another difference between the two program types
is that XDP programs can be offered
to the network interface controller,
which essentially means that your program will be run
before the packet enters any subsystem
into the network stack.
On the other hand, TC programs have to be attached
to a network interface, but much later in the network stack,
which means that they are triggered later in the kernel.
With both programs, you can drop, allow,
and modify your packets.
And with an XDP programming,
you can also retransmit a packet.
This option is actually super interesting for us
because it means that you can essentially receive and answer
to a packet even before it reaches the network stack,
which in other words means that you can do this
even before it reaches any kind of network firewall
or monitoring on the hosts.
Skipping the network side also explains why XDP programs
are mainly used for DDoS mitigation
and TC programs are usually used to monitor
and secure network access at the pod or container level.
So what you need to remember about this slide is that first,
XDP programs can be used to hide natural traffic
from the kernel entirely
and TC programs can be used to exfiltrate data
on its way out.
First, let's see how we used XDP programs
to receive commands with the rootkits.
So we implemented a client for the rootkit
and this client communicates with the rootkit
by sending simple https requests
with custom routes and custom user agent.
So after going through the Load Balancer,
the request eventually reaches the host
and triggers our XDP programs.
Then our program pass the requests, the http routes
and understand that this request
is not meant for the web app, but it's meant for us.
So after sending the other,
sorry, after reading the user agents,
the rootkit executes the requests that come in
and moves on to the final step.
So this final step is probably the most important one.
It overrides the entire request
with a simple health check request,
and we do this for two different reasons.
First, we don't want the manager's request
to reach the web app
or any kind of user space monitoring tool
that might be bringing
and that might detect the unusual traffic.
And second, we want the client to receive an answer
in order to know if the request was successful.
So, as I said before,
we could also have drop the packet entirely,
but since we're using TCP,
the Load Balancer would have retransmitted the packet
over and over again until the request times out.
And this would have generated noise
and increase our chances of getting discovered.
That said, if you were working with a UDP server,
this would be a totally valid strategy.
So let's have a look
at how we can send Postgres kernels remotely.
All right, so on the left of the screen,
you can see two different shells.
Those shells are connected to the remote infected hosts
on AWS and on the right, this is my local shell
and this is the attacker machine.
Okay, so let's start
with trying to log into the Postgres database
using the normal password.
And again, the rootkit is not running yet.
So as you can see, the bonsoir passwords works fine.
And then they start the rootkit and restart to log in again
and as expected and as you've seen before
during Sylvain demo, it doesn't work.
So you have to change into, hello,
and this time it will work
there you go.
Okay, so we're gonna try to do the same thing,
but instead of hard coding the new password
with the rootkit, we're gonna define remotely through C&C
what the new password should be.
So, as you can see, we have a custom client
that will make a request to its CPS,
it needs to be a CPS request to defcon.demo.dog
and then we will provide both the role
and the secret to override the normal secret with.
So the request that will go through is a very simple one
with the custom routes
and the user agent will contain the new password
that will be used at runtime.
So as expected, we get the 200 okay from the health check,
which essentially means that we know
that the new password now is defcon and not hello anymore.
So as you can see, hello that doesn't work,
but if I'm change it to defcon, here you go.
This time it does work.
Okay, so this is how we send the command to the rootkits.
Now let's see how we can exfiltrate data.
So to exfiltrate data,
the client has to send an initial request
to specify what kind of data and what kind of resource
we want to exfiltrate.
So the XDP part of this process
is basically the same as before,
but this time the XDP programs stores the network flow
that made the request along with the requested resource,
sorry, in an eBPF map.
And the reason why we do so
is because when the web app answers the health check,
we want to be able to detect the packets
that are meant to be sent back to the clients.
So when the XDP answer reaches the TC egress classifier,
our eBPF program looks at the network flow
and overrides the answer with the requested data.
Now, the question is,
what kind of data can you exfiltrate with the rootkit?
And the answer is, well, pretty much anything
that is accessible to eBPF.
And the reason for that is, as I said before,
multiple program types can share data through eBPF maps,
regardless of what those programs are supposed to do.
So basically you can exfiltrate things like file content,
environment variables, database dumps, in-memory data
if you start looking at the stacks of the programs.
Anyway, you can pretty much exfiltrate whatever you want.
So let's have a look at a simple demo
that we can exfiltrate Postgres credentials
along with the file content of etcpasswrd demo.
All right, so again, the two shells on the left
are the ones connected to the infected hosts on AWS
and on the right, this is my local shell.
So the first request that I make here
is to do Progres list,
which basically means please list all the credentials
that you have detected so far since the rootkit has started.
And as you can see,
the answer was, sort of the health check answer
was overridden with the content of a map
that we used to store the passwords
that we've collected at runtime.
And again, remember that with Postgres,
you don't need the clear password to login.
You just need the hash password
that is stored in the database.
Here you go.
So now we're gonna try to do the same thing
to dump the contents of etc password.
So to do so, this is a two-step process.
The first thing you want to do is tell the rootkits
to start looking for this specific file.
And as soon as a user space process tries to open the file
and read the content of the file,
our rootkit will actually copy the data
as it is sent to the user space application
and say that into an eBPF map
so that it can be retrieved later.
So this first request will tell the rootkits
to start looking for etc password.
And now let's go back to the host
and trigger some kind of pseudo operation
so that our user space process tries to open the file.
Here you go.
And then this time, instead of saying, add,
we're gonna say, get,
and this will dump the content of the etc password file.
Here you go.
All right, so the cool thing about this technique
is that it applies to any un-encrypted network protocol.
So for example, we also implemented it for DNS,
which means that you can actually use it to do DNS spoofing.
So the only difference between the normal way of doing this
and the DNS spoofing is that instead of using a TC program
to override the answer of the request,
you will actually switch to see an XDP program
because DNS requests are made from the host
instead of received by the host.
All right, so let's move on
to our network discovery feature.
So I know everybody knows what it is,
but I have to say it anyway,
network discovery is the ability to discover machines
and services on the network
so that you know where you want to go next
in the infrastructure.
And also discovering services is a super important step
when you are trying to pivot between hosts,
because it will tell you what kind of attacks
you might want to try.
So the rootkit
has two different networks discovery features.
One of them is passive, the other one is active.
And you can control both of them
through command-and-control.
So I'm gonna get into more details later,
but basically the only difference between the two
is the kind of networks scanning you're looking for
and also the level of traffic
that you are willing to generate on the network.
So first, let's have a look at the passive option.
So the passive option
is simply a basic network monitoring tool.
So it will do pretty much the same thing
as any other eBPF-based network monitoring tool,
which means that it will listen
for any ingress or egress traffic
and then generate a graph
from all the collected network flows.
It will also show you the amount of data
that was sent per network flow.
And to implement this feature,
we used our TC and XDP programs.
So the TC programs were used
to monitor the egress traffic and the XDP programs
were used to monitor the ingress traffic.
So for this version of the rootkits,
we are limited to IPv4 and TCP UDP packets,
that said support for IPv6 and all the protocols
could have been headed easily.
So the reason why the passive option is pretty cool
is that it will not generate any traffic on the network.
In other words, it is basically impossible
to detect that someone is tapping into your network
and more specifically at the network
that creatures this specific infected host.
However, this doesn't work
for services that do not communicate
with the infected host.
And so in other words,
the graph will definitely not be complete.
And that's also why we implemented the active method.
So the active method is a simple ARP scanner
along with a SYN scanner.
So we implement it using only our XDP programs,
which means that the entire process
is done without involving the kernel stack.
And although this will be a slower process,
you can use this method to discover hosts and services
that are reachable by the infected host,
but that are not communicating usually
with the infected host.
And again, the rootkit client
will generate a nice network graph for you
once the scan is complete.
So on a technical level, this feature of the rootkit
is actually quite interesting because, as I said before,
eBPF cannot create a connection from scratch.
So in other words, we had to figure out a way
to generate hundreds of SYN requests
while dealing with this limitation of eBPF.
So let's see how we solve this problem.
So in order to send a SYN request,
you first need to know the MAC address of the IP
that you want to scan.
To do so, we use the same trick
that we've been using so far,
which is to override the requests from the rootkit clients.
So when our XDP program receives a scan request
for a specific IP and a specific port range,
it will override the entire request with an ARP request
for the target IP.
And then instead of returning XDP pass,
which is what we've done so far,
and also which would send the packets to the network stack,
our eBPF program returns XDPTX.
So what XDPTX does is send the packet out
to the network interface controller it can inform.
In other words, our HTTP packet
was transformed into an ARP request
and broadcasted back to the entire local network.
So eventually the target IP will answer the ARP request
and will be able to store the MAC address
of this specific IP in an eBPF map.
However, during this entire process,
the TCP packet that was used to send the HTTP request
was never acknowledged by the kernel.
And that is simply because it never made it's way
to the kernel in the first place,
which means that the Load Balancer or the client itself
will eventually try to retransmit the packet.
And when this packet is retransmitted
and when it eventually reaches our XDP program,
we will do the exact same thing.
But this time, instead of, because we know the MAC address,
instead of overriding the request with an ARP request,
we're gonna override the request with a SYN request.
And more specifically a SYN request
with the first port of the provided port range,
the target IP and the MAC address of the target IP.
And assuming that the remote IP or the remote host
doesn't have any kind of prediction against SYN request,
sorry, SYN scanning, it will answer either resets
or SYN plus acc to this first request.
So reset would mean that the port is open and SYN PLUS acc
would indicate that there might be a set of sprinting
running on the host.
And this is where basically the network loop happens
and the reason why
we were able to generate hundreds of packets
while dealing with the limitation of eBPF,
that is the inability to create packets.
So whenever we get an answer from a SYN request,
we override the received packet with another SYN request
on the next port.
And we also switch the IPs, switch to the MAC addresses,
and send it back again to the target IP.
And we do so in a loop
until we go through the entire port range.
So eventually the clients will try one last time
to retransmit the initial HTTP requests.
Because once again, during the network loop,
we never answered the second retransmit.
So eventually when this third retransmit
reaches our XDP program, we will override the request
with the usual health check requests
so that the 200 okay answer will make its way
back to the client after the request was handled
by web app and user space.
All right, so let's see it in action.
So on the right of the screen,
this is a shell to the infected host on AWS
and at the bottom here, it's another one.
And at the top, this is my local shell on my machine.
So the first thing you want to do is to start the rootkit.
Then second is to start dumping the logs of the rootkit.
So an eBPF can actually generate logs using a trace pipe.
Obviously you would not want to do this in a real use case
for a rootkit, but this is a great way
of visualizing the scan as it goes through.
Yeah, so that's why I'm doing this
and that's why you will see what the rootkit does
at runtime.
And then let's make the scan request.
So what I'm saying here is please scan the IP 10.0.2.3
from port 7990 and for the next 20 ports after this one.
So the first thing you can see
is the request is immediately changed into an ARP request
and we already got the answer for this ARP request.
So next step, when we get a retransmit,
we will change this into a SYN request.
Here you go.
So the SYN request went through
and then you can see the loop that happens.
And we increased port one by one
until we reached the final port
requested by the port branch.
And then now we are waiting for the third retransmit
and this retransmit will be the one
that we override with the health check request,
which means that we will eventually get here,
go the 200 okay, in other words,
the answer from the user space web app.
All right, so now what you want to do
is retrieve the output of the scan
and exfiltrate all the network flows
that were detected at runtime, and here you go.
So you would say network_discovery_get and eventually ...
So it actually requires a lot of different requests
because there's a lot of data to exfiltrate,
but eventually you will get the entire list of network flows
that were captured by the rootkit.
Here we go.
So you have all the different individual flows
and then more importantly,
you will have a graph generated for you.
So this one is the passive, sorry, active graph.
So as you can see, range there, you can see the ARP requests
and replies between those different hosts.
And then in gray, those are the SYN requests
and the reset answers.
And in red is the only SYN plus acc,
and so from the remote host.
All right, and then you have also the passive graph,
which is the one that we saw before.
Okay, so now let's move on to our RASP bypass.
So RASP stands for runtime application self-protection.
So in a few words,
a RASP is a new generation of security tools
that uses runtime instrumentation
to detect and block application level attacks.
And more importantly, it leverages its insight
into the application
in order to make more intelligent decisions.
So simply put, it is some kind of advanced
input monitoring tool that can detect malicious parameters
and can understand if a malicious input
will successfully exploit a weakness from one of your apps.
So the textbook example of a RASP
is usually a SQL injection.
So the RASP route implement multiple functions,
instruments, sorry, multiple functions
in the libraries that you use
such as for example, the HTTP server library
or the SQL library.
And it will check at runtime
that the user control parameters in your queries
are properly sanitized.
If not, the RASP will stop the query
before it reaches the database and redirect the client
to an error page or some kind of error message.
In other words, a RASP relies on the assumption
that the application runtime has not been compromised,
which is exactly what we can do with eBPF.
So just a little disclaimer before I move forward.
I want to stress the fact that we are playing
outside of the boundaries
of what a RASP can protect you from.
And more importantly,
this bypass does not apply to one specific RASP,
but to all of them
because this is one of the core principles
of how a RASP works.
So let's have a look
at how RASP products a good web app from a SQL injection.
So let's say that you have a web app
with a simple products page
and get parameter to specify the category of the products
that you want to see.
Chances are your web app
uses the default go database SQL interface.
So this is a generic interface that you can use
to query your database without having to worry
about the underlying driver
and the database type that you're using.
More importantly for us,
since it is such a generic interface,
this is usually where the RASP tools instrument your code
because simply it's much easier to hook at this layer
rather than having to hook onto all the underlying drivers.
So when your request is handled by the web server,
the query will be formatted
with the provided category parameter
and eventually the web app
will call the query context function of this SQL interface.
This is when the RASP checks the query
and makes sure that everything is normal.
And if it is, the execution will resume its normal flow
and the underlying driver will be called.
So in our example, we use SQLite,
so the SQLite driver is called.
Eventually the query makes its way to the database
and the answer is sent back to the client.
However, if the RASP detects that something is wrong
or detects some kind of SQL injection,
it will block the query and redirect the client
to an error page.
All right, so now let's see what we did
to bypass this protection.
Well, the answer is actually pretty simple.
We added a uprobe on both the database SQL interface
and the SQLite driver interface.
What this allows us to do is call one of our eBPF programs
right before the RASP checks the SQL query
and trigger another one
right before the SQL query is executed
by the database itself.
And thanks to the BPF ProLite user helper,
we can override the input parameters of the hooked functions
so that the RASP sees a benign query
and the database executes our SQL injection.
And the cool thing about this
is that we can even do it conditionally,
which means that we can bypass the RASP
only if one specific secret password
was added to the beginning of the query.
Perfect, so let's move on to the demo.
So as you can see, we have a very simple web app.
So it's a shoes retailer.
So you have a lot of different products
and you can filter by category.
So let's try to do a SQL injection
using the get parameter.
So the injection was simply be select star
from user and because the RASP is not running right now,
the SQL injection should work.
Here you go.
And if you scroll down this time,
you will see the users along with the passwords, perfect.
So now let's restart the web app with the RASP,
which is what I've just done and try this again.
So let's go to the shop
and then override the category parameter, perfect.
And this time the RASP blocked the request
because it detected
that someone tried to do a SQL injection
and the SQL injection would actually have succeeded.
Great, so now let's start the rootkits
by providing the path to the web app
and then try to refresh the page.
So again, this should also be blocked by the RASP
because we haven't provided the secret password
for the bypass to work.
And the secret password is of course, defcon.
And when we say defcon,
the entire process that I described before
will be triggered.
And as you can see, the RASP did not detect it.
So that's all for our RASP bypass.
I hope you had fun.
Just before I hand it over to Sylvain
for the detection and mitigation strategies,
I wanted to say that, unfortunately, we won't have time
to talk about the container breakouts
that are (indistinct) into the rootkit.
However, they have been presented
during our Black Hat call this year.
So if you are interested, feel free to check it out.
That being said, Sylvain, take it away.
- So let's talk about detection and mitigation.
How could we detect and protect ourselves
from this type of rootkit?
We could do this at different level.
First, if a vendor provided you eBPF programs,
you should go through an audit and an assessment phase
of their programs.
Some changes that the code has to be GPL.
It probably uses some internal candid symbols,
so you can ask for it.
What should we be looking for?
The program types that are used,
but also the eBPF helpers used.
So communication.
So maps between programs may indicate the control risk
in the case of that the vendor program is compromised.
We developed a tool to assist in this auditing phase
by inspecting the health files containing eBPF programs,
it is able to list the use entities, programs, and maps,
and compare to graph of the interactions between them.
The tool was run on our rootkit with the following result.
We can identify on the graph that the XDP program
has stored information into maps
that are also used by some kprobes,
which correspond to the command-and-control capabilities
of the rootkits.
It is also possible to mitigate at runtime
the loading of such programs by monitoring calls
to the eBPF syscall and logging (indistinct)
It would even be possible to protect the eBPF syscall itself
by either (indistinct) syscall to it
to need some trusted processes.
As the programs inspected before then and rejected
if they contain suspicious patterns
or make use of some dangerous helpers.
We could also compute and validate
the signature of the programs before (indistinct)
exist to the kernel itself.
Using TLS everywhere for network traffic
also helps mitigating the risk of a rogue eBPF program
that intercept network data.
Now if we are not able to block
the loading of such a rootkit,
how difficult would it be to detect its presence?
Even if it's possible, though very challenging
to write anonymous perfect rootkit,
we should concentrate on the action the rootkit will have
to block and lie about the result of such actions.
For instance, our rootkit (indistinct)
we are loading kernel modules
because such a module would have the ability
to lead the eBPF programs and the active kprobes.
Now let's imagine that we insert a module
that executes a specific action only known to us.
The blocking of the module by the rootkit
will then be easy to detect.
Monitoring the network traffic at the infrastructure level
could that detecting hijacked connection
or strange back into our transmission.
Our rootkit being far from complete and far from perfect,
it should be relatively easy to detect it.
That being said, we hope it will bring to light
the potential and the risk of such an eBFP base rootkit
while presenting some interesting technique.
The code of both the rootkit and the monitor
is available at the servers, please have a look.
Thanks for your attention and have a great conference.