- Hello, everyone.
Today we're going to talk about Phantom Attack,
evading the system call monitoring.
My name is Rex. - My name's Junyuan.
- Imagine an attacker compromise your Linux infrastructure.
So the attacker first compromise a web app
through a web app remote call execution,
and then he launches as reverse shell.
Then they discovered a vulnerability on the system,
he can activate privileges
using the sudo vulnerability CVE-2021-3156.
Then he's looking for secrets on the system.
So he read the /etc/shadow file.
And then he discover additional
lateral movement opportunities
by reading the SSH process environment variable.
Then he lateral move to the second machine
using SSH hijacking.
As he is celebrating this moment,
he discovered that his reverse shell connection is gone,
and it doesn't take him too long to discover
that his IP is completely blocked.
Now let's take a look at the other side of the story.
While all this is happening,
our security engineer has received
a bunch of Slack messages for the alerts generated
by his latest cloud workload protection software.
And the reason that the software can discover
all of these activities precisely
is because it monitors the system calls
and other process-related data.
So for example,
when the attacker launches the reverse shell,
there will be a connect system call.
And there may be additional system calls.
Depends on the reverse shell that he uses.
This is similar for the other activities.
And through this talk we're going to use
openat system call as an example.
So let's take a look at how one can use system calls
and other process information
to detect a attacker reads /etc/shadow.
So here's an example rule.
The rule is trying to detect untrusted programs
reads the /etc/shadow.
Let me explain what the rule means.
It detect that there is a openat or open system call.
What is the read permission,
and the file name is equal to /etc/shadow.
And also the program is not in the allow list
that allows to read the /etc/shadow file.
So from this rule,
it should be very obvious that the ability
to precisely monitor system calls
and other system correlated data
is critical for the detection for this attack.
The agenda of this talk is we will talk
about system call monitoring in more detail.
And then we'll talk about the two open source
system call monitoring project that we analyze.
And then we'll talk about the first vulnerability,
the TOCTOU issue, which we use Phantom v1 attack to exploit.
And then we'll talk about the second vulnerability
of semantic confusion issue,
which we use Phantom v2 attack to exploit.
And finally, we'll conclude the talk with takeaways.
With that, I will hand over to Junyuan
to talk about the system call monitor.
- Yeah, so as Rex mentioned,
system called monitoring is very important
to detect threats.
So what is system call monitoring?
We can define a technique to verify
whether the application system call coming forward
to the rule that specify the program behaviors at runtime.
Here is the graph showing how system call monitoring works.
When application system call is invoked,
system call code path is executed.
If there are any hooks in a code path,
the attach program will be called
to collect subsequent data.
For example, this is called arguments.
The data are sent to user space monitoring agent.
The morning agent will check if application system calls
conform to the user-defined rules.
If not the case, it may generate errors.
So typically at least two steps should be include
for system call monitoring.
On step is called syscall Interception,
which is to get notified
if target's doesn't call are invoked.
In order to intercept system call,
you can use tracepoint or raw_tracepoint.
Both of them are static hook placed in a kernel code.
Raw tracepoints are eBPF alternatives to standard tracepoint
It's faster because it provides low access
to the arguments without processing.
For this call interception,
the kernel provides two raw tracepoints,
sys_enter and sys_exit.
These raw tracepoint can be called with functions,
trace_sys_enter and trace_sys_exit, respectively.
The first arguments of this fashion
are pt_register structure,
saving user registers in user kernel mode switching,
and it includes system call arguments.
The second parameters is syscall number.
If any progress attached to this raw tracepoint,
it will be executed with the same arguments as functions.
Tracepoint has low overhead,
but it only provides a static interception.
Different from tracepoint,
kprobe or kretprobe provides dynamic hook in the kernel.
Using these, we can register the progress
on kernel instructions.
For example, on system call code path.
When the instruction are executed,
it will trigger register progress.
Kprobe can be insert on almost
any instructions in the kernel.
However, kretprobe can only be insert
in function entry and exit.
Kprobe provides dynamic hook,
but it's low compared to tracepoint.
And you need to know exactly
how data is placed on a stack or registers
in order to read the system call data.
You can also use the LD_PRELOAD trick
to intercept system call, but it's not working in all cases.
For example, application is statically compiled.
Ptrace system call provides another way
to intercept system call, however the overhead is high.
The second step of system call monitoring
is called syscall data collection,
which is to collect system call data.
For example, system call arguments.
After we are notified by simple events,
the program used to collect system call data
is called tracing program.
For example, you can use the tracing program
to collect system call arguments.
As we mentioned before,
tracing programs can attach to different hooks,
like tracepoint, raw_tracepoint, kprobe, or kretprobe.
When the hooks fire,
tracing programs are called to collect data.
There are different ways to implement tracing programs
to collect system call data.
You can use Linux-native mechanisms,
like ftrace or perf_events.
You can also implement the trace programs
in kernel module or eBPF programs,
which allow the execution of a user code in a kernel.
The open source projects, Falco and Tracee,
both used similar techniques to monitor syscalls.
Falco is originally created by Sysdig.
It's one of the two security and compliance projects
and the only endpoint security monitoring project
in CNCF incubating projects.
It has 3.9Ks GitHub stars.
It actually consumes kernel events
and enrich them with information from cloud native stack,
like Linux containers and so on.
Falco supports both eBPF and kernel module implementation
for the tracing program.
Tracee, on the other hand,
is originally created by Aqua Security.
It has 1.1K GitHub stars.
It's basically a runtime security
and forensics tool based on eBPF.
So unfortunately, the open source projects
or other projects using similar techniques,
are vulnerable to be attack during syscall monitoring.
The first vulnerability is time-of-check, time-of-use.
During time-of-check, tracing programs collect syscall data.
During time-of-use, syscall data used by its kernel
is different from what tracing program check.
Let's take openat system call for example.
The second parameter is called finding,
which is the pointer pointing to user space buffer
between time-of-check and time-of-use.
This pointer is vulnerable to be modified from user space.
So we do introduce Phantom v1 attack,
that can exploit the TOCTOU issue.
The second vulnerability is semantic confusion.
It means kernel interprets data differently
from tracing programs.
For example, symbolic link is interpreted differently
by the kernel and tracing programs.
You can also introduce Phantom v2 attack
that can exploit semantic confusion.
We will also demonstrate Falco is vulnerable
to both Phantom v1 and v2 attack,
while Tracee is only vulnerable to Phantom v1.
In order to understand TOCTOU,
we used openat system call for example.
We used kernel version 5.4.0,
but regardless of the kernel version,
if the monitoring software use tracepoint in this way,
the TOCTOU vulnerability will exist.
To simplify, we only show the code
that is related to that tech.
When openat system call is invoked in applications,
syscall handler execute trace_sys_enter function
with two arguments, as we mentioned before.
If any tracing program attached to this sys_enter tracepoint
the program will be executed.
After that, syscall handler look up syscall table
and jump to openat system call to open the file.
Before return them to applications,
the handler will call trace_sys_exit
with exactly the same arguments as trace_sys_enter.
So similarly, if there are any tracing programs
attached to sys_exit at tracepoint,
the program will be executed.
As we mentioned before,
the second argument of openat system call
is filename pointer pointing to the user space memory.
The filename is passed through do_filp_open function
and the kernel copy it to kernel buffer temp
using getname function.
After that, kernel use the kernel buffer
to call internal function, do_file_open, to open the file.
This is time-of-use for system call arguments by the kernel.
If we divide the openat system call code path
into two parts based on the getname function,
you get two step copass, CP1, copass one,
and CP2, copass two.
In CP1, the filename pointer
hasn't been copied to the kernel buffer.
In this case, no matter where we place the hooks in CP1,
the attached tracing program
will have to read user space buffer
in order to get the filename.
This is vulnerable to be changed in user space attacker.
For example, if we attach tracing program
to sys_enter tracepoint or to do_sys_open using kprobe,
during time-of-check the tracing program
will have to read the user space buffer to get the filename.
In CB2, user space memory has been copied
to the kernel buffer,
making it not vulnerable to be changed from user space.
For example, if we attach tracing program
to the entry of do_file_open function using kprobe,
the tracing program can read a kernel buffer tab
to get the filename.
That's kernel buffer is not vulnerable
to be changed for a TOCTOU attack.
However, if the hook are placed improperly in CB2,
TOCTOU is still possible.
For example, if tracing programs
attached to a sys_exit tracepoint,
it will read the user space buffer to get filename.
If, as we mentioned before, we use kernel version 5.4.0,
but regardless of the kernel version,
if the monitoring software use the tracepoint in this way,
this vulnerability will exist.
Falco is the vulnerable to TOCTOU,
and the vulnerability is tracked by CVE-2021-33505,
with a score of 7.3.
In particular, the vulnerability exist for Falco
is a version older than 0.29.0, or open source sysdig.
It also affects our commercial versions space
on the open source agent.
This was confirmed by an open source maintainer.
Please contact the vendor for divergence.
The reason why Falco is vulnerable to TOCTOU vulnerability
is that it use sys_enter and sys_ext access point
to intercept system calls.
In that case, user space pointers are read directory
by Falco trace program in both kernel module
and eBPF progress implementations.
In order to demonstrate the generality of TOCTOU,
we evaluate the syscalls in front of rules.
Just know that we only can see the system calls,
that includes use of pointers as arguments,
like openat system call.
And we found that Falco is vulnerable
to most of these calls that we evaluate,
except execve system call.
Because Falco doesn't read user pointer
for execve system call arguments directly.
Instead it read the data from kernel in a structure.
So we evaluate the Tracee 0.4.0.
And we found that it's vulnerable
to many system calls, like connect system call.
One of the thing I need to mention
is there's no CBE given because the Tracee ask us mentioned
that TOCTOU attacks on syscall wrappers or tracer
is a well-known issue, and Tracee is no exception.
And also agrees on the fact
that there's no CVE or novel findings,
and therefore you could talk about it publicly.
I will let the audience to interpret.
So I will hand over to Rex to explain
and demo Phantom v1 attack.
- All right, so the high-level idea
to exploit the TOCTOU issue is fairly simple.
So first of all, we want to trigger
the target system call with malicious arguments.
And we'll let the kernel to read the malicious argument
and perform the intended malicious action for us.
After the kernel reads it,
we will override the data structure
pointed by the user space argument pointer with benign data.
And at sys_exit the tracing program reads a data structure
pointed by the user space pointer
and checks the benign data against the rule,
and therefore it will not fire.
Although the high level plan sounds simple,
there is a few technical challenges
that we need to overcome.
First one is when does the kernel thread read it,
and how can we synchronize the override
with the kernels that read?
Are the racing windows big enough
for these system calls that we're going to attack?
And how do we ensure the tracing program
gets the overwritten copy on time?
So before I dive into the step-by-step exploitation,
there are a few primitives that we use in the exploit
which I want to talk about.
First one is Userfaultfd system call.
The system call is designed in a way
that a user thread can handle page fault.
But page fault is traditionally handled by the kernel.
So what's the initial design intentions of this?
This was designed for memory externalization.
In the case where you're running a distributed program,
you can run compute nodes and memory nodes.
When the compute node needs a particular memory
that doesn't exist in the compute node,
the kernel triggers the page fault
and the user space fault handler.
It's going to reach out to the memory node
to get the desirable memory.
On the other hand, if the compute node has memory pressure,
it will send those memory pages back to the memory.
One very important fact about Userfaultfd
is that once the kernel thread triggers the page fault,
the kernel thread is completely paused
and wait for the user space program to respond.
As some of you may already be aware of,
this has been used quite a bit
in exploiting kernel race condition bugs.
The other tool primitive I want to talk about,
one is interrupts and the other one is scheduling.
So an interrupt notifies the processor with an event
that requires immediate attention.
It will diverge the program control flow
to an interrupt handler.
Let's look at the picture on the right side.
So we have two cores and we have two tasks
running on each core corresponding.
On Core 0, Task_A issues a Syscall_A,
and then the control flow transfer to the kernel thread
to handle the system call.
While it is running, the user thread on the Core 1
triggers an interrupt and the way it triggers the interrupt
is indirect interrupt using system call.
Once the interrupt is triggered,
its Core 0 will execute the interrupt handler.
And after the interrupt is handled,
it will return back to the system call routine
which handles the system call.
So there are different ways
to indirectly trigger a system call,
indirectly trigger interrupt using system call.
One way to do it is to trigger a hardware interrupt.
So this can happen when a program issues
a connect system call.
The CPU that is dedicated to handle network interrupt
will get interrupted.
Another way to trigger this interrupt
is called inter-processor interrupt.
This can be done by issuing end protect system call.
So once the end protect is issue,
the memory page permission has changed,
and therefore all the CPUs that are caching
those memory permissions need to be updated
with the right memory permission.
The scheduling primitive that we use, one is a scheduler.
This will change the scheduling priority
of a particular task.
This is optional in the exploit
because for system calls with a longer TOCTOU window,
such as networking system calls,
we find that it's not needed
to reliably exploit the TOCTOU issue.
But with system calls related to files,
the TOCTOU window is typically smaller.
And with the capability,
we can a hundred percent reliably exploit the TOCTOU issue.
And then the second primitive we use is set affinity,
which will pin a task to a particular CPU.
Okay, so let me talk about
the step-by-step exploitation in detail.
Initially, we need to do some setup.
So we set up three threads, a main thread,
a userfaultfd thread,
the userfaultfd thread can run on any CPU,
and also an override thread.
So the main thread will pin to CPU 3.
The choice of three here is because we run our experiment,
one of our experiments on a four core system,
and CPU 3 is used to handle the networking interrupt.
But if you're using IPI interrupt, it can be any CPU.
And then the main thread will map a memory page A.
The page is not allocated and it will register
a userfaultfd thread to handle the page fault
generated for this page.
On the override thread side we pin it to a different CPU
because we want to reduce the interference
between the override thread and the main thread.
And then the override thread will just block
on condition mutex once it start.
After the setup, the main thread will trigger a system call,
in this case openat,
and it will specify the filename argument
to point to page A.
Now keep in mind that page A, at this point,
is not allocated.
And so the kernel thread would trigger a page fault.
Once the the page fault is triggered,
the userfaultfd thread will write page A
with the malicious filename
and then it will release the condition mutex.
And then it will issue out a system call
to return execution back to the kernel.
Now, once the conditional mutex is released,
the override thread must start running.
And it will first write a benign file name.
In the last stage of the attack,
once execution returned back to the kernel,
the kernel will use copy_from_user
and you will get the malicious pathname.
This is a time-of-check.
And it keeps executing until a sys_exit.
The trace program will read the register
and reference the file name value again.
This is the time-of-check.
So this is the TOCTOU window we have.
And let's see how we can use the override thread
to make this override successful.
After it writes a benign name,
we issue a memory consistency update using CPU instructions.
And as the memory consistency update takes place,
we want to increase the TOCTOU window
such that we have enough time to update the value
for all the CPUs.
So what we do is that he will issue
a interrupt using system calls.
And this will increase the TOCTOU window size.
And then the memory consistency update applies.
So the filename now points will be a benign filename.
And then the tracing program will read out
the benign filename and check it against the rule,
which will not fire.
And now let's take a look at the demo.
Okay, so in our first demo, we're going to attack Sysdig
using connect system call, as an example.
So what we'll do is we will run Sysdig,
and ask it to monitor connect system call,
and just filter the data on our attack program.
And then on another window we will use TCP dump
to monitor all traffic going to port 80.
And now we'll run our attack program.
The attack program, it will connect 1.1.1.1,
then you will override this IP address
with a IP address from LinkedIn.com.
Okay, so we can see that the TCP dump
report traffic actually go into 1.1.1.1.
This is what happens on the wire.
Let's just stay where we port at.
Okay, this program, attack_connect,
is actually connecting to 13.107.42.14,
which is the IP address on my LinkedIn.com.
And therefore we showed that we can write this
with arbitrary value.
Okay, so now we'll show a second demo.
In this demo, we'll show an attack on Sysdig
on the openat system call.
Now our demo set up is slightly different.
So for openat, the TOCTOU window is small.
So if we don't use a Capsis nice,
sometimes the attack will fail.
But if we are assigned Capsis nice,
we can achieve 100% reliable exploitation.
Keep in mind that because the override thread
has the highest scheduling priority.
So sometimes you can write the filename faster
than the kernel reads it.
So when that happens the system call
will get the benign filename,
and the Sysdig will also get the benign filename,
so there's no harm being done,
and no alert being fired as well.
And therefore we've run the experiment
15 times using our script.
And each time we check what is the syscall rate
and what does Sysdig read.
And then we'll compute the results
based on each observation.
So what we'll do is we'll print out
the number of success attack,
the number of attack that has no harm,
and also the success rate.
Okay, so as you can see here,
in the 15 experiment we succeed three times, no failure,
and 12 times there's no harm.
And therefore, the success rate is 100%.
Okay, so lastly, we'll show a demo on Tracee as well.
And here we'll attack a connect call in Tracee.
The attack set up is very similar
to what we have for the Sysdig.
Again, Tracee will monitor the connect system call
and will just filter the data on our attack program,
reported by Tracee.
And then have TCP dump monitoring on port 80.
And then we run the attack.
As you can see, the actual traffic
going on the wire is going to 1.1.1.1.
But Tracee reported that the traffic
is going to 13.107.42.14, which was overwritten by us.
I just want to reiterate that the CAP_SYS_NICE is optional.
It depends on the system call that you are attacking.
And for networking system calls
typically this is not needed.
And for file base, typically this is needed
to achieve a 100% reliable exploitation.
And then I'll talk about the second attack,
the semantic confusion.
The idea of the attack is fairly simple.
The kernel and the tracing program
can interpret data differently.
So we use file link as an example.
When the kernel reads a link,
it will try to resolve the link and read the actual file.
But when the tracing program read the link,
it will just take the link as the argument
and use that to track in the...
So Falco is vulnerable to this semantic confusion attack
because it didn't resolve the link in the system call.
There's no CVE given because they mentioned
that symlink and symlinkat, link and linkat
are all monitored by Falco.
But practically, detection team need to track
all these links at or link, linkat
to all these file-based rules
if the attacker is using those.
And Tracee is not vulnerable to this attack,
because it uses a mitigation in RSM hook,
which are Junyuan will talk more in a later slide.
So I just want to quickly show the example
of Phantom v2 attack in Sysdig.
Remember this rule that we talk about at the very beginning.
And see that the filename, actually,
we checked whether the filename is /etc/shadow, okay?
So in order to exploit this,
we can create a file link, /tmp/shadow,
pointing to /etc/shadow.
And then the tracing program will read
the same link, /etc/shadow.
And then the system call openat sees the /tmp/shadow
and track against the rule.
It doesn't match /etc/shadow,
and therefore the rule is bypass, okay?
So with that, I will hand over back to Junyuan
to talk about mitigations.
- Yeah, so for mitigations,
there are basically two approach for the Phantom attacks.
So one is to detect the potential exploit is happening.
This was proposed and partially implemented by Falco team.
It has include in Falco's release in version 0.29.0.
Basically it's trying to detect
the variant behaviors used by the exploit.
For example, it detects the privileges use
of userfaultfd system call.
This was implemented and detect
a user register's memory address range,
and also detect a user copying a continuous memory chunk
into a use of the register range, and so on.
The second way to mitigate Phantom attacks
is to read the data actually used
by system call or the kernel.
In order to do that, you can hook LSM functions
to get the doses and call data that is actually used.
The LSM hooks function is a list of checkpoints
that are placed in a kernel
before operations happens on the objects.
So here is the table showing a list of the LSM hooks
used by Tracee version four.
And the second column shows the system call
that is protected by the LSM hook from Phantom attacks.
You can also read those data
that's used by the kernel from kernel data structure.
For example, in order to read the arguments of execve,
tracing programs can read it
from the mm structure from the kernel.
I will hand over back to Rex to conclude.
- Okay, so basically in this talk,
we showed that Phantom attack is generic
and it exploit the fact that kernel and tracing program
can read data at different times.
This is exploited by Phantom v1.
And we also showed that the Phantom attack
can exploit the fact that kernel and tracing program
can interpret data differently, is exploited by Phantom v2.
We demonstrated that kernel raw tracepoint on system call
are not ideal for secure tracing.
And for other tracing implementations, such as kprobe,
because it'll be vulnerable
if it is not implemented properly.
For mitigation, one can use detection
for abnormal usages on userfaultfd,
or to ensure that the kernel and the secure tracing program
reads the data and interpret the data in the same way.
- If you're interested in discuss further,
feel free to contact me on Twitter
and we'll share the GitHub link at Twitter as well.
So before we conclude,
we also want to thank all these people during our research.
Chris and Joel, for the discussion
on the eBPF and kernel tracing on TOCTOU.
Also Yu on TOCTOU.
And lastly, we really appreciate the Falco open source team.
They are very professional handling the issue.
We had really good discussions on there.
Thank you, everyone.