- Hi everyone.
Thank you for joining my talk, Fuzzing Linux with Xen.
My name is Thomas Lengyel
and I'm a Senior Security Researcher at the Intel.
And I also maintain a variety of open source tools
such as the Zen hypervisor, LibVMI, drop roof.
And I also participate in the Honeynet project,
where we usually wrong Google summer code
projects during the summer developing open source tools
to fight against malware.
What this talk is about is that we had this task
of fuzzing the device facing input points
of several Linux kernel modules, kernel drivers,
and we had to build new tools to get it done.
We open source then, we found a bunch of bugs and fix them.
I will talk about those,
but really the point of this talk
is to show you how we did it.
So you can go out and do it yourself.
To start, let's talk a little bit about feedback fuzzers.
They are not just about feeding random input to your target.
They use feedback as a mechanism
to better exercise your target code,
and they do it by effectively collecting the execution log
or called the coverage.
Then you're running the fuzzer and you can use that
to compare execution from run to run,
to determine if the fuzzer
was able to discover some new code
that hasn't been seen before.
The idea is simply that if you've discovered some new code
region that hasn't been exercised before,
it's worthwhile to focus on that
because the chances of finding some new bugs is higher
on code that hasn't been exercised as much.
Obviously what feedback fuzzers need the most
is determinism.
If the target code behaves radically differently
from one run to the other,
then the fuzzer might just get stuck
and focusing on inputs that don't actually lead anywhere,
because it will think that it's opening up new code bats
when it's in fact just noise.
So if you have garbage in, you will have garbage out.
So Xen VM forking is supposed to address that.
Shortcoming it is effectively a way to add determinism
to kernel code execution.
If you think about the kernel, it's pretty on deterministic.
You have interrupts firing all the time.
You have multiple treads and scheduling
it's, you know, as far away from deterministic
execution as you can get.
So VM forking allows you to split the run time of a VM
into multiple VMs and populate the memory
of these fork VMs from the parent one.
So to make this as fast as possible,
you effectively can just run the fork
is executing a memory access
where it's a reader an execution.
You can just populate those page table entries,
using the fork with a shared page table entry.
And you only have to actually duplicate the entire page
of memory if the fork is writing something into memory.
To get even better speed,
once you have a fork medium set up,
you can actually just reset to the state of that fork VM
just copy the VCPU registers from the parent and throw away
any of the duplicated copied pages,
but keep the shared pages in place
that will get you the best performance.
If we take a look at numbers,
if you run these operations in a tight loop,
you can create about 1300 VMs per second.
If you're doing a reset,
that's about 9,000 resets per second.
So these numbers are fairly okay for fuzzing.
Obviously you will not see these numbers
because these are the theoretical max.
If you're doing nothing just resetting the VM,
obviously between those resets,
you actually want to run your target code.
A couple other buildings the blocks
dimension here to really understand
what we will be doing is most importantly,
Xen's introspection subsystems.
So this is what I've been working on for the last 10 years.
If it allows you to really pick into the wrong time
executions state of a guest,
you can read and write the memory
translate virtual addresses,
but it also allows you to pause the VCPU
of the VM at various hardware events
and get a notification of those hardware events
in your regular user space application in dom0,
which makes development of tools,
introspection tools really quite convenient.
You can get a notification for CPU IDs breakpoints
from single stepping EPT faults and a bunch of other things.
The other really cool feature that just got upstream
into Xen it's called VM trace.
We did this in collaboration
with the polish CERT.pl & Citrix and Citrix,
and this is an effective way
of turning on into a processor trays
to record the execution of a full VM from dom0,
where the CPU itself will store enough information
about the execution of that VM and with low overhead.
So that later on that log can decoded
to reconstruct the execution of that VM see what happened.
And obviously this is what we will be using
to collect the coverage information.
So if you look at the full flow of how the fuzzing setup
is working on Xen, if it start from the parent VM,
you boot up your regular VM,
and then the target code is reached
you compile the target code with the magic CPU ID in place
so that we will signal to the fuzzer
that this is the point you want to start fuzzing.
The fuzzer will find that CPU ID when it's executed
we'll create a fork,
which we call the sync VM, in that sync VM
we'll look up the virtual address
of various kernel functions
that usually get called on something bad is about to happen
in the kernels, such as a panic is happening
kasan or ubsan, a built-in error detection systems
in the Linux kernel trip.
We add a break point to the entry of all of those functions
and we create another fork.
This is what we call the fuzz VM.
This is where we will actually be performing the fuzzing.
This works effectively by taking the input
that's generated by the fuzzer.
In this case, we are using the AFL American fuzzy lop.
We take the input that's generated by the fuzzer
and we write it straight into the VMs memory.
I'm positive to see what happens if we catch a break point,
that's going to be at one of those entry addresses
that we break pointed earlier.
Great, we just found a crash
that we would report back to AFL.
If we catch the magic CPU ID again,
that would be the end harness.
So you have a start harness and then the end harness,
if you hit the end harness,
then you know, nothing bad happened.
If neither, then we report a time-out.
Afterwards, we can take a look at the Intel processor,
trace log, decode it and we use that to report the coverage
back through AFL so that if we don't understand
if something new happened while fuzzing that VM,
and then we just reset the state
and go to the next input from AFL.
Let's take a look at the demo
of how this actually looks in practice.
I am creating a new LUBUM 22004 VM
and I will be booting, UBUM to a Linux five at 10 kernel
that has the harness already compounding to it
and booting that DSLR and PTI disabled,
just to make debugging easier later on.
And what we will be fuzzing is a USB driver
and that's Kernel.
I effectively have it thumb drive USB stick
attached to a USB, 3 hub, and I fire up the gefX fuzzer
to listen to in that magic CPYD
which in here is called magic Mark is executed
so now he's just listening to see, even that CPYD happens,
the VM finished booting,
If you a log in and really initiate some interaction
with USB thumb drive that will trigger
that harness I have pre-compiled in there.
So I ran F disc and you see that, F disc
never returned, it never finished.
And that is because that VM is now paused
because KFX caught that CPYD
and we have the information about where the target buffer
in the target sizes that we want to fuzz.
We can go to that virtual address
read out that memory to be used as the seed for the fuzzer.
So this is the normal execution
would be that the Kernel was just about to do
while executing the F desk,
and we will start new dating from that structure
to see what can happen if that input
is malformed or malicious,
we will be using the F++ here,
fuzzer is up and running.
We are opening up a bunch of paths, as you can see,
and in less than a minute, there is already a crash found.
And we'll go into the details
of what's the discretion about,
but this is actually a real bug in the Linux kernel.
That was discovered just like that.
So at this point, and you're probably wondering like, okay,
what the hell did we just fuzz?
You know what the bug is?
And you're right.
There is more to fuzzing than just running the fuzzer.
In this engagement, we discovered that
really the biggest pin points is not running the fuzzer
once the fuzzer is up and running.
It's great, you can go and take a walk, grab a coffee.
It's awesome you don't really have much to do.
It's all automated the real pin point is
performing analysis,
figuring out what the fuzz in the first place.
And then once the fuzzer finds something,
getting enough information out about the crash
so that you can create a report or fix the bug.
So how do we do all of those steps?
So let's start with analysis.
Once we're fuzzing, there is DMA, and DMA
is memory that the kernel makes accessible
to an external device.
And this is to facilitate fast IO operations,
shared memory, you have have better speed.
And the way this works is that the device has direct access,
to that memory so that it doesn't have to go through the CPU
and the MMU to actually read or write through that memory.
There is what's called the IOMMU,
but usually the IOMMU restricts access to other pages,
pages that are explicitly made accessible by the kernel
to the device, will be allowed to be accessed by the device
through the IOMMU.
So the IOMMU is not going to protect you against
a malicious device that is placing random stuff
on the DMA page that it's allowed to do.
So we figured, okay,
let's take a look at the Linux source code,
see where DMA memory is getting excess.
It should not be too bad.
Like when you have a system called what the Kernel is doing,
when it's received some buffer from user space,
the first thing it does, it copies it into kernel memory
and does its processing there.
So we figured, well, that's how DMA works as well.
The kernel should copy the DMA memory
first on internal buffer and go from there.
But boy, were we wrong?
It turns out that the Kernel is accessing DMA memory
all over the place.
There is no single function that copy from the DMA,
once DMA memory is established,
the Kernel can access that and those access it
all over the place.
So even just figuring out where Linux reads from DMA
is not trivial.
So what we did was we looked through the source code,
looking for hints of when the Kernel
might be doing the DMA read.
And it's quite painful because, you know,
just by looking at the source code,
you don't necessarily know whether some pointer
is a DMA memory or not.
So what we did, we looked for the iron man cookie
or the best one was actually to look at
the DMA conversion functions that go, you know,
dig in the, a little into to CPU.
That is a pretty clear indication that, you know,
the memory that is being,
or the data that is being read might not be in the right
and the format that the CPU expects.
So that usually means
that there was some cross-communication
with an external device
and then take the output from F trace,
which is a built-in subsystem in Linux
that allows you to trace
the execution of the kernel internally.
Cross-reference what we found
that we think is DMA access and see,
but better those functions
actually get called during execution of the kernel.
Cause we found a bunch of these excesses,
but they, those were functions
that never actually executed during run time.
And those are not really good targets to fuzz
because if you can get the code to execute,
then you can fuzz it.
So this was not great.
So we also decided to just, you know,
be old school and dig through the spec,
maybe, you know, we get a better understanding
of what's going on here because the kernel code
is not the easiest thing to read.
So looking at the spec itself,
it kind of find pictures like this
that are immensely helpful to, you know,
try to understand what the hell is going on.
Obviously this subsystem, as you can see is quite complex,
but really the biggest boost that we got for our engagement
here was to just discover what the name of the rings are
that this subsystem uses
for device to kernel communication,
and these are events during transfer rings
in the command ring.
So just knowing those names,
we were able to just, you know, crack the,
there is a variable called the event ring
and see where that is being accessed.
So what we found is this location very, yeah,
there is, what's called the event ring.
And this is a function
that gets called from the interrupt handler.
Obviously what happens is that the device or the USB hub,
it's placing some data on this ring and sends an interrupt.
And then the kernel goes and processes, whatever,
you know, structured the device sent.
And it de queued that from that derink page,
which is DMA accessible.
So what we just fuzzed about the RNs set up was
is just after that, you know, structure is dequed
from the DMA page.
We have the harness start and we transferred information
about the where the pointer is
and what the size of the structure is.
And then we have a couple points that we want to stop
the fuzzer and go through the next iteration
effectively, whenever this function would return,
we want to stop the fuzzer.
We want to fuzz everything that's in this function
and whatever this function calls
as for about those harness functions
actually look like they are really just CPU ID instruction,
where you stuff the magic information
into reducers that the user space tooling
dom0 that can receive.
So these are effective,
you can think of them as hybrid goals.
All right, so once we found that bug,
you know, what's the next step,
with me, on forks are a little special
on Xen in that they are not fully functional VMs.
You can turn them into fully functional VMs,
but for fuzzing it's obviously there's no point,
but because of that, there's, you know,
a little bit of a pain and actually figuring out
what happened in them because you don't get to just log in
and get her to logs because there's no network,
there's no disc, no console, there's no I/O into VM forks.
They're literally just running with CPU and memory.
But fortunately, the D message buffer
that the Linux kernel uses to store information
about run time events and errors,
and whatnot is just sitting in Ram.
So we can go and carve it out.
The way we do that is we're gonna use GDB SX.
That's been shipping with Xen for over a decade
at this point and it's really just the minimal GDB bridge.
If you build the kernel with debugging information
and frame pointers, you can access the information
of the kernel state using just GDB.
So let's take a look at how this works.
So this is where we were.
We just found a bug.
We want to figure out what happened
with what is the bug?
So we will take KFX.
We will re execute,
but instead of taking the input from AFL,
we will just use that file that was found by AFL.
We will inject that into a VM fork
and see what happens with the debug output.
We see that, okay, ubsan prologue tripped.
So ubsan prologue is the function
that gets called when the kernel starts to construct
the ubisan report.
So we want to stop the VM fork
after it actually finished printing the ubisan report
into the D message buffers.
So we go stop ubisan epilogue.
And at that point we can really just, you know,
attach the debugger, do it,
then read out the message buffer.
So I fire up GDBSX attached to that domain,
go into the source folder where I compiled that kernel
load up the symbol file for the kernel
attached to the bridge
and print the D message buffer using LX-D message.
And there we go, right at the bottom
of the message buffer received the report
that ubisan generated for the bug
that the fuzzer just found.
This is an array index out of bounds there in XCTI ring.
Awesome, so this is pretty much how you,
three-ish the errors that you've made using the fuzzer
for most of the cases, this has been perfectly sufficient.
You have the source line,
but we have to take a look at usually it's pretty
straightforward of where the bug is coming from, but not,
not all the time.
Sometimes the bug three girls in code that's far away from
the driver that we were actually fuzzing, right?
So there is some call chain from the start point
that we are fuzzing from that reaches some deep layer
of the kernel and that's where some bug happens.
Then figuring out what's going on,
there is a little bit more difficult.
So let's look triaging beyond the basics.
Here's the harness that we use to fuzz the IGB
network driver, so these are network drivers
that receive packets and packets.
Here we have an interrupt handler
or final package is received by the kernel
and the kernel goes and reads this RX
description, buffer that the device, places on the ring
that has information about, you know,
what size of the packet that was just received this.
So this is not the packet itself.
This is metadata about the packet
that the device itself constructs.
And we want to force that.
So what we do is we jump in just after that RX description
buffer was received from the ring.
We will start fuzzing there, and we want to stop fuzzing
when loop, loops around.
We also have a harness stop when that loop breaks out.
That's not shown here.
So using this harness, we found the following bug,
we get a kasan loop pointer D reference
in a function called gro_pul_from_frag0.
All right, for the, you also get a helpful stack
called trace wherever you see that, you know,
there's kasan report.
And just before that, there's a mem copy.
So gro_pull _from _frag0 calls, mem copy.
All right, let's take a look at their function.
It turns out that this is not in IGB itself.
This is in net/core/dev.c
So this is some deep layer of the Linux networking stack,
very, receives this SKB buff structure,
and does a mem copy from one place to another.
We have no clue what those are, but this mem copy,
obviously trips on Upointer D reference.
So it's either the source or the destination
is corrupt and it got corrupted because, you know,
the fuzzer found a way to corrupt it.
So at this point, the idea I had
was all right let's take a look, you know,
which one of these pointers is the corporate.
We would want to stop the execution of a VM fork
for just that, at the mem copy.
So we would be able to take a look at the state of the VM
and the registers that contain those pointers
to see which one is no.
So we want to stop the execution at that mem copy.
The way we do this, it has a couple of steps,
and we need a couple bits of information for it.
First of all, we want to figure out,
you know, what it is the address of kasan report,
because that's where we want to execute
that VM op tail using single stepping,
because just before kasan report is reached,
obviously we will have the mem copy.
So what we do is we just execute
the VM with that crashing input.
And we see that kasan report is indeed tripped
and we see the virtual address
of that function kasan report where that is.
So we want to, at this point, create a VM fork
place that buffer that we know
that will trip kasan report into that memory.
So we will use this tool called RW mem,
and we were able to ride the contents of that file
into the target buffer.
And that will allow us to execute this VM,
to reach the crashing input and record what happened.
Obviously we could use processor trace as well for it,
but I found single stepping to be just as effective.
And it's a little less convoluted.
So now we have that VM fork set up with that crushing
input the injected into this memory.
So now we just use the tool stepper
that will who's MTF, single, stepping to go all the way
and stop at the virtual address of kasan report
when that's reached.
And we will put that output of that into a file.
And if you take a look at what this file contains,
so this is effectively just the,
this assembly of each instruction that executed,
and this reaches kasan report at the end.
So there's a ton of instructions in there
just looking at that is not all that helpful
for the task that you're trying to do.
But what we can take from that is
really just dig the instruction pointers that were observed
and translate it using the kernels debug singles,
using address De wine,
that will actually get as the source lines
of what each of those instructions actually are.
So now, if you take a look at this decoded log,
what you'll see is that each instruction pointer
and what you know, source line corresponds to.
So at the bottom of this file, we see immediately,
are XO gro_from_pull0 is the mem copy from flag zero
is, and there is the mem copy that trips,
the move pointer D reference.
So I just take the last instruction.
That's still at the mem copy before kasan report trips.
And I want to re execute that VM to stop at that men copy
and the last instruction in that mem copy
to be able to see, you know, what the register state is.
So again, I just got a fork.
I use RWN same way I did before
and just change the domain ID.
And now I want to stop on this address,
which is the mem copies address.
I don't actually have to save the output
because I know where it's going,
but now this domain ID 61 is paused at that mem copy.
So I can just go and take a look at the register a state,
take a look at the source pointer or OSI
is the register that holds the source pointer,
in this case, an RDI pulls the destination one.
Oh, well, it kind of looks like both source and destination
in that mem copy is a no pointer.
So they're both corrupted.
So this approach did not really yield us anything
that we could use to figure out what went wrong
since it looks like that entire SKB buffer is just bogus.
So what else can we do?
Well, the idea is that, well,
if we can't figure out just the mem copy what went wrong
at the location where the bulk trips is,
we can compare the execution that goes to kasan report
with the execution that is normal.
So we can take the normal input that was used as the seed.
We know that this is the input that the kernel
would have executed with normally,
and that does not cause a crash.
We see that it reaches the harness signal finish.
So we will just create a fork from that
use stepper to go all the way to the end harness,
just as we did before, you'll stop on this address,
save the output to a lock file.
Now we can take the instruction pointers
from this lock file, decode it,
using the address to De Wine.
Save that as well.
And now we have the decoded log
for both the execution that goes to kasan report.
And that goes to the end harness and we can just diff them.
The very first line in this diff
is going to be, or the execution diverge
is from the normal one.
And we have the source line, and just go straight there,
look at the code and bam,
this is the first line in this execution that only happens
when the fuzzer with the input that the fuzzer found.
So it turns out that the SKB buffer
is constructed by the driver
and it's being passed to, you know,
those deeper layer kernel subsystems,
but the way it gets constructed here
is based on information
that came from that RX description buffer and it's bogus.
So obviously what needs to happen
is that even if the Aurizon description buffer
says that, oh, there is this bit set.
There needs to be a little bit more sanity the checking
in place before that SKB structure is manipulated.
So if you actually look at the latest kernel code,
you'll find that this code has been fixed
and it's effectively,
it was just missing a sanity check on data
that was coming from from DMA.
All right, let's look at a couple more bugs just for fun.
Can you spot the bug here?
How about this one?
If you ever notice yet, but kind of the team of these bugs
is about the same.
You got some DMA source,
input that is used without input validation,
and just is used for whatever the kernel decides
in this case, for example.
This is used as the slot ID is derived from DMA memory
and is used for the RA index.
Well, what can go wrong there?
So yeah, we found nine null pointer D references,
three array index out-of-bounds.
We found some infinite loops in the interrupt handlers,
but also (indistinct)
the kernel contributed user memory access as well,
which is not great.
And these are all pretty much stem from the same problem
and that the kernel does not treat DMA memory
as a security boundary,
DMA memory is kind of treated trusted.
And consequently, it means that all of these devices
are treated trusted.
And when you are talking about USB devices,
well, it's not great that all USB devices
are treated like that.
Not a problem case that we want it to look at
is when these kernel codes might perform double fetches.
Double fetches is effectively a race condition
where you can read, you can have problems
where you have time of check, the time of use,
where the kernel is performing
even if it did perform some sanity checks
on memory that's DMA.
The problem would be that even if you do sanity checks
on DMA accessible memory, by the time you finish,
your sanity checks, the data might have changed underneath
because the device has access to that same memory.
So if it wins the race, you might finish security checks,
but the data is still corrupt.
So obviously we would want it to detect if that happens.
And the idea was to, okay,
let's remove EPQ permission from DMA pages
and just create a record of run DMA pages
are being accessed.
And if it's, you know, if we get a page fault
where the kernel is reading some address from DMA,
and it's the same page, be the same offset twice in a row.
That's the strict as the definition of a double fetch.
We can detect that and report that as a crash through AFL.
So we can go and take a look at the code to see, you know,
whether to the double fetches, a security concern.
We taught it would be rare,
but it turns out that it happens all over the place.
Some kernel drivers, three DMA memory, as you know,
totally trusted.
So they would just keep going back
and fetching the same memory left and right,
but so far we haven't found a strictly
speaking security issue
because it turns out that, you know,
the same byte is being fetched,
but different bits are used from that byte
so, far it hasn't looked dangerous,
but obviously this practice of just treating DMA memory
is bad and it needs to change,
but we've received considerable pushback
from kernel maintainers for various reasons,
performance regressions, you know, fewer of regressions,
but ultimately to close this class of bugs,
DMA memories should really be treated the same way
as user space memory is like,
every DMA memory should get copied into a local buffer
before, you know, being used by the kernel.
And that's absolutely not the case today.
All right, so we found a bunch of bugs.
We fixed them mission accomplished.
Not so fast.
As you recall, the way we found the DMA input points
was just through reading the source code
and doing some experiments with afterwards,
but there was this lingering feeling of like,
hey, did they really discover all DMA input points?
Like, you know, what data do we have to back that argument?
And I mean, we had a bunch of people look at the code,
so that gave us some confidence, but, you know,
we can put a number on it.
We also got bogged down by, you know,
just documenting all the bugs we found.
And at some point it just became nonproductive
to keep staring at the code because it was just annoying.
So, let's do better.
This tool called DMA monitor.
We added to the project to be a standalone EPP
fault monitoring tool.
So this effectively came after,
the double fetch deduction code was added.
And the idea is that if we can already detect when DMA
is being accessed for double fetches,
well, we can use the same approach to detect when DMA
is being accessed that all.
We can really just trace who is accessing DMA
and where by using EPP faults.
The only thing we need is to know where the DMA pages are.
Fortunately, the Linux kernel has its own internal DMA API
that all kernel modules
should be using to set up DMA for devices.
And then that there is a function that is used to allocate
memory to be used for DNA, DMA LL contributes.
We can hook that function using a break points
through the hypervisor and hope the return address
when the function finishes,
and that will get us the virtual address of all pages
that the kernel uses for DMA.
And then we can just remove the APQ permission
for all those pages on the fly
effectively, giving us a way to log all the coats sides
that read from DMA as the kernel is running.
So let's take a look at how this works in practice.
I'm booting up a, the same VM
and on the right I'm firing up DMA monitor.
I just tell it, you know, what is the domain.
What is the debug Jason of the kernel
is the EMI Alec attributes is hooked as the kernel
is booting and then we pretty much immediately
start to see a ton of DMAX as is happening
as the currently still booting.
As you can see, there are quite a few pages
allocated for DMA,
and we can grab through that log
and see when it's, you know,
the access is just to read access.
We can take the instruction pointer for each
sort through them and just take the unique ones.
There's still a ton of them, but, you know,
we can feed this through addressed line
to getting you know, explicit list of all the places
that the kernel touched DMA from.
So, this is quite a few places,
but at least now we have an explicit list that we need to go
through and take a look and see, you know, better.
The data that is being read from DMA at these locations
is complex enough to warrant fuzzing, which is awesome.
They didn't have to look at the source code
to figure out where to start.
So this is, you know, miles better
than what we were doing before.
Because we have the list that we have to take a look at
instead of having to keep, you know,
parsing everything in the kernel
to see if that's DMAX to not.
There were some still corner cases,
the DMA monitor either though it's way better
than what we were doing before.
And that's because some of the times the DMA access
that's the kernel is doing is just reading
something from DMA and stashing it some structure
and returning, and then the kernel is going away
and doing something else.
So we were like, okay, well,
where is that data going to get used after the DMA access,
nothing warranted fuzzing,
but that data is still sitting now in the kernel
in private kernel memory,
but it's still can be potentially malicious.
So, you know, where is it getting used and is it safe?
And we were like, well, you have no idea.
We didn't want to go back and reading the source code.
Cause it's very hard to follow that type of data, you know,
life cycle in the kernel.
And it's very hopeful and it's very manual and annoying.
So that's where this next tool idea came from
that we call full-VM taint analysis.
The goal is to really just track the tainted data
propagation into the kernel.
We know where the data is coming from.
We have the source, that's the DMA access.
So we want to obtain that address and track, you know,
what the currently is doing with that data
and where that data lands
and when you know how it affects
the execution of the kernel.
We old views of young trace inter processor trace record
the execution of the kernel with very low overhead.
And after some time,
replay that record of the instruction stream
through the Triton DBIS state engine.
So that's a separate open source project.
That's really awesome to be integrated with.
And that will tell us, you know,
what the instruction pointer is get tainted by the data
that we just read from DMA.
And that will tell us, you know, all the locations
where the control flow of the kernel
depends on tainted data.
So let's take a look at this as well.
Here's a VM fork that I know will perform with the VM
access at this page.
So I have you'll fire up DMA monitor on it.
I don't pause it then just be seen
you know, there's a single DMA access to that page
where something was read out from the DMA
and stored somewhere in the kernel.
So if you don't know, you know,
at this point like where else that data is getting used.
So now the idea is to use VM taint to figure it out.
Maybe he'll use VM taint to save the state first.
So this saves the stack and the resistors
of the starting point into a file
that we will need for the taint engine
to create another fork.
We start the collection of the processor trays buffer.
If I get into a file
and on pause the VM fork,
now it's running and it's recording the execution
into that buffer.
Now, we'd let it run for a second or two,
we pause it and we can start decoding that processor trace
and feed it through the taint engine
type into taint to, the taint that log file
and take a look while that is processing
what it found so far?
So right off the bat, we see, you know,
where that move copy of the data,
what register got tainted and from there,
you know, where else, what else got tainted
during the execution of the kernel.
And there we go, right of the bat, we can see, you know,
all the different instruction pointers
that got tainted from just that single DMA access.
And if you do this, you know,
for the boot of the kernel,
you can really check the full life cycle of DMA source data
through the execution of the kernel.
Without even having to open up the source code of the kernel
and just giving you right away, all the locations
that the control flow might depend on tainted data.
You go take a look.
If it's, you know, it looks complex enough,
you put the harness around it and you can start the fuzzer.
So this code is released as well as everything else.
Most of the code is upstream in Xen,
but this code you can grab from GitHub.
There're also a couple of goodies that I wanted to mention.
This is pretty new some of the targets
that we wanted to fuzz kind of difficult
to get working in a Xen VM.
So we came up with this way of being able to transplant
and the state of the system from one hypervisor to another.
So in this case, we can take the,
take a snapshot on QEMU/KVM or Simics,
and load it up on Xen because VM forks
really only need the CPU state
and the memory of your targets to be fuzzable.
So you can use all of those different hypervisors
to take a snapshot and loaded up on Xen and falls away.
A couple of things we want to work on next
or already are working on
top of the list is automation putting an end to end,
automated fuzzing system together
is what everyone is asking about.
So that's absolutely something we are looking at
would be also pretty awesome to capture systems
that using Intel DCI, which is a USB three bays debug
connection that you can attach through a bare metal system
and capture the full system state.
This would allow us to, you know, really fuzz any code
that runs on any system, including bios and SMM code.
So this would be pretty cool.
Well, other idea we have is creating the sense
if the ring zero mode tool adding nested virtualization
support, so we can force hypervisors,
obviously with Intel DCI.
We would be able to capture hypervisors state as well.
So that might not necessarily be a requirement,
but, you know, still would be cool to have.
And a couple of things I didn't cover in this talk
that are already possible using VM forking and all the tools
that are available open source,
like fuzzing, other operating systems, fuzzing Xen,
user space binary's are absolutely something you can,
you know, fuzz with this system, black box binary's
you know, even malware.
So if you're looking for ideas,
here's a couple of things that are already possible.
So thank you, that was my talk.
If you have any questions or comments, please reach out
and thanks goes through a whole bunch of people
who made this work possible.
So this was not a single person's job.
This was large teams working on this.
So thanks everyone for your involvement
and absolutely for the open source community
for releasing all the tools that make, you know,
rapid security development like this possible.
So, thanks, I hope you found some cool information
in this talk and you know,
the goal here is to get you to go out
and go fuzz the kernel, because we found some bugs,
but you come back there are more to be found.
So thank you.
Looking forward to your questions.