- Hi, I'm Laura Abbott,
and I'm here with my colleague Rick Altherr
to talk about breaking TrustZone-M using
a privilege escalation on the LPC55S69.
We are engineers at Oxide Computer working
on building a new server.
Computers haven't changed much in many years,
and we are looking to fix that.
No, I promise this isn't a sales pitch.
I'm giving background to explain what we were building
when we stumbled across this issue mostly by accident.
One of the reasons we are looking to build a new server is
to have some level of security assurance
at the hardware level.
As part of the server,
we are building a hardware Root of Trust using
a security-focused microcontroller.
All of our software is written in Rust,
and the goal is to have everything open-source
by the time the product ships
as we are big believers in transparency in software.
This will be a theme throughout the presentation.
We had some pretty specific requirements
for what we were looking for in hardware.
We need a strong assertion regarding the integrity
and authenticity of Root of Trust firmware.
We want to be able to use the Root of Trust
to extend trust to other parts of the system.
And if we're not confident
in what's running on the Root of Trust, we can't do that.
Part of building up that trust involves having
a tamper- and impersonation-resistant unique ID.
If the device does get compromised,
we need a way to reestablish that trust.
Other features you might think of,
such as cryptographic accelerators or key storage,
were considered pluses, but not strict requirements.
We did a pretty wide survey of hardware,
including programmable logic devices.
One requirement we didn't explicitly state,
but turned out to be pretty key,
is that the hardware must be available for us to purchase.
This ruled out more than one device.
We were also biased towards devices that can be programmed
with open-source tools, because again, transparency.
There was no one standout winner among all the candidates.
And based on our trade-offs we made,
we decided the hardware that best met
our needs was the LPC55S69 from NXP.
The LPC55S69 is
a dual-core ARMv8-M Cortex-M33.
Core zero has TrustZone-M
and Memory Protection Unit support for isolation.
Core zero does not have TrustZone,
but it can be restricted at the bus level,
similar to other DMA engines.
Rick will talk more about this in a bit.
There's cryptographic accelerators
for AES, SHA, and doing field math with ECC.
The chip supports secure boot using RSA signatures,
as well as Authenticated Debug
There's a PUF, physically unclonable function,
for use with keys.
Part of the flash can be locked down
after programming to prevent modification.
Other variants of the chip support CAN,
which is common in automotive use cases.
There are a lot of appealing things about this chip,
and it seems like it can be used in a lot of places.
But, of course, we have to ask ourselves,
why should we believe this is secure?
NXP presents this as a great chip
to use in your IOT and other edge cases.
And a lot of the written material sounds great,
but it's marketing material.
How does this actually work in practice?
Trust, but verifying all that.
What we found when we started working
with the chip was that the documentation was not great.
It was frequently pretty confusing.
And this was especially true when it came to secure boot.
This is the diagram for the user manual
for the LPC55 showing the boot flow from power on.
There's a number of steps here
and also ways things can go wrong.
When trying to make secure boot work,
we ran into some issues.
So naturally when the manual doesn't provide information,
you turn to the application notes.
And the application notes give
a slightly different boot flow chart
with a few more steps.
This updated flow chart mentions the NMPA,
which I'll talk about more in a bit.
It also mentions the Debug Mailbox,
as well as the recovery boot path.
These are two entirely new code paths
that the manual apparently
just didn't feel the need to mention.
This pattern of making a passing reference
to something significant or omitting something completely
was a recurring experience with the NXP documentation.
We continued to run into surprises
while reading the documentation.
As part of our due diligence,
we were reviewing all the application notes.
I found one labeled camera interface in LPC55,
which was very surprising,
given the manual does not mention
a camera hardware block anywhere.
The application note contained the line,
"There is a hidden coprocessor in LPC55S69
which can handle the signals of camera."
When I mentioned this to my colleagues,
they thought I was joking because a hidden coprocessor
or even a camera sounds like a bad idea
in a chip that's supposed to be secure.
We had to ask NXP for details about this,
what this thing even was,
because they usually don't give out this information
unless you're a high-volume customer.
We did eventually find out
that this core is named EZH.
It's a custom core
with a custom instruction set architecture really designed
to offload wire protocol conversions
and that don't need to run on the main ARM processor.
The documentation also contained a passing reference
to DICE CDI.
DICE stands for Device Identifier Composition Engine
and Compound Device Identifier.
This was a feature that comes out
of the Trusted Computing Group,
and is designed to give it cryptographically strong identity
to the chip based off of the first mutable code.
The idea is that if you change the first code
that's running, you get a new identity.
This is a really nice feature
for what we're building in Root of Trust
and being able to extend trust,
but unfortunately, NXP decided to defeaturize this
by removing detailed documentation.
The feature is still actually there in hardware,
you just can't know anything about it.
And then there's the ROM patcher.
But before we go into this part of the story,
it helps to have an understanding of TrustZone-M.
And I'll pass this off to Rick
to talk more about TrustZone-M.
- Thank you, Laura.
So what exactly is TrustZone-M?
According to ARM, the ARMv8-M architecture extends
TrustZone-M technology to...
I can't do this to you.
I got bored the first time I read this,
and I've practically fallen asleep every time I've tried
to read it since.
TrustZone-M is very similar to TrustZone-A,
which you probably know as just TrustZone.
This is the system we've had for a long time
on A-profile devices used in cell phones
and other embedded systems.
In TrustZone-A, the hardware is divided
into a secure world and a non-secure world,
and then you have multiple execution modes
within each of those worlds.
Now, in an M-profile device,
there's a couple of key differences.
The first is that you only have two execution modes
instead of the four that you would normally have
on A-profile device.
On an M-profile device,
you're gonna have handler mode,
which is similar to what you might have as an OS kernel,
and thread mode,
which is roughly what a user space thread would be.
The second key thing is that M-profile devices have no MMUs.
And there's no concept of virtual addressing at all.
In fact, everything is physically addressed.
And while you do have the option to include an MPU
in a given chip,
it depends on the chip,
and so you can't rely on memory protection
in any form in the chip.
So given all of this, how does TrustZone-M work?
TrustZone-A relies pretty heavily on the MMU,
so how does M-profile actually deal
with distinguishing between S world and NS world?
Well, if all you have is physical memory addresses,
you use physical memory addresses.
You divide up the memory space into secure,
non-secure, and non-secure callable ranges.
If you're executing from within a secure range,
you can do data accesses to any address in the system,
but your next instruction fetch must be
from another secure range.
Now, if you wanna leave the secure mode
and go into non-secure,
you have to use the special instructions BXNS or BLXNS
in order to indicate I am switching to non-secure mode
and therefore the next instruction fetch will come
from a non-secure range.
Once I'm in the non-secure range,
data accesses can only go to non-secure ranges.
Also, my next instruction either needs
to come from a non-secure range,
or it needs to come from a non-secure callable range,
and specifically be a SG,
or secure gateway, instruction within an NSC range.
And why does this exist?
Well, conceptually it's okay for a secure world
to arbitrarily call into any part of the non-secure world.
We trust the secure world more,
but a non-secure application shouldn't be able
to arbitrarily jump into any code path in the secure world.
So instead, you create this special NSC range
that you place these SG instructions in,
and that creates an explicit entry point
for the non-secure world to call into the secure world.
And in practice, an NSC range typically gets used
like a big jump table.
You have an SG instruction immediately followed
by a branch that goes back into the secure range.
Now, when you have an NSC region,
it more or less acts like a secure region.
The only special detail is
that it can actually have these SG instructions,
and it's where NS code can actually call into S code.
Okay, so how do we actually set up these S,
NS, or NSC ranges?
Well, it depends on which part
of the chip we're talking about.
If we're talking about the CPU,
you start with the SAU, or Security Attribution Unit.
This is similar in concept to an MPU
where you're actually defining a range
with a base and limit address,
and then specifying what security attribute applies
to that range.
Now, this is actually defined architecturally
as part of the ARMv8-M security extension
that defines TrustZone-M.
So if you have a part with TrustZone-M,
you're gonna have a SAU;
however, the number of regions available
to you is implementation-defined.
In practice, you would define four,
maybe eight ranges,
which is kind of similar to how MPUs often provide.
Now, one key piece about this is
that the default state is that any region not covered
by a SAU region, any address not covered by a SAU region
is implicitly considered to be secure.
So in effect,
the SAU allows you to mark ranges
as lower security state
than what they start out as.
Now, there's another piece
where the chip vendor might have reasons
why they want certain areas of the chip
to be considered secure or non-secure.
So there's the Implementation-defined Attribution Unit,
or IDAU.
And this is a piece of fixed logic attached
to the CPU core that does an address
to security attribute computation,
it's just that this is fixed by the chip vendor.
Now, in one of ARM's documents,
they provide an example implementation
where you just take bit 28 of the physical address
and use that to indicate secure or not secure.
Sadly, a lot of implementations actually just picked up
on this and ran with it, including the LPC55S69.
When you do that,
you end up with a memory address space
that looks like this.
You see that things like SRAM and code are a single device,
but they have double the address space,
and it becomes a secure and a non-secure region.
So this means that you actually have two different ways
of accessing the same physical device,
and we call those the non-secure and secure aliases.
Why is this important?
Well, when you do an access from the CPU core,
that address is fed to both the SAU and the IDAU.
Those independently compute what the security attribute is,
and then the most secure of the two is what's chosen
to actually do the restrictions.
So if you use the secure addresses...
If you use a secure range from the IDAU's perspective,
it can never be lower privilege.
The IDAU will always win as having it be a secure address.
And this means that you can write secure code,
a secure world code using secure aliases
and never have a chance of it being anything else.
On the other hand,
if you use a non-secure region from the IDAU's perspective,
you have to go look at the SAU
to decide what its security attributes actually gonna be.
It could be secure, non-secure, or non-secure callable.
And this all depends on what the chip vendor actually did
as far as what the IDAU layout is,
and then the application
as far as what the SAU configuration is.
Okay, so what happens once you have that request
and you've computed what the security attribute is?
Where does the policy get enforced?
Well, this is mostly implementation defined,
but they do have some common names for things.
So first, you're typically gonna have the CPU core sending
out a request over the AHB interface.
And in a TrustZone-M world,
you actually extend the typical AHB matrix
to include the security attributes.
And that's called "Secure AHB" or "S-AHB."
Some implementations actually take this a step further
and include the ability to do policy-based restrictions
on passing through the bus matrix based
upon the security policy.
It's not common, but it exists.
When you reach the peripheral side of the matrix,
then you encounter the MPCs and PPCs,
which are really the policy going into peripherals.
Now, why do we have two different ones?
Well, the PPC is more of a general,
here's a single security policy for a peripheral.
This is the, I wanna have this entire GPIO interface
be available to the secure world or the non-secure world.
But in the case of something like an SRAM,
you might wanna do more fine grain policy
of having it be something like every 2K,
I wanna alternate through secure and non-secure.
And so that gets a separate MPC
that allows that fine granularity enforcing
of security attributes.
Okay, so what about other AHB initiators?
We have more than just the CPU core.
Well, again, left mostly to the implementers to find out,
but common terminology,
you might have a Secure AHB-aware initiator.
This is really what's happening with the SAU
and IDAU in the Cortex-M33s,
is you essentially have all the built-in logic
to make the decision about what
the security attribute should be,
and so you're just asserting that as part of the request.
If you have a legacy AHB device
and you wanna just insert it onto a Secure AHB,
you do need a way to actually set
what the security attribute should be.
So this is typically called an "MSW"
or "Master Security Wrapper."
And the common implementation
is to have it be a single security attribute gets applied
to all requests from that initiator.
So when you put this all together,
you get something like this,
which happens to be the AHB bus matrix diagram
from the LPC55S69 user manual.
And so you can see that it starts
to look like a network flow diagram, roughly.
And you can think of it in kind of firewalling rules,
where the IDAU and SAU and the MSWs are sort
of setting your ingress policy and labeling,
and then you might have some enforcement
within the matrix itself,
but then when you hit the egress side,
you can enforce more strict policies about,
because it came from this source
or it has this security attribute,
it is allowed or not allowed to go into this peripheral.
So the implication about all of this is
that most peripherals are gonna show up
in both the secure and non-secure ranges.
This is because from the IDAU's perspective,
you need to have non-secure ranges
in order for you to have non-secure access
to a space at all.
If I only had a secure alias for a given device,
I could only ever access it from the secure world.
And that could be useful.
The rest of the system is designed
to be implemented by the application software.
You configure the SAU, the MSW, the MPC,
and the PPC to set up whatever policies you want.
This is where you say, "I want GPIO block A
to be in the secure world."
Now, given this level of configurability,
you run into a lot of potential for mistakes,
and those mistakes can have devastating consequences.
For example, you could accidentally leave
the non-secure alias active for a code section
that is supposed to be secure only.
So with all of that in mind,
we'll go back to Laura to talk about
how we actually found the ROM patcher.
- Thanks, Rick.
So back to the ROM patcher.
Actually, really back to the secure boot flow diagrams.
It turns out that just having a box labeled "Validate image"
and "Validate CMPA and NMPA" isn't really descriptive
for all the ways things can go wrong.
I managed to brick a stack of boards
by screwing up various settings.
Physically, the chips are fine,
but there's no way to change any of the persistent settings
or get new code on.
As part of NXP's security for the boot ROM,
standard SWD debugging isn't enabled at power on
until right before the jump to the user image.
This means that if you screw up a setting,
there's no good way to figure out
where it's looping at the ROM
because you can't attach a debugger.
The lockout for these settings
also happens before ISP or the Debug Mailbox Handler,
which are designed to be able to access parts of the code,
so there's really no way to be able to undo this.
In some respects this is a positive sign
because it means that lockout will happen
if something does get compromised,
but it's a pretty big problem for transparency
in that we don't know all the details of what's going on.
All I really wanted was a complete flow chart showing
all the possible ways I could brick my board
so I knew not to do that.
NXP, unfortunately, did not provide that flow chart,
but what they did provide was a ROM
that was not read out protected,
so it was possible to save the ROM,
load it up in Ghidra, and start reverse engineering.
I had never really done serious reverse engineering before,
so this was a really fun project for me
to figure out what exactly the chip was doing.
I spent a lot of time comparing the addresses
against the manual to see what was being accessed.
NXP divides the flash space of the chip
into two distinct parts:
the regular user flash for applications
and the protected flash region.
The protective part of the protected flash region refers
to the fact that pages
in this region can only be changed under some circumstances,
or maybe not at all.
The protected flash region holds
a bunch of different settings.
The region can be divided up
into a couple of different areas.
The CFPA and the CMPA are designed to be programmed
by the user of the chip in the field
and at manufacturing time, respectively.
That's what the F and M stand for.
They contain settings for secure boot,
and screwing them up can result in a bricked board.
The NMPA is controlled by NXP
and can't be modified.
The details of what each field
and protected flash reading does was not fully described
in the manual.
NXP's region in particular was really not documented.
But apparently everything was more fully described
in an attached Excel spreadsheet.
And by attached, they mean attached to the PDF.
I was not aware this was something you could do
in terms of attaching things to PDFs.
But you'd download the attached Excel spreadsheet
and some parts are missing.
But we can show the rows,
and we finally get to see the full address space
of the protected flash area,
including the NXP area
which showed this really interesting region labeled
"ROM_PATCH, and doubly interesting
because nowhere in the documentation was the ability
to change the ROM mentioned at all.
Going back to why we were interested
in this chip in the first place,
part of how we build up our Root of Trust is assuming
that the ROM always executes the same code.
The ability to change the ROM is potentially concerning
and breaks a lot of our assumptions.
How concerning, though?
The only way to figure to know is
to figure out exactly how this block works.
Fortunately, the part for the disassembly
that references flash region give a full program sequence
for the ROM patcher.
Pretty sweet.
NXP's ROM patch controller can support up to 16 patches.
A single patch slot can patch one 32-bit ROM address.
The block cannot be used to patch addresses
outside the ROM region.
Up to eight slots can replace the specified address
with any 32-bit value.
Other slots can be replaced
with an SVC instruction corresponding to the slot number.
SVC is the system call instruction on ARMv8-M.
What happens here is the ROM sets up the system call table
in RAM to allow for patching longer instruction sequences.
The ROM patch settings do not persist across reset,
which does lessen the impact here,
but changes made to a secure address are
also aliased in the non-secure region,
as Rick talked about earlier.
This is a diagram showing the registered layout
of the hardware block.
To set patch,
the entire path fulfiller must be turned off.
The target address is set in one of the address registers.
If we're just doing the value replacement,
you write the value to one of the corresponding registers.
The value registers are reverse-indexed
which is a bit of an interesting quirk.
The control bit determines
if you're doing a value replacement or an SVC replacement.
Individual patches can also be turned off and on.
Importantly, these settings are not sticky,
so you can change settings as much as you like.
This means it's possible to patch one set of addresses,
turn off the controller,
then patch another set of addresses.
It's worth noting that this is probably a custom NXP design.
Previous generations of ARM had a standardized flash patch
and breakpoint unit to a lot of remapping
of arbitrary flash regions.
This was associated with at least one persisting exploit
that was presented at a previous DEFCON.
ARMv8-M removed the patching feature
of the unit probably because the designers realized
that remapping could potentially break isolation
with TrustZone.
In fairness to NXP, there are legitimate reasons
to want to have a hardware patcher.
Running software is hard,
and you do need a way to fix bugs in your ROM.
There's a reasonable motivation to want to bring this back,
but they probably should have had a bit
to be able to prevent modification
from curious firmware engineers.
The ROM is the first code that's executed at startup.
And if that were the only use of the ROM,
this block would be significantly less interesting.
The ROM also exports a bunch of functions
which are designed to be called at runtime
by user applications.
These are all features that are either hard to sequence
or NXP wants to keep proprietary
and would also be served by providing source code
to see what exactly these are all doing.
All of these APIs expect to be called from secure mode
as all the addresses are for secure aliases.
The Image Authentication APIs
also required privileged mode.
The addresses for calling the functions
are accessible via tables inside the ROM itself.
Okay, this is DEFCON here,
so clearly it's time to do something interesting
with this hardware.
We can put this all together to do an exploit.
For step one, we find ROM API used by secure code.
Step two, we used the ROM patcher from non-secure mode
to change the code of the API
since that code exists in ROM space.
Step three, have a secure code invoke that API,
and now you've won.
I'll pass it back to Rick
to talk about writing an exploit in TrustedFirmware-M.
- So now that we have a plan of attack
for how to use the ROM patcher
to create a privilege escalation,
we had to go actually build a full proof of concept.
And initially we wanted to start very simple,
but it turned out that it was much easier
to illustrate the severity of the problem
by taking a unmodified secure world
and demonstrating it with a non-secure app only.
In this case, we chose TrustedFirmware-M.
So what is TrustedFirmware-M?
This is a reference implementation
of ARM's Platform Security Architecture
which is really a set of APIs
for common security services offered
from a secure world to a non-secure app.
This is designed to allow you to have different vendors
for your secure world and non-secure world code.
Think of it as I buy a secure TEE type environment
from one vendor, maybe it comes with the chip,
and then I write my own code
and I only have access to the non-secure world.
So this provides a lot of core services like cryptography,
attestation, and Internal Trusted Storage,
and even some lifecycle management for your device,
which makes a lot of sense, and it's actually quite useful.
Thankfully, TrustedFirmware-M has upstream support
for the LPC55S69.
So I could just grab this and build it.
Now, the piece that we're really interested
in is the Internal Trusted Storage.
And why are we interested in that?
Well, as you might expect
from something named "Trusted Storage,"
it deals with persistent data.
And specifically, it's something that is required
to be implemented in the secure world
in most implementations of TrustedFirmware-M.
So this gives us an API
that's callable from the non-secure service space,
it's required to be implemented in the secure world,
and it's going to interact with persistent data storage,
which means it will touch the flash.
Does this have a good chance of using a ROM API?
Well, yes it does. It uses the ROM Flash API.
So then we can get into building our actual PoC.
Now remember, we're just building the non-secure section
of a full PoC.
We're using an unmodified TrustedFirmware-M.
So we take that TrustedFirmware-M,
I used version 1.2, which was the most recent at the time,
and I build that as an unmodified image.
And then I build a completely independent non-secure app
that does the following:
we need to write a little payload
that fits within the capabilities of the ROM patcher.
Remember that we only have a limited number of words
that we can actually write into or patch
into the ROM address space.
And so I needed a payload that would fit in that space.
Then, the ROM Patch Controller will be used
from non-secure mode via its non-secure alias
to copy that payload into empty space in the ROM.
So we're essentially moving code from the non-secure world
into the secure world.
Once we've done that, we can patch the Flash_Write ROM API
to invoke that payload.
Once we do so, non-secure world can call into the ITS API
to do any sort of write, and our payload should run.
We just have to verify it.
So we're gonna talk a lot about addressing and layout,
and so it's important to understand,
in these devices, you have a single flash region.
And TrustedFirmware-M,
in a configuration with no bootloader,
simply puts the secure and non-secure images right next
to each other in address space.
And then you have, you know,
your other persistent data storage.
Now, this is important because it goes into how the SAU
and the MPC get configured.
The secure world actually sets this up.
And when we flip on debug mode
for the TF-M secure mode or secure world,
it gives us the spew about that configuration.
And I verified that it doesn't change
when you turn off debug mode,
but it was just handy to be able
to have it spit it out to me over a serial port.
But if we look carefully,
you'll notice that the SAU was being configured
to mark out that section of non-secure code.
Now, if we were to go back and look
at our IDAU implementation for the LPC55,
we would find that the zero range up
to 1000000 is actually marked
as a non-secure range,
and so this is really using the SAU
to also set it as non-secure so that both sides agree.
This is a non-secure range.
The remainder of that flash region
will automatically become secure
because there's no SAU region covering it.
And then there will be a additional secure alias
for the entire flash as well.
Now, we also have the NSC base,
which, if you note, actually starts with a one
in the address, and that's using the secure alias.
Remember that NSC can actually be in a secure range,
so it's perfectly fine and legit to use secure region
for that, the secure alias for that.
But because of this being configured,
when we're executing in non-secure,
we should not be able to access the secure mode flash.
It's there, but there should be no non-secure window
into that region from zero to 40,000 hex.
Okay, so the first thing we do is look
at the Flash ROM API
and look at specifically to find a method
that we can instrument
and provide a call-out to our payload.
Now, it turned out that the Flash_Write method
is actually pretty simple
and provides a really nice entry point
where I can replace two move instructions
with a single branch and link instruction.
And I have a lot of registers
to play with it at that moment,
and it's really easy to replay those two instructions
inside my payload.
And so constructing the payload is primarily just doing
a bunch of copies of constants
from a non-secure section of memory
and copying it over into secure.
Now, this is really kind of a neat trick
because remember, secure mode can do data accesses
from non-secure regions.
So I don't actually have to put the full constant pool
in the secure world;
I just have to have the code that reads it be
in the secure world so that it can write
to the secure aliases of the other devices
that I wanna interact with.
So in this case, the particular example,
we're using constants to load in
and overwrite the control registers for both the the SAU
and the Secure AHB to basically extend the ranges
and open up both the secure mode code region
as we saw in the flash layout,
as well as the secure memory region,
and extend those in the SAU regions
that thus turn them into a non-secure space.
Now, if we were trying to access them
via the secure aliases, this would fail,
but a non-secure execution via the non-secure alias
with the SAU in this configuration,
it should be marked as non-secure,
even though it's accessing the secure code
that's at the zero range of the flash.
So how do we verify this? Well, it's pretty simple.
You just need to dereference zero.
Remember, everything's physically addressed,
so zero is where the secure code actually lives.
And we just do that from a non-secure space.
Assuming that everything worked,
we'll just get a dump out of the actual contents.
So here you have an NXP LPCXpresso55S69.
This is just an off-the-shelf dev board for that part.
I have two USB connections at the top.
The orange one is directly connected
to the USB interface on the LPC55.
The black one goes to
an onboard SWD debugger called the "LPC-Link."
And the main reason I have that is that I can,
it has a serial console or is connected
on main LPC55, and it exposes it as a USB ACM device.
So just a convenient way to get the serial.
We're actually gonna use the ROM bootloader
on the LPC55 for most of what we need to do.
So the first thing is we need a copy
of both the TrustedFirmware-M,
I'm using the tag for the 1.2 version,
and then we're also gonna get a copy of our PoC.
Now, I've said that we use an unmodified version
of TrustedFirmware-M.
There's actually a bug in their build system,
so I grabbed an upstream patch
that fixes that bug for this particular version.
Then the build system for TrustedFirmware-M uses pip,
so use a virtual env for that,
and then go ahead and have it fetch all of its requirements.
Then I'm gonna go ahead and install cmake that way.
And then, they use cmake
as part of their build system,
so we're going to go ahead and say,
"Build for this development board,
using the GNUARM too chain
with the TrustedFirmware-M bootloader disabled."
We're not using that in this case.
And then also build our PoC as the non-secure application.
Okay. So now we're gonna go ahead and build that.
Now, I did grab our PoC during the recording of this
from a private GitHub repository.
By the time this is available,
we will have made that same repository public,
and set the default branch to be the appropriate code
so you can play along at home.
Okay.
So with that built,
we can look and here's the non-secure world
and the secure world binaries.
Now, what we wanna do is erase the flash and load these in.
So the first thing we're gonna do is use NXP's tool
for this for interacting with the bootloader,
which is called "blhost."
And it's not the most reliable tool,
so what we have to do is hold down the ISP button
and press the reset button,
and it's supposed to jump into the ISP mode.
And then we can talk to it over USB.
So in this case, we got it on the first try there.
Now, I'm going to actually reset it again
and do the same thing between each step,
just because I get slightly better success that way.
Oops.
'Kay, so what we're gonna do
is write just the secure application,
the secure world, into the flash.
Great.
Now, if I reset and let this run,
you'll see that we get the debug spew that I showed earlier,
showing us the SAU configuration.
And then it actually launches the secure application,
and then hangs because there is no non-secure world
to execute right now.
So now we're gonna go ahead and put it back into ISP mode.
And we're gonna write in the non-secure world.
And then if I reset,
you'll see that it actually does that same dump
of the SAU configuration.
Now, this time, it actually starts to boot
our non-secure world.
There's a bug that I couldn't figure out where
when I first tried to use the ITS it crashes,
but if I just hit reset again,
it's gonna scroll by pretty quickly.
And I'll go back here to show what actually is happening.
So we started up and we got into this normal thing,
and the LPC ROM PoC started as the non-secure application.
And within that, we start a couple of threads.
One is this TOUCH FLASH API.
And this thread is literally just incrementing
a counter using the ITS.
It reads from ITS, increments by one,
and then pushes it back.
Now, we actually wait for five seconds
and before we apply our ROM patch.
So this is just the operation normally,
and then the patch begins.
And this is just a separate thread
from non-secure space executing the payload
that we looked at earlier.
It's loading the payload,
patching the Flash_Write API
and doing all that work.
And in the meantime,
the counter thread then, you know, keeps on running,
except now it's actually running
our PoC every time this counter is invoked.
And so then a little bit later
after we've let that just sleep for a while,
the PoC thread then is also running
from non-secure code again,
actually goes in and dumps out the beginning addresses
of the secure space.
So that that happens,
and then the counter just continues on,
and we, you know, print out a tick every once
in a while to show that everything is still alive.
So now that we've gone through our proof of concept,
let's talk a little bit about how we went
about disclosing this to NXP
and doing a public disclosure.
Like most people and most companies,
we started with the responsible disclosure process
with a 90-day response window.
And we started off with coming up with our own template
because, frankly, we're a computer manufacturer,
not a security company.
We didn't to have to do a disclosure
and didn't have anything ready for this.
So I put something together
that wrote up a lot of the details
and tried to provide impact and CVSS scores.
And then the first question was,
well, how do I send this to the vendor?
Where do I even find the contact info for their page?
So this is what you have at the header of NXP's website.
And I wasn't really clear to me where to go.
Maybe the Support section?
Nothing particularly obvious there.
How about the Company section?
Hmm, no.
No, nothing there either.
Okay, well, maybe that's at the footer.
That that could be. No.
No, not the footer.
Okay, well, back to the Company.
There was a contact us.
Maybe I can go there.
Oh, no. No.
But wait, there's a section that says Contact Support.
Oh, there it is hidden at the bottom
of the Contact Support page.
Now, when I did go to this page finally after finding it,
it does tell me how to email NXP's PSIRT
and gives me their GPG key
along with some details about their process.
Okay. Sent that off.
We did that on December 16th, 2020, and off it went.
And 30 minutes later, we got a response.
Now, this is not an automated response,
but it does actually mostly give the equivalent
of an automated response.
Like, yes, we got it,
but we'll get in touch at some point.
So I don't really count this
as part of the 90-day like timeline
for a response and coordination.
And in fact, I waited a very long time
for them to follow up.
If you noticed, this is December 16th, 2020.
Here on January 11th, 2021, I had to ask.
Like, it's been almost a month. Have you confirmed anything?
Is it look like a real issue?
And they came back a day later and said, "Oh yes.
In fact we did confirm it and it's an issue,
and thank you for reporting this.
Yada yada. We'll be following up."
And then the coordination kinda went a little wonky.
You'll notice from the dates and times,
like there was a lot of time between these interactions.
And it started with,
"Hey, we really looked at your PoC a little bit more closely
and, you know, we think we were doing some silly things."
Now, this was our first PoC
that did not use TrustedFirmware-M.
And so it's a little bit understandable,
but this is where we came back and said,
"Look, here's one with a very clean slate.
This does not use any modifications
to your example code for how
to write a secure world application,
and we can definitely do privilege escalation."
What was really annoying about this was an insistence
on trying to demote the severity of the issue in this case.
A long time after that, there was a,
"Hey. Yeah, this is a real problem.
And can we have 45 more days?"
Now, note this was almost 60 days
after we originally reported the issue
before they actually acknowledged that it was a real issue
of the severity that we told them it was.
So we had some internal deliberation and decided,
"Hey, you know what, tell us about what else is
in this chip because we are trying to use it.
We wanna be a customer.
And if you can tell us that,
we'll gladly give you 45 days."
Well, they just took the 45 days
and didn't really hold up their end of the bargain.
We did get some material
under NDA answering a few of our questions,
but not a whole lot.
One of the more interesting ones is we asked
to be able to access or audit their ROM API source code
because we already reverse engineered most of it.
We just wanted to see what else we were missing.
And this is the statement we got back.
And oddly enough, they gave us permission
to share this publicly.
This is not a statement that I would want
to see from a company making a security chip.
After we had had some further discussions
and were working out a timeline, I asked about,
"Hey, when are you gonna actually have a CVE?
Like, what are you gonna do about that?
We wanna have a public disclosure. This is a big issue."
And honest to God, the response was,
"What's a CVE?"
And we said, "Okay, you know what?
We'll just file ourselves.
I've done this before.
Not a big deal."
But then it comes out a month later,
they come back and say, "Oh.
Hey, we're gonna file a CVE on this.
Who should we put it as the reporter?"
And I'm like, "No, we already have the CVE number assigned.
Here's all the details."
Like we kind of took the wind out of their sails,
but it was very delayed.
So in the end,
we spent almost 4.5 months working
on this disclosure from start to finish.
And a lot of it was waiting for them to even acknowledge it.
In the end, from the Oxide side,
we did push out the CVE,
we did a blog post going into some of the technical details,
we did some tweets.
And actually, given how poorly the response went with NXP,
that was enough to fuel a spite CFP for DEFCON.
So here we are.
From NXP's side, their disclosure,
I call this public,
but in practice, they wrote a security bulletin
that we received under NDA
and we're told that, in fact,
they're only going to email that out to select customers.
I have searched and not been able
to find this anywhere publicly.
I also did not receive it outside
of our discussions with the PSIRT channel.
I have no idea how you get these security bulletins.
Somebody gets them, but good luck.
As far as off-the-shelf customers,
they did update the user manuals.
If you know what you're looking for,
the two most recent revisions
to the LPC55S69 user manual do talk
about the security regions,
but they are very light on what the underlying cause was
and are more prescriptive in terms
of you must write these registers to these values
without explaining why.
So Laura, what should we take away from all of this?
- Thanks, Rick.
Wrapping this up, this is the full scope of devices
with this ROM patcher.
The title of this talk mentions the LPC55S69,
but it affects multiple shifts in the LPC5500 family.
All those numbers in there actually mean something,
but I'm not a chip marketing expert.
Anything in the LPC55S6x is affected.
The S in the part name stands for "secure,"
which means it has TrustZone.
There is a variant, LPC552x,
which does not have TrustZone.
This means it isn't vulnerable to a TrustZone breakout
because it isn't there.
This vulnerability can still be used
for privileged escalation however.
Slightly worrying, the LPC552X does not have
the extra level-setting bits without TrustZone,
so the only prevention against it is the NPU.
One of the questions we asked NXP
when we were discussing was if there was an ability
to lock the patch configuration to prevent changes.
The answer there was no.
And it sounded like this was a deliberate product choice
with the idea that maybe they would need
to give a patch to customers later.
This seemed like a bad choice,
but I don't work for NXP product strategy.
I mean, maybe they could've put the lock bit in there
and let these risk guide,
but again, out of my hands.
NXP did mention that the upcoming LPC55S3x
would include the ability to lock the ROM patch,
which was really good to hear.
We're not sure if this is all effective devices.
NXP was willing to acknowledge the TrustZone vulnerability,
but wasn't willing to admit this was a privilege escalation.
There may be other chips
which have the same controller we don't know about,
which is really concerning from a security perspective.
Finally, some important takeaways.
Code is hard.
Configuring hardware is hard.
Configuring hardware security is really hard.
TrustZone-M is hard to configure correctly.
There's a lot of knobs.
Requiring code to drop permissions is a great way
to accidentally end up with something still exposed.
We've seen this in Linux plenty.
Reference implementations are frequently incomplete.
The trusted firmware PoC we gave could've been prevented
by setting a few lock bits to prevent changing the SAU,
but they just weren't set.
Just taking a reference implementation
is no guarantee you'll be secure
or you won't hit odd edge cases.
You need to be doing audits more view
of all the reference code.
Transparency is so important in your hardware.
In order to drop privileges,
you need to know that you need to do so.
The ROM patcher block was labeled as reserved
in various parts of the memory map
because NXP did not want to acknowledge it existed.
The old manual before their update would tell you not
to write anything for the reserved area
for various security protections.
This means that even if you were following the manual
to the letter, you would not be protected.
Oxide and NXP really seemed
to have a fundamental disagreement about
how big of an issue this is.
Our stance is that unless you have full knowledge
about what your hardware is doing,
it is not possible to write a truly secure implementation.
Turning that reference code
into something correct is incredibly difficult
without full documentation.
Hardware makers should really want
to make it easy to write secure code
and to treat undocumented hardware
with the antithesis of security.
And really, please just give us source code for the ROM.
The conclusion here should not be
that NXP should have re-protected the ROM
so that we couldn't find the bug.
That is not driven to be an effective solution.
Rick and I are not professional reverse engineers.
We're developers who think this kind of problem
is really neat.
I'd like to say there were no more bugs lurking in that ROM,
but I really don't know.
People who are much more experienced
in this area could probably find more bugs.
There's certainly been a history of bugs in the ISP mode,
some of which have been presented at DEFCON.
NXP could really increase the confidence here
by letting us all review their code.
Just saying that the third party reviewed the code
doesn't really tell us anything
if we don't know what was found or fixed,
especially if the audit didn't actually manage
to find this hardware block.
Hiding things doesn't actually make it more secure;
it only delays the time till bugs are found.
I really look forward to the future
when hardware manufacturers
have complete correct hardware documentation available
so we can all have more secure and open software.
Thank you.