- Hi, I'm Allan Cecil.
And with me today, I have-
- Dan Petro.
- We are presenting "You're Doing IoT RNG".
We're talking about vulnerabilities
that exist in virtually every device out there
in the IoT world.
That's a critical issue that we need to talk to you about
because random numbers are used everywhere.
- Before we get too far though,
let's talk a little bit about random numbers
and why they're so important to security.
Numbers of course are kind of how computers work.
So a random number can be a stand in
for all sorts of things
that we don't normally think of as numerical.
So encryption keys, authentication tokens,
and our lovely friend, business logic.
But one of the things you're going to notice
throughout this presentation
is that the vulnerabilities
that we're going to describe here
have a lot to do with these specific logic
of a particular application
in a way that is hard to replicate on map.
So a lot of the vulnerabilities that we'll discuss here
are kind of specific to particular applications
or frameworks and are necessarily the kind of canned exploit
that you might expect from a widespread CVE.
- One of the problems though
is that computers are notoriously bad
at making truly random numbers.
And that's because computers need to be deterministic.
If you did math and every time you got a different value
out of pi or you had a penny two processor
that just kept messing up floating point numbers,
you'd have problems.
And we have seen this in the past.
So we make computers to be very deterministic.
However, sometimes you need stuff that isn't deterministic.
You need some entropy.
And that's where our hardware RNG comes in.
That's its job, is to make entropy,
to make a source of randomness,
to be a seed for some way of getting random numbers.
So that solves the problem, right?
- Now is actually a good time to bring up
that there's two major kinds of random number generators
that you'll find, a pseudo random number generator
and a true random number generator.
You might think from the names of them
that you want the true new random number generator
or the TRNG, that that's the good one
and pseudo ones are the bad ones, but it's not that simple.
Really the distinction is only that one is made in software
and one is made in hardware.
And the names really lead you astray here.
I particularly hate the name, true random number generator
for the hardware ones.
It sort of implies a kind of quality behind them
that isn't necessarily present.
I suspect this naming comes from big RNG
as a propaganda term.
But in any case, the pseudo random number generators
also come in two major forms,
the cryptographically secure random number generators
or CSPRNG for short and regular ones.
So the regular ones are things you'll find
in like libc random or the Mersenne Twister
that are meant to be fast.
They're really efficient.
They're just pieces of software.
In both cases, they work basically the same though.
You take an initial seed number that could be any length
somewhere between 32 bits all the way up to 128 bits.
And then you stretch that entropy out
indefinitely into the future.
You could produce the stream of seemingly random numbers
from that seed, right?
So it's an entirely predictable set of numbers
given that initial seed.
And so the distinction
between a cryptographically secure random number generator
and a not secure one
is basically that the regular ones like the Mersenne Twister
or libc random
or any number of other common implementations
will leak information about the internal state
and, therefore, the seed as you go along.
So there's really no secure way to use those.
They're never safe to use for security,
critical pieces of information.
Anytime you actually need secure information,
like a crypto key or something like that,
you should definitely be using a CSPRNG.
And then the security of the hardware RNG
is something that we investigated in this research.
One unfortunate part about the design
of these hardware random number generators
is that there's not a lot of information known
about the details of how they actually work
or they're basically black boxes.
So if you wanted to find out
about how your favorite IoT device,
the hardware random number generator actually works,
you're kind of out of luck.
One notable exception is the STM32.
Wanna give them a shout out from STMicroelectronics.
They actually have really great information
about the details of the inner workings of their RNG
as well as proof of correctness,
a kind of proof of the good quality random numbers
that came out of it.
But there's two kind of basic digital designs
that you'll find for how to produce random numbers
in hardware, at least in a low cost way
that you'll find in IoT devices.
Analog circuits and clock timings.
The first one is an analog circuit.
So you're probably more familiar with digital circuits,
that is circuits that are gated by a clocking function.
Analog circuits are kind of the opposite.
There's no central gate by a clock.
So what you can do is set up a analog circuit
that sort of looks like the diagram we have here
on the top right.
And there's a bit that flows back and forth
between these (indistinct) functions.
So it's either a zero or a one
and a zero and a one kind of back and forth again,
and sort of spins in this infinite loop,
going back and forth between zero and one
at a function at a rate rather that is not exactly random,
but sort of arbitrary.
So if you were to pull that,
if you say, ask the analog circuit
at any given point in time to find out what the bid is,
the value will be pretty random.
And that's a pretty good way
of designing a hardware random number generator.
- There's also a method where you're using multiple clocks
at the same time to get a difference
between a delta, between the measured differences
between those clocks.
If you had two clocks that were derived
from the same source,
the result should, in theory, be identical.
So you would always get a one or a zero
depending on when you sampled it.
But if you allow them to run freely
and measure the delta between the two,
you can get a pretty normal distribution of differences
between the two.
Sometimes this happens on situations
that you wouldn't expect
where the designers didn't deliberately do that.
One of them happens to be the original Super Nintendo.
They had a 21 megahertz clock
for the central processing unit and a 24.576 clock
for the APU, the audio processing unit.
The result is that speed runners playing Super Metroid,
they have to deal with random timings
due to moving data across the bus
when going through door transitions.
Sometimes these random situations
happen on unrelated devices
and in a HAL harbor random number generator,
you're taking advantage
of what just sometimes happens accidentally
and using that deliberately to get random numbers out of it.
There are some issues that come up
if you're calling too often, if you're running too fast,
you're calling too frequently,
for instance that output call
from that analog circuit method.
If you're calling it too often,
you're gonna get the same number twice in a row
because you didn't give it enough time to transition.
And the same thing could happen with the clock method.
If you're calling it too frequently,
you could also end up with accidental sinking.
It could just be that both of your clocks happen to align.
So they're both operating in exactly the same offset.
There's no guarantee that you're going to be perfect,
but it's usually good enough that it's fine
as long as you're not calling it constantly.
How IoT does RNG is interesting.
Most new IoT system on a chips have a hardware RNG device
built into them as of 2021.
That hardware RNG is an entire peripheral
devoted to just RNG.
So it must be secure, right?
- The thing about IoT and programming on IoT devices
is that there's not really any operating systems
to kind of smooth over the errors that you might make.
Typically you just run C/C++ on bare metal.
So if you needed a random number
for the security critical piece of information,
like a crypto key,
you just call the hardware RNG peripheral directly,
usually through a HAL functional hardware
abstraction layer function
and something that looks basically like this.
This of course is pseudo code,
but it looks basically the same
across basically every SDK and operating system.
So there's the HAL_get_random_number of function
and there's kind of two parts
that we really care about here.
For one, there's an output variable, the outnumber,
that's the actual random number that we care about.
If you're familiar with C, it's an output variable,
basically you send it a pointer to the number
and then the function will overwrite the value
at that pointer.
And then the actual return code, which has an error message.
The error message that'll tell you
in case something went wrong along the way.
There's lots of things that can go wrong
when you are talking to a piece of hardware, right?
The peripheral might be broken or something went wrong
over the bus.
Maybe there are a random number generator peripheral
just wasn't ready yet.
Maybe the relative positions of Jupiter and Saturn
for all we know are not aligned.
In any case, an error can occur
in the calling of this random number generator function
and it'll let you know about that in the return code.
So we wanna ask the natural question,
how many people out there in the world
actually check this error code?
So as it happens,
almost nobody actually checks the return codes
of these HAL RNG functions.
Basically everybody out there just sort of makes a call
to the random number generator peripheral
and just uses whatever result that happens to give you.
You can see two results here from a free RTUs,
which is a popular IoT operating system.
And the MediaTek 7697.
You could see both of these calls here
are basically what we looked at earlier.
There's a return code that you're not seeing
that it's checked and then the output variable
that's put into it.
So maybe you're wondering,
all right, so you didn't check the return code
of the HAL RNG function.
What's the worst that could happen?
- Undefined behavior.
We don't know what will happen.
And that is a pen tester's favorite phrase.
The worst that can happen
might sound like it's the number zero.
And in fact, we have seen examples of this.
The XKCD joke that you always have to reference
when you're talking about random numbers
is that you ask for a random number
and it always returns a static value, a constant.
That isn't quite what usually happens, but it's not far off.
We've seen large swaths of zeros in a one gigabyte file
of supposedly random numbers.
And even in our own implementations,
when we were trying to do it properly,
we ended up with results
that had large quantities of zeros in them.
But that's not the most insidious one.
The worst is where you have partial entropy.
It looks like it's random,
but it isn't as random as you thought it was.
A good example is you make a call,
you get a 32-bit unsigned energy back.
It has four bytes of random numbers.
And the first call looks fine.
You call again, it still looks okay.
Then you call a third time and you get zeros
and you call a fourth time and you still get zeros.
But if you're not looking carefully enough,
you might not notice that you got a whole bunch of zeros
in your calls.
This partial entropy can be really tricky
because it substantially reduces the actual randomness
that you're working for, the strength of the random number
that you're getting.
And it might seem like this is not an issue,
but this seems to happen pretty often.
- So this is actually how this entire research project
got started.
We do a lot of IoT engagements at Bishop Fox
under what we'd usually call a product security review.
We like to say that if it breaks when you drop it,
it's a product security review.
And so one time we had a client
that was developing an IoT device
that does a lot of cryptography in it
that was kind of a security device.
And I was reviewing the code and looked at the code,
but he used a lot of a random number generation
in the process of doing a lot of this cryptography
and was curious about how it did random number generation
on such a tiny, low power device.
And it turns out
that there was a hardware random number generator
on the sock that our client was using.
And on a Lark, I sort of asked to see like,
"Hey, what is the quality of the random numbers
"coming out of this thing?"
I was sort of curious,
I hadn't actually seen the output of one of these hardware
random number generators previously.
And I didn't really expect it to be bad.
I was thinking
that the hardware random number generator
surely is the gold standard for RNG, right?
And so when we got the results back,
we ran it through some statistical randomness tests
and it failed basically all of them.
And then upon further inspection,
looking at the actual binary files
and a couple of gigabytes of output from the RNG,
the large swaths of it were just zero.
Surely I thought that this was a mistake,
that this can't have been right.
So we then embarked on this sort of longer journey
to investigate to see, was it just a single buggy chip
or was it some crazy buggy code?
And this entire thing kind of blew up from there.
So you might think that encryption keys of zero
are about as bad as it can get, right?
Surely it can't get worse than that.
Well, I'd like to introduce you to Petro's law.
If I could have one eponymous law,
one law named after myself, help me out here, DEFCON,
it would be this, "It can always be worse."
No matter how bad you think it is, it can always be worse.
So what could be worse than encryption keys of zero?
Uninitialized memory, that's what.
So take these three lines of pseudo code, for example.
These are pseudo code of course,
but if you just hop on to GitHub and look around,
you'll find lots of examples of this in the real world.
So you declare, but not instantiate a number
that you're going to be using as your holder
for a random number.
This is declared on the stack.
You then pass this to the HAL random number function.
However, if the HAL function works in such a way
that it doesn't perturb the random number variable
when there's an error condition,
like it doesn't set it to zero say,
then what will happen is when you go to use it later on,
the value will just be whatever happened to be present
in RAM prior to the call.
So this can actually happen quite regularly
in the real world too.
For instance, if you're doing some cryptography
with a Diffie-Hellman key exchange,
this will involve generating a random number
and then sending it over the network
to a potential adversary.
So this is something that's quite realistic at practice.
- So let's talk about some real world instances.
In 2019, a study against over 75 million
publicly available certificates
found that over 435,000 of them were vulnerable to attack.
In the study, they specifically called out
lightweight IoT devices
are particularly prone to being in low entropy states.
And I'm not saying that our research here
directly points to what they found,
but it's a pretty clear link
that when you have these many devices
that are producing such low entropy results,
it seems pretty likely that this was what they found.
- So you might've thought
this was gonna be a very simple case
of just simply blame the users, right?
Those pesky users aren't checking the return codes
and we just need to make sure
that they're calling the functions correctly.
Well, it's actually not quite so simple.
So take this for instance,
this is some pseudo code at the top here.
This is from the MediaTek 7697 documentation.
You call the random number, you check if the status code
is not equal to okay
and then you're sort of handle the error.
Well, the handling of the error comment right there
is doing a lot of heavy lifting
because when you need a random number
for some security critical code,
you can't just simply move forward
without that random number.
It's sort of important to the core thing
that you're trying to accomplish there.
So generally speaking, you're kind of given two options.
One is to spin loops.
So you can just kind of wild loop call
the random number generator function again
over and over and over again until you get an OK status.
Basically, you're going to use 100% CPU indefinitely,
maybe forever if the RNG to referral was broken,
waiting for a result.
That's not very good.
But the second option is just to quit out entirely,
kill the entire process
or if you're in the networking stack,
if you're trying to make a TLS key for TCP connection
that you're in, then it's going to involve
killing the entire TCP connection.
That's not a very good option either.
- Both are so unacceptable that it really leaves developers
with only option three, which is YOLO.
You can't just spend loop
because it'll lead to broken, buggy, useless devices,
same thing with quitting and killing the entire process
and starting over.
This is gonna make a device no one would want to buy.
So we force users into this unwinnable situation.
RNG and IoT is fundamentally broken
and it's not the fault of the user.
So let's talk about the right way to RNG.
And the right way to do this
is to use a cryptographically secure
pseudo random number generator.
It's a mouthful, but a CSPRNG has some distinct advantages.
It never blocks execution.
It has API calls that don't fail.
It pulls from multiple entropy sources,
more on that in a second,
it always returns crypto-quality results
due to stretching out the amount of randomness
that you have.
In short, it's a much stronger system
than just relying on a single source of entropy.
- The way that the CSPRNG subsystem works
is you start with a number of entropy sources.
So these can include the hardware random number generator,
but also lots of other things that an operating system
might have access to such as interrupt timing
from various devices, networking receive times,
like tiny nanosecond receive times
has a quite a bit of entropy to the network.
You XR them all together into this big entropy pool.
So it's important to know about this,
is all of them are XR together
so that it's not sufficient
to break just a single one of these entropy sources
in order to predict the output
of the random number generator,
an attacker would need to simultaneously predict
all of them.
So that's very strong.
Additionally, what you can do is then read
from the entropy pool
by a cryptographically secure random number generator.
These are typically just a hashing function,
like a Linux kernel, just MD5 the entropy pool.
And then in order to produce a more numbers afterward,
MD5 the last output with the entry pool itself,
so you can kind of chain basically key stretching
your way out to produce the functionally infinite
amount of entropy from a static amount.
Also because we can study these hashing functions offline,
we're very sure about the strength of the results.
So this also makes remediation tricky here,
that you see it's not as simple case
of use zigged view should have zagged
where there's a bug and a piece of software somewhere
and whoever made the software can just kind of fix that bug,
we patch it and move on with our lives.
This is a case of a missing feature
and one across the very heterogeneous landscape of devices
and pieces of software.
The most likely place that you'd implement a feature
like this, CSPRNG subsystem, is one of the emerging,
the IoT operating system, something like FreeRTOS,
Contiki-NG, (indistinct) or point of many others.
And we'd highly recommend using one of those
if you're making a new IoT device from scratch
because the individual device, SDK
is seeming like they're unlikely
to get this kind of a feature.
Those SDKs are usually very thin filled with example code
that's mostly around hardware enablement
and not so much around making a full device end to end.
So yeah, this is gonna be a tricky remediation process.
Now this vulnerability might be with us for some time.
- This is how RNG and IoT devices should work too.
But right now it doesn't.
And the rest of this talk is all about convincing you
why you really do need a CSPRNG subsystem.
It's absolutely critical to have this.
Now, let's talk a little bit more about using hardware RNG
to seed an insecure PRNG because this'll matter a lot
when we start moving into the rest of this talk.
Nobody codes from scratch.
We were talking earlier about blaming the users
for doing it wrong.
Well, if you're a user and you're grabbing reference
or example code and that reference library
or that code that you're working with has vulnerabilities,
it propagates those vulnerabilities out.
And one of the places this shows up
is some IoT devices and operating systems use the hardware,
but only to seed
an insecure libc pseudo-random number generator.
What that looks like is you're getting a nice,
potentially random number
from the hardware random number generator,
you're using that to seed libc,
but everything after that
is not necessarily cryptographically secure.
You're not really using the hardware
even though you might think you are.
This shows up in the MediaTek link 7697
and specifically in the Contiki-NG.
I'm gonna let Dan describe this
in a demo we have for you.
- So for this demo, we built an IoT security camera device.
It takes pictures every few minutes,
just like a real security camera
and post them to a publicly accessible website.
So the only thing keeping an attacker
from being able to view your photos
is that each file is named in this long random file name
here chosen by the camera.
Now before you go thinking
that this is unrealistically vulnerable,
this is how discord works
and lots of other applications like it.
Anytime you take a photo
and send it to a friend over discord,
it's publicly accessible.
The long random file name
is the only thing that's keeping people
from seeing your photos.
Our device, however, is built using Contiki-NG,
a popular IoT operating system.
When you call the operating system to get a random number,
it will use the hardware RNG on board,
but only to seed the insecure libc rand function.
So we don't know what the seed is,
but we don't have to because we can derive it.
So suppose one day you take a photo with your camera
and post it on social media.
Wow, what a cool camera you bought, how fun?
But what you didn't know
is that an attacker can use this file name
to derive what the original seed was that generated it.
That's because that's how the libc rand function works.
Our attacker here uses ANN Twister to find the seed.
Once they have, they can use that seed
to determine every past and future value from the RNG.
So our attacker can just plug in some of those numbers
back into the camera website and view every photo
that the camera has ever taken,
even if they've never been shared before.
All right, so let's have a word about exploitability.
This comes up a lot anytime you give a talk
about random number generation or crypto stuff,
is how exploitable is this really?
And the answer here is very exploitable,
but it's not going to be a canned exploit.
You're not going to see you're doing it wrong.pi,
that just sort of exploits things in the wild.
It's gonna have to do with the particular business logic
of the device that you're speaking to.
It's going to be very particular
to individual Internet of things, devices.
So there's not gonna be just like a simple CVE
that you can kind of apply universally across the board
to every device.
That comes with a one asterix, one possible exception here,
and that's with asymmetric keys.
You see one of the things
that often causes the hardware RNG functions to fail
is calling them very rapidly in succession.
So you call them too quickly and then it runs out of entropy
and just starts giving you zeros.
So one really common way to make that happen
in the real world is to make a 2048
or 4096-bit RSA Key, right?
In order to get that many bits from the hardware RNG,
you're going to need to kind of call it in a loop
in succession very quickly.
And those sorts of keys are very sensitive to low entropy.
It's not like an AES key where if you're missing 32-bits
off of 128-bit key, you're still probably fine,
whereas with RSA keys, that's not the case.
In fact, there's a different talk at this DEFCON
going on right now
called the Mechanics of Compromising Low Entropy RSA Keys.
We did not plan for this.
That was just a thing that sort of happened.
So this is actually a thing
that you can check for empirically.
You can look at the RSA keys coming out of IoT devices
and see if they are of poor quality or not.
And that's the sort of thing you can do
from the outside of black box as well.
So if you're a pen tester and you have an IoT engagement
coming up, how you're going to actually exploit this
is going to depend greatly
on whether it's a black box approach
or whether you have the source code
to the application of the device itself.
As a black box approach, it's gonna be much trickier.
You're gonna want to look at the output of the RNG
from however the application is using it.
The easiest way as we mentioned
is using the asymmetric keys,
the device produces an RSA key
or a certificate of some form,
then you can look at those cryptographically
to see if any known attacks work against it.
That's probably gonna be your best bet.
Second though, look for any opportunities to tax the RNG,
so any opportunity for the attacker
to influence how often the RNG is being called.
So for instance, if the device is producing some ID value,
and that is done at the request of a user,
then try requesting that very, very quickly
and see if the numbers start becoming zero
or if they're lower entropy.
Other than that, trying to actually measure
the entropy of values that come out of an application
can be very hard
because very often it's gonna be permuted in some way,
you're not going to get the raw hex values
that come out of the RNG.
Typically it's gonna be produced as a six-digit pin
or something like that,
and an urge to perform statistical analysis
on the kinds of output of the RNG,
you're going to need a very large sample size,
like a gigabyte or so.
And in order to get a gigabyte of six-digit pins,
you're gonna have to make a lot of calls.
So that might actually be very difficult in practice.
With source code, however, things become a lot more visible.
You can look into to see how the hardware RNG is called
and see if the return code is being ignored.
I hesitate to recommend
actually implementing a CSPRNG subsystem on your own
at this state, since there's a lot that can go wrong there.
There's a lot of moving pieces that go on there.
There's certainly a lot that you can mess up,
but at least consider it if it's critical enough.
- Okay, you've done everything right.
You're spin looping,
you're blocking until the hardware random number generator
gives you valid non-zero results
or validated that you've used every library correctly.
And that your libraries aren't written in a bad way
that are perhaps just seeding libc.
You validated all of the code you're using,
surely you've got it right, now right?
Nope, there's still some usage quirks.
You're still likely going to do it wrong.
In fact, you will do it wrong.
This is the same level of difficulty
as trying to write crypto code.
And for the most part,
we know not to try to write our own crypto code.
It's kind of a well-known law.
Well, this is my law.
My law is, don't write your own RNG code.
You will do it wrong.
Now, you might potentially have some documentation.
Maybe, if you can find it.
On some usage quirks you're gonna run into,
for instance, the LPC54628 has a warning on page 1,106
out of 1,152 that says, "When you're calling it,
"you have to throw out 32 results, use one,
"and then throw out the next 32 results
"and repeat the process."
It is the only way to guarantee
that you're getting proper random numbers out of it.
But how would you know
if you didn't sort through a 1,000-page document?
And if you saw this code written down
and you went through the comments,
unless they specifically called out why they were doing it,
you would think that this was buggy code.
This doesn't seem sane.
You can't try writing RNG code on your own.
It's as bad as trying to write crypto code
and even worse at ...
This is absolutely a situation
that emphasizes you can't blame the user for this.
There's no one that's gonna get this right.
One of the things we've touched on in this talk,
our hardware dev kits that SoC, silicon-on-a-chip vendor
is released to allow developers to debug
and flash their devices.
An IoT developer is going to build their advice
around an SoC.
And they'll be writing their code in C/C++
or something similar on a PC and then flashing the device
and testing it.
The dev kits provide a variety of different features
for testing things
and each vendor does it a completely different way.
For instance, this is a spark fund board
that is using an NRF based SoC
and it has a USB port for debugging and flashing
as well as an SD card.
Each vendor implements it completely differently.
And for instance, this is an older version of an STM32.
This particular design has a debugger at the top
that can flash the device.
The actual device you'd be flashing is this SoC down here.
This portion of the board is still a dev kit.
It has all of the functionality in that chip
exposed on the pens on the sides.
But you can do some interesting things.
You could flash this chip and debug it with this portion.
And when you're satisfied with the device,
you can actually snap off this entire upper portion.
Sometimes you'll do this because you'll develop your device
and then use the dev kit itself.
One of these designs that we did on the side
on my task bot project
was with this Cyprus PSoC 5 based board.
To flash this device, you plug this in into a computer
and to actually use it,
you're using a completely separate USB port
on the other side.
In our case, we connected the actual dev kit
to a board we manufactured.
This was made by total in the task bot community.
This allows us to connect to a video game console
and pretend to be a controller as in for instance.
You can also just make your own hardware
that doesn't have any of this
extra debugging functionality at all.
And that's what we did with this task team 32 bot
made by own source.
Same concept, this allows us to connect directly
to a video game console and pretend to be a controller.
We just have cables that go from ethernet
to say an Nintendo or Super Nintendo.
But in this case, we've taken the SoC right here
and incorporated it into a board of our own design.
And that's generally what a IoT device developer
is going to do.
Now, there are a variety of different advantages
of doing that.
One of the boards we worked with was this newer version
of an STM32 that incorporated everything
over just a single USB port.
Rather than splitting it out like the PSoC 5 did,
this has a USB serial interface,
it has debugging functionality,
there's various different ways that you can talk
just over USB.
This was really interesting to work with.
And ultimately the bulk of our research time
was working with devices like these
to try to access the hardware RNG that was on the SoC.
This particular device had really good documentation.
So a lot of credit to STMicro for that.
They even went so far
as to provide a proof of their randomness.
And when we started trying to reproduce the results,
we actually ran into several problems.
One of the issues we ran into
was that even with a reference librarian code
and documentation, we still ran into problems
with not properly spend locking
and blocking execution of the program.
For instance, in our early tests,
we were using something called byte circle,
just an analysis tool that allowed us to throw the data
at this program.
And it would show the byte distribution in a nice circle.
So out of 255 possible different combinations,
what we found is that all of the values we were getting
were very low numbers for some reason.
When you looked at the data in a file,
it looked like it was properly random.
But what we found
was that we were actually calling it too frequently.
We weren't blocking properly
and as a result, we were calling the hardware RNG
too frequently and starving it, getting bad results
out of it as a result.
One of the other challenges we ran into
was some of the devices made it very difficult
to even get the data off of.
MediaTek could be flashed over USB,
but to actually exfiltrate the data,
we had to use this crazy method over wireless
and do various Git calls.
It was pretty complicated,
and really the bulk of our research was just spending time
trying to get accurate numbers out of these devices.
I bring up all this complexity
because I wanna highlight dwangoAC's rule as it were,
don't try to make your own hardware RNG code.
It's as bad as writing crypto code.
I've said it before,
but I can't emphasize this point enough.
We spent a long time trying to do this
and still managed to mess it up several times
in the process.
IoT vendors really have it rough
because if they do release a device
that is standalone like this, how do you flash it
in the field?
Once you've released this, you've broken off the debugger,
there's possibly no debugger on it at all.
There's a lot of hardware out there
that possibly has badly implemented hardware RNG
that vendors simply can't fix.
So to pen testers, this is gonna be a perennial finding
that will pop up for years to come
because there's already hardware out there
that's potentially using insecure RNG methods
and it's not easy to fix.
- Okay, so let's take a look
at what the quality of the entropy looks like
coming out of these hardware random number generators
is raw, no more bugs,
no more silly usage quirks and library shenanigans.
Let's look at the actual numbers
coming from the hardware RNGs themselves.
Are they good, are they bad?
So we ran a bunch of the RNGs
through some statistical analysis
and we've got some cool results to show you.
What you're looking at here is a histogram
for the MediaTek 7697.
That is a diagram of all bytes
and how often they occur zero to 255 from left to right.
So what we should see is basically a flat graph.
Every byte should occur just as often as every other byte,
maybe with a little bit of fuzziness at the top.
But that's not what we see.
What we see is this obvious sawtooth sort of pattern
that occurs down the line.
And if there's two things that don't go along well together,
it's obvious repeating patterns and crypto keys.
Do you feel comfortable using this for your encryption key?
I for sure don't.
Next up is the Nordic nRF528040 for this,
there's this obvious repeating 12-byte pattern
of just zeros, there's a 0, 0, 0.
It's a little bit hard to see in the picture there.
The highlighting doesn't capture the third zero.
But this happens in hacks 50 bytes and that's super bad.
Obviously repeating patterns of any kind is bad,
but especially fully zeroed bytes
coming from the Nordic board
was something that we saw that kind of stood out
and made it fail
all of these statistical randomness tests there forward.
This example was so egregious and so peculiar
that we thought for sure for a really long time
that it was just our instrumentation that was at fault,
but we don't think it is for three reasons.
One, we spent a really long time investigating our code,
trying to figure out how this could possibly be happening
from it and never figured out how.
But two, it's a 12-bit pattern,
which is very weird if it was exactly one byte or no byte,
then perhaps you'd think
that was just a Knoll terminated string
was kind of getting copied around incorrectly somewhere.
But exactly 12-bits is really curious.
And also the amount that jumps by 50 bytes.
Sometimes it'll actually jump
by a little bit more than 50 bytes
which kind of offsets things a little bit.
So there'll be maybe 80 bytes
or something like that in hex.
And then it'll kind of continue on the pattern from there,
which also wouldn't really make sense
as an inconsistency in our instrumentation.
- So aside from just a distribution of values
and aside from just taking all of the bytes
and putting them on a graph
and seeing how frequently they are,
you can do a lot more statistical analysis.
There's a lot of tools out there including dieharder
that you can use on large datasets
to see if there are repeating patterns.
And we relied on this a lot
when we were doing our own implementation on the STM32.
When we first started, we were failing all of the tests
and it took us a while to get the code right.
We thought we were spent looping properly, we weren't.
It turns out it's very difficult to do this properly
even when you think you know what you're doing.
Once we got through all of the other tests,
we still had one of the dieharder tests
that continued to fail,
which was the RGB minimum distance test.
- So the minimum distance test,
what it's doing is it takes a bunch of random numbers,
interprets them as integers,
and plots them in end dimensional space,
and then calculates using a simple algorithm
what the minimum distance
between any two of those points are.
And it should fall within expected parameters.
What this is doing really is it's checking for repeats
or nearly repeated values
since any repeats would cause the minimum distance
to become smaller than it normally should be.
So what you're seeing here in terms of a failure
is a P value of exactly zero,
which means that it's very confident
that this should not have occurred randomly
or the chances of it having occurred naturally
is very small.
At the same time, you're watching this,
we'll be releasing our code that gathered
and analyzed the hardware RNG entropy itself.
It's nothing terribly special,
but it took a long time to get right
and working across a bunch of IoT dev ports.
And while we don't think that there's any errors
that would have impacted the results,
consider that even if it turns out that there's a bug
in our code, what does it say
that two computer security experts spent months
studying and analyzing our RNGs
and still couldn't get them to work properly?
And what does that say
about the state of IoT security more broadly?
- Okay, so what are the conclusions we've come to
after talking through all of this?
The first is that this affects the entire IoT industry.
It's not a single vendor, it's not a single device.
It's not a single particular quirk, this is widespread.
And if you take nothing else out of this talk,
the point we really wanna make
is that the IoT world needs a CSPRNG subsystem.
This can't be fixed by just changing documentation
and blaming users.
You really need RNG code that is well-vetted
and you should consider it dangerous
to write it on your own, it's just like crypto code.
And also you should never use entropy
directly from the hardware.
You don't know how strong or weak it might be
as we just showed you.
- Okay, so what can you actually do about this?
Well, it depends on what camp you're in.
If you're a device owner, keep an eye out for updates,
IoT devices aren't the best in general
with software updates.
And you're gonna wanna make sure that yours is ready
to take them when they're available.
IoT device developers, we highly recommend using
one of the emerging IoT operating systems.
We don't have a strong preference for one versus the other
at this point.
But if you're baking a new device from scratch
or maybe even just updating a current one,
we highly recommend using those rather than writing raw C
on the bare metal yourself.
If you're an IoT device manufacturer
or an iOS developer themselves,
implement one of these CSPRNG subsystems,
that's the only secure way to do it,
there isn't a way to get around it.
You might even consider deprecating
or straight up disallowing users
from using the hardware RNG raw itself.
And if you're a pen tester,
keep an eye out for this
because it's gonna be a perennial finding for years to come.
- I am Allan Cecil(dwangoAC.)
- I'm Dan Petro and thanks a lot.