- Hello, DEF CON.
Today, we present to you our research
on the DNS vulnerability class
in DNS-as-a-service providers.
My name is Shir, and I lead the research team at Wiz,
the cloud security company.
With me in the room is Ami Luttwak-
- Hi. - The CTO of Wiz.
- Thank you, Shir, and it's really great to be here.
And so, let's start with a bit of background about us.
So we are the Wiz research team.
Wiz is a cloud security company.
The team is composed of experienced security researchers,
many of them with background
from Microsoft Cloud Security group,
and our goal is to do groundbreaking cloud research
to find vulnerabilities, misconfigurations,
risks in cloud environments that customers are not aware of,
and would impact anyone using the cloud.
So we started looking into DNS-as-a-service.
Why DNS-as-a-service?
So first of all, DNS, as we all know,
is the lifeblood of the internet.
It's probably one of the most important
services that we have.
Now, DNS-as-a-service has huge impact.
If you think about it, it holds your domain,
it holds your internet routing.
Now, what's cool about DNS-as-a-service,
unlike usual cloud services, is that also
it controls not only cloud activities, right,
when you use DNS-as-a-service, it also impacts
your on-premise activities.
So potentially, DNS-as-a-service has
huge impact on an organization.
Now, on top of that, what is really great
for researchers is the DNS protocol is
one of the oldest protocols that we use.
It's more than 20 years old.
It's incredibly complex,
and there's so many different implementations of it,
both from the DNS providers,
but also think about millions of DNS clients
that we have out there,
each of them implemented in different way,
creating a very, very interesting attack vector
for researchers to explore.
So we started from looking into Route 53,
which is the DNS service provided by AWS,
and it's highly popular across AWS customers.
- So Route 53 is built on thousands of DNS servers
that host DNS zones for all AWS customers.
We mapped around 2,000 servers in the Route 53 platform.
Whenever the customer is hosting a domain name
on the platform, they get four shared name servers
to manage the desired domain.
On that right side of the slide,
you can see a simulation of one out of the four DNS servers
Amazon provides each domain.
The server stores the DNS zones for wiz.io, for example,
and several other AWS customers.
If the server is queried for one of the domains
under management, it will return the appropriate records
for that domain.
While studying Route 53, we discovered that
anyone can register any domain they want on the platform.
There is no restriction on whether the domain
is already hosted on the platform,
and there is no ownership verification.
The only limitation is that
if the domain already exists in one of the name servers,
it will not be possible to register it again
on the same server.
And it makes a lot of sense,
and it's indeed not possible.
Trust us, we tried.
Basically, anyone can register any domain
on any of the name servers,
as long as it does not already exist there.
But is there anything dangerous here?
For me, as a security researcher, it felt like
we had too much control here.
But almost any DNS-as-a-service provider works this way.
So is it really a security problem?
All right, this is an example that
if you try to register Wiz again on the server,
it would be impossible, because it's already there.
- So we started with a very, very simple
and interesting research question.
If we can register any domain, right,
what domain can we possibly register
that will give us interesting access to data, right?
So we wanted to register a domain
that is not already present or the name server,
and that for some reason DNS clients will
actually query for that.
So we started into a quest to think what domains
can we register that no one thought about,
and we could actually somehow break the DNS?
- So we thought about it, and came up with an idea.
What would happen if we register
one of the official AWS name servers on the platform?
What would happen if we registered
one of the Route 53 name servers under the same server?
So we chose an arbitrary name server.
In this case, it's the ns-1611, as you see on the slide,
and we tried to register it on the platform enough times
that at least once it would fall under the management
of the same name server.
Let's do an illustration of that.
So as you can see in the slide,
we have an illustration of the ns-1611 name server,
which is an official Route 53 name server,
and you can see it already contains DNS zones
for several of a lot of these three customers.
What do you think would happen if you managed to register
the name server domain name ns-16-11
under its own management?
I totally don't know what would happen,
but we must check it out. - Definitely.
(Ami laughs)
- So we tried it, and it worked
As you can see on the slide,
our new domain is now managed within a name server
with the same domain name.
We were really excited.
We didn't know if it will have any impact,
but we had a really good feeling about this one.
So to test the effect of what we did,
we decided to specify an IP record,
IP record pointing to our server.
So now, if a DNS client will connect
to the ns-1611 name server and ask it about itself
it is our IP address that will be returned.
At this point, we were really curious to know
if anyone is asking Amazon's server about itself,
and if anyone is trying to connect with us,
because of the manipulation we did.
So we ran tcpdump on our server,
and hoped to see something interesting.
Surprisingly, we started receiving thousands of requests
from thousands of different IPs.
We realized we were onto something.
The next step in the research was
to analyze all these DNS queries,
and figure out why these queries are being sent to us.
- So wait, why are we getting any traffic?
And no one was supposed to ask the name server
for their own domain.
The name server actual domain is registered
on other name servers, right?
So why are we getting any traffic?
And what's more weird was that
it wasn't actual regular traffic.
It wasn't even DNS traffic.
It was dynamic DNS traffic,
which is a very specific protocol that
you wouldn't even expect to see
in this type of internet traffic.
Now, we were getting a lot of data.
I'm talking about IP addresses,
computer names, domain names.
So we started investigating.
We started to look into the data.
So basically, we saw, we were getting millions
of extra requests from endpoints,
and when we started looking into it,
we saw it's a lot of data.
We were seeing internal IPs, external IPs,
names of computers.
We very quickly understood
these are names of endpoints
within many, many different organizations across the globe.
And the scale was totally unbelievable.
I'd say within a few hours of sniffing the traffic,
we saw more than a million unique endpoints,
and they would belong to,
based on the initial analysis we did,
we saw more than 50,000 organization,
15,000 organizations, all of them using AWS
as a DNS service.
Now, we said, "Okay, let's try to figure out
"what organizations are we seeing here," right?
So it was quickly we understood that
we are stepping into an unbelievable tap
of worldwide organization and traffic.
We saw Fortune 500 companies.
We saw more than 130 government agencies
right in the traffic.
So we knew we were probably onto something big,
but the problem is that we had no clue
what we were seeing and why.
So what do we know so far?
We registered a name server domain
on the name server, right?
We have no idea why, but for some reason,
millions of endpoints started sending
tons of dynamic queries to us.
But again, why?
Why are we seeing dynamic DNS?
Before we are able to answer
that simple, mysterious question,
we have no way to actually understand what we are seeing.
So we decided we are going
to step into the world of dynamic DNS.
- So what is exactly dynamic DNS?
Dynamic DNS is an extension to the DNS protocol,
which is specified in RFC 2136.
It allows clients to dynamically update DNS records
of a target DNS server,
and it's commonly used to help devices find each other
in internal Windows networks.
Let's see how it works.
So when my Windows computer joined the company's network,
and receive a new IP address,
as you see on the slide, it updates the local DNS server,
which is called master server on its new IP address.
Now, when Ami is trying to connect to my computer,
he can query the local DNS server about SHIR-PC,
and the master server will answer it
with my current IP address.
So far, sounds like a great feature.
At the moment, it is still not clear to us
why endpoints sends dynamic DNS updates to our server.
These are requests that should never leave
the internal network.
Could it be that the endpoints
think of our server as their master server?
How does it even know to find the master server?
So it turns out Microsoft has its own algorithm
for finding the master server,
and it does not work exactly as specified in the RFC.
Just before we go into the logic of the algorithm,
let's do a brief refresh on DNS records
for those who have not touched DNS recently.
So in order to fully understand the algorithm,
it is important to remember only three types of records,
and we'll make it very quick.
The first will be the A record.
A records specify the IP address
of which the domain points to.
The second would be the NS record,
which specify the name server of the domain name.
The third is the SOA record,
which is short for Start of Authority.
The SOA records contain
administrative information about the domain,
and its first parameter is the master server.
This is the server in which clients would attempt
to update doing dynamic DNS updates.
Usually, this server will be
on one of the domain's name servers,
and in Route 53, the default value of this field
will be the first Amazon's name server
from the name servers list.
And now that we have all refreshed our memory,
let's get into the algorithm.
Now, this is the primary name server.
So when a Windows endpoint want to update
the internal master server for its new IP address,
it first needs to find it.
The endpoint will send the SOA query
for the internal DNS resolver
on its own fully qualified domain name.
The internal DNS resolver knows that
the wiz.io DNS zone is managed internally,
so when queried it will return
the internal master server within the SOA response.
Now, when the endpoints find the master server,
it will update it as we saw in the previous illustration,
and the update succeed.
But what happens if the corporate DNS resolver
is not set correctly,
and does not contain a DNS zone for the local domain?
Or what would happen when the computer
leaves the organization,
and starts working with external DNS resolvers
provided by home routers?
In that cases, when the computer query
for the corporate domain, the DNS resolver will treat it
as an external domain and will return the master server
of the external domain, just as it will do
with any other domain.
This is where the problem starts.
So imagine an employee of Wiz decided to work from home,
like most of us lately, and connected to their home Wi-Fi.
The computer got an internal IP address
from the home router,
and now trying to find the local master server to update it.
Because the external DNS resolver are not familiar
with wiz.io, it is going to resolve it
just like any other domain.
Because the domain is managed on Route 53,
it will return the wiz.io master server
as specified in Route 53.
The endpoint will then try to update the master server,
which is an Amazon shared name server
that manages thousands of customers.
Obviously, Amazon servers does not support dynamic DNS,
so the update request would fail.
The thing is that Microsoft's algorithm
does not give up here.
The algorithm believes it still has a chance
to find the master server.
So the next step would be to check
if the wiz.io's name servers have records
for the master server.
So the DNS client connects directly
to the IP address of the name server,
and ask them what is the IP address of the master server.
In our case, it is the same domain name as the name server.
And here happens the interesting part.
In fact, Windows DNS clients queries
the name server for itself.
And if you remember, we have registered this DNS zone,
and here we can return any record we want.
So we returned our IP address to the DNS client,
and now, the computer will update our server
with DNS updates, which is the malicious actor server,
and this is very crazy.
- So now we reached the point
that we started understanding what's happening, right?
Because what we know, we know that Windows endpoints,
they use a custom algorithm to find the master DNS.
The algorithm queries the name server for its own address.
When you are in external and using Route 53,
what happens is that this means that
you're actually querying the name server for its own domain,
and that explains what happened.
That's why we are able to register our malicious DNS server,
and we were receiving this dynamic DNS traffic
from millions of endpoints,
because all of the organizations using Route 53,
when these devices are actually outside of the domain
and outside of the company,
they will actually use this algorithm,
and we would be able to get traffic from those devices.
So we understood what's going on.
The next step was to figure out
how much data are we actually getting,
and what can we find from it?
So when we started looking into the data,
we quickly understood that we are actually building here
what we called a nation state intelligence capability,
because think about it.
We saw IPs for millions of endpoints
from more than 15,000 organizations,
but it's not just DNS requests.
We are seeing external IPs, internal IPs, computer names.
We are starting to map all of those organizations.
So let's see what we can do with this scale
of intelligence capabilities.
So we started from what we call IP-based intelligence.
Imagine that you can map companies
and a portion of companies around the world
and map their global sites,
map where their employees are at, right?
So we looked at one company, for example,
and view, and just see how amazing this is.
So this is one of the largest services companies
in the world, and we got around 40K endpoints
actively reporting from this company.
Now, we are seeing a mapping of all of their global sites,
all of their actual offices,
the employee locations, right?
So from just the external IPs, we can map,
create a map of offices and home locations of employees
across all of those companies.
It's pretty amazing.
And we can go even deeper, right?
This is an exhibit of a specific office,
where we detected 600 endpoints of that company, right?
So imagine an intelligence capability
that maps for you in one single tap into the DNS
without any trace actual structures
of organizations across the world
and all of their different offices and locations.
So, but it doesn't stop there.
Then we started thinking,
"Okay, what interesting data can we pull from this
"if we're an intelligence agency now?"
So we started looking at companies
that are in violation
of the Office of Foreign Assets Control.
And we saw interesting things.
For example, this mining company,
it's an international mining company,
and interestingly, we found six employees
working from the Ivory Coast,
which is definitely a place that is not allowed
in that regulation at all, right?
And we saw so many interesting examples like that.
So we found a subsidiary of a large credit union
with a branch in Iran.
Again, 13 endpoints working from Iran
based on our newly revised intelligence capability, right?
So it's not only mapping all organizations.
We can now start finding violations.
We can ask questions across so many different agencies.
You remember, we are actually seeing
also government agencies.
I wonder where are all of their offices, right?
Now, it doesn't stop there, because if you remember,
external IPs is just a small portion of the data, right?
We also have internal IPs.
So what can you do with the internal IPs?
If I have internal IPs from different endpoints
from the company, I can start building
the map of the network,
the internal logical network of the company.
So for example, these are the segments.
This is the employee segment.
This is the ICT.
Here's the operational network.
Remember that we also see names of devices,
so we can start really understanding all of the segments,
so building an intelligence map
of the organization externally and internally
across thousands of organizations.
Now, we also had computer names
and the computer names actually hold information
about the endpoint, right?
And many times, you get employee names.
You understand the actual role of the machine.
You can see the specific, the build
the machines is using, right?
So we are actually getting quite a lot of information
about those companies based on the IPs,
the computer names.
Here, we see this is finance,
and we see all of those machines
are part of a specific segment.
Okay, perfect, I am starting to build
my internal map of this company.
That's perfect.
Now, just so we understand the scope here,
we looked at a specific DNS provider.
Then we asked, "Wait, is this only this DNS provider?"
So we started looking at others,
and we soon found there is many others
also susceptible to the same vulnerabilities, right?
This is not just a Route 53 vulnerability.
This seems to be something that is shared
across most of the DNS service providers,
and if you think about it,
we don't have to stop at the cloud providers.
You have shared hosting.
You have domain registrars.
There is so many different service providers using,
again, I think the shared a concept here
is that they provide DNS services
for many, many different companies,
and there is a chance that many, many of them are vulnerable
to this attack of name server hijacking.
So we started from AWS.
We reported the vulnerability,
and it was fixed really, really quickly
by the AWS Route 53 team.
Again, I think that within a week, or two it was fixed,
and no one can now utilize this vulnerability in Route 53,
because you are not allowed anymore to register
those special domains in the name server,
and we are in disclosure process
with several additional cloud providers,
and we believe there is many, many more to come,
and it's part also of what we call the industry
to start looking into,
and actually check across all of the DNS providers.
- So as Ami just said,
AWS fixed the vulnerability very easily.
They added all the names of their name servers
to an ignore list, which is simple, and very effective.
Users trying to register one of the official
AWS name servers on the Route 53 platform
are now receiving the following error message,
which says that the domain is invalid,
and you can see it on the slide.
When we reported our discovery to Microsoft,
they explained to us that this is a known misconfiguration
that occur when the organization works
with the external DNS resolvers,
and it's not considered as a vulnerability.
So we would like to offer a solution
for both platforms and organizations
who would like to protect themselves
from these kind of attacks.
So first, we will start with the platforms.
DNS providers who want to ensure they are not vulnerable
should make sure it is impossible to register
their own domains on the platform.
DNS providers who want to have even a better security
can also do ownership verification
to ensure users only register their own domains.
And in addition, it is very important to make sure
that the platform user cannot register a reserved name,
as specified in the DNS RFC.
The RFC is full of reserved domains
that should not be allowed to register,
and their registration may lead to unexpected behavior.
For organizations that want to make sure
they're not vulnerable, we recommend making sure
that the primary name server in their SOA record
does not point to a different domain
owned by the DNS provider.
As you can see in the slide,
and now you can see it,
the default primary name server
that a domain owner receives when they add their domain
to the Route 53 is the first
of the four name servers, which manage the DNS zone.
Changing the primary name server
to an invalid sub-domain of the organization,
or the even the real primary master server
will fix the issue,
and attackers, and will not allow potential attackers
to register domain on the platform,
as you can see in the slide.
And yeah, we are very close to the end.
- So just a few summaries and take-aways.
First of all, what's really cool here is that
we were able to get
to nation state intelligence capabilities
from a simple domain registration.
Just a simple for the registration got us so much power,
and we believe what we saw here is a new class
of DNS vulnerabilities.
This is just one idea of a domain.
Think about how many different
interesting domains you can try to register,
and remember today, you can basically, right,
register any domain that you want
that would trigger unexpected results.
No one thought, and we honestly didn't understand initially
what would be the impact of registering
the name server on itself.
I'm sure there's many other magic domains
that you can register,
and it opens up so many interesting questions,
like all of this traffic was dynamic DNS.
Why is it actually, dynamic DNS was built as a protocol
like 20 years ago for on-premise networks.
Why are we still seeing it outside in the internet?
What are the implications of this protocol
still being active on the internet,
and potentially endangering both the endpoints
and the DNS servers, right?
So there's so much here that we see
as potential research areas for us
and also for the community,
and we believe the scope here is huge,
'cause it's not a single service.
We found it across multiple DNS providers,
and we are pretty sure that this vulnerability
and many others will probably impact
many, many, many of those DNS providers.
- All right, thank you very much, guys.
- Thank you. - It was very fun.
- Yeah?