hello everyone welcome to my
presentation named finding hidden games
via url shortener services
in this presentation i will talk about
the security risks of url shortener
services
and how can we take an advantage of it
let me introduce myself a bit i'm odd
cushion i'm working in invictus security
as
application security manager i'm writing
various of security tools
you can check them on my github account
also i'm writing
blogs about security and you can visit
my website to
read them
let's start with a highly debated topic
secured by obscurity
usually people are arguing against
secured by obscurity
since it doesn't provide an actual
security
it relies on hiding an information from
an attacker
if the attacker somehow finds the
information it's a game over
however in some cases secured by
obscurity can be useful
and sometimes you have no other option
to trust the
obscurity let's check some examples
let's say you have an admin panel on
your website and you don't want to
attackers to find it
the best practice is allowing users
the best practices allowing users for
that endpoint by checking their ip
addresses right
but in some cases this might not be
possible due to different reasons such
as cost
lack of engineers to do it etc in this
type of cases
best option would be hiding the url with
random strings
so that an attacker won't be able to
guess it the same goes for the shareable
urls such as google docs as well the
shared links should be accessible by
everyone
but the attackers shouldn't find it
easily
therefore they are generating the urls
with random strings
but of course it's not impossible to
find those hidden urls
there are different methods to find them
when you visit or
share a url it's locked into different
places
for example if you didn't make a
restriction with robots.txt file
google search engine may index your
pages therefore
an attacker will be able to find it with
a basic search
also archive orgs wayback machine saves
everything on its database
you can cure the wayback machine to find
sensitive urls
another source is alienvault's
openthread exchange service
alienvault also caches lots of different
urls for uh threat intelligence purposes
however you can take an advantage of it
by searching for
sensitive urls there is a great tool
named get all urls which allows you to
search on the
mentioned sources it's widely used by
bug bounty hunters and it works really
well
the other method for finding sensitive
url is brute forcing them
however it won't be very efficient if
the website uses
random and long strings
let's check the following examples if
the website uses
common names for the url pets it will be
possible to reinforce it
if you have a good crafted word list you
can send lots of requests
with tools such as go buster
the second example harder to guess since
uh since
it also uses numbers in the url path
uh you won't be able to guess it with
common word lists uh you have to tweak
it by yourself
however it's still not impossible the
last example is
google docs with which creates urls with
random and long strings when you
calculate the brute force pool size
you will see that it's impossible to
guess it with an online attack
now the fun part starts people are
widely using url shortener services to
share urls
so why they are using it because it
it's because url shortener services are
converting long and ugly urls
into short and beautiful ones also some
websites such as twitter and linkedin
makes the
url shorter by default therefore some
urls are already
shorted even if you wasn't intend to do
it
so how does how does those services work
uh when you provide a url to shorten a
service
it generates a short and unique string
for it
so when you visit that generated url it
basically redirects you to the original
one
so what is the risk here why is a big
deal
in the previous slide we said that it's
impossible to brute force a google docs
url since it's a long
and random string uh it will take
months to reinforce it but now we have a
very short url
it's seven characters longs so in order
to find the long and complex google docs
url
you just need to brute force a seven
characters long string
let's compare their brute force search
spaces
the google docs citrix contains upper
lower alphabetic characters
and numbers its length is 44
and the search space is encrypted
incredibly long and impossible to brute
force it
on the other hand bitly string contains
upper lower alphabetic characters as
well
and numbers however
its length is 7. therefore the search
space
is equals to 3 trillion but of course
you can say that it's still too much for
an
online tank and you have a point here
however our main goal shouldn't be
finding that
exact exact string right if we brute
force random seven character strings
or we can follow a pattern uh we will be
able to find lots of bullet urls
on our way for example you sent the
first request to bitly and
received 404 error after done you
increase the last character by 1
and send it again you got 4 or 4 error
again
and in the next request we got 300
redirection which means that the url
exists
we can take a note of the redirected url
and continue to search
if we make enough guesses we will be
able to guess lots of valid urls
but of course to gather an optimal
amount of urls
we need a high uh firepower we might
need
lots of servers to make continuous
guesses every day
but isn't it so expensive uh how can we
achieve that
thanks scott we are lucky about this and
we don't need to spend tons of money on
the servers
because there is a volunteered group
named url team out there
url team has lots of tools
uh allows you to brute force various of
url shortener services
lots of different people are using this
tool to brute force
those services every day and they are
saving the found urls on their platform
for example they brute force 18 billion
google
service urls and they were able to find
three billion working urls
which means that if you used google
service to make your sensitive url
shorter
most likely your sensitive url is now
saved
to their database and the attacker can
find it
if they are looking for it
url team also provides a guidance for
each shortener service
for example what kind of url pet they
are using what http methods
they are using what is the character
sets they are using
also how much delay you need to put
between every request
how can you understand if there is a url
or not
also those services will be able to ban
you
if you send lots of requests in short
amount of time
therefore we are using delays here
also they are providing status quo so
that you can understand if you are bent
or not
those guides are really helpful it
allows you to brute force them
practically so that you won't waste your
time and resources
i quoted a tool named url hunter to
parse url teams raw data
and allows users to make smart searches
on them
the tool is written in go and you can
find it on my github account
so how can you benefit from this tool
you have three
you have different search options the
first one is single keyword
in here url hunter will search the given
keywords on the database
and will show you the matched ones for
example
when you search for example.com it will
both match with
example.com blabla and another.com
referrer equals to example.com
the other search method is using
multiple keywords
in here url hunter searches given
keyword with and logic
therefore both keywords must be present
in the url
for example if you search for both
example.com and admin keywords
it will match with example.com secret
slash admin panel
but it won't match with example.com
something else
the other search method is using regexis
it's good to catch specific patterns
such as credit card data
map coordinates or something else it's
totally up to you
now let's watch url hunter's demo video
you won't be
sore uh so you're a hunter
what's your own hunter doing there is
downloading to
archives from the url hunters i mean
url teams repository
and of course those uh archive files are
big since it they are contained millions
or billions urls in it
and after done it's basically unzipping
them
and now it searches for the given
keywords in it
for example in this example we are
searching for
docs.google.com and
with it should contain spreadsheets
in it
also it searches for trello dot com
keyword
so now we are checking the output
and as you can see uh url hunter matches
with the given keywords on the
located databases
ah sorry about that the screen recording
software is gone matt
okay also there is another project name
uh there is on another project is came
out after i released the url hunter
tools
uh many of you probably already know the
greyhead warfare's s3 bucket search tool
which allows you to search on publicly
xboxed exposed packets
it's a great project and i'm a huge fan
of it
and now they support xposed urls via
shortener services as well
also it has an advantage over eurohunter
url hunter downloads the archive files
in order to make
a search on them therefore it takes time
however greyhead warfare downloads the
archives on their own server
and allows you to search on them quickly
therefore
huge kudos to them
so let's talk about what kind of
sensitive data we can find there
what we should expect to find as a
summary
i can easily say that it's a gold mine
for bug bounty hunters
also intelligence researchers can also
get huge benefits from it
let's explain them with examples
the first one is finding sensitive
google docs and drive files
lots of companies are keeping their
internal documents and files on those
services
to share their documents or files they
are usually get a
publicly shareable link since they think
that nobody will be able to
find them however it's possible to find
the files with specific queries the most
common url prefix
is docs.google.com
a slash company name and the extension
i really don't know why this works why
there is a company domain in the url
uh probably they are using a paid google
service or something but i really
couldn't find it
but it makes it easy to find sensitive
documents
belongs to a specific company
the other most common sensitive urls are
publicly exposed trello boards
some employees keeping their company
works inside their
personal trello boards and shares them
publicly
those boards can be useful for bug
bounty purposes
you can find sensitive data there and
report it
right away you can find those kind of
urls with the following search logic
it should include bots trello.com
and company name
the other obvious sensitive data is
admin panels or hidden paths
if you can't find a login panel you can
try default credentials
or even conduct brute force attacks also
you can search for paths of software
that has known vulnerabilities
the search query will be similar with
the previous ones
it should both contain company.com
and admin or private
or something else
another useful data could be urls that
contain
password reset tokens some websites
allows you to use the password reset
link for multiple times
as a search query you need to determine
that
what kind of settings are included in
the urls
and you need to search for them for
example
the search query should include both
company.com and
reset token
some websites are sending station tokens
with
get requests if you know a website does
that
you can search for them in order to log
in on behalf of other users
for example an example search query
could be
includes company.com and php session id
which refers to a session token
another important thing is map
coordinates
platforms such as google maps are
carrying the coordinate information
in the url section let's say an
intelligence agency is expecting a
terror attack on a specific location
they can search for those coordinates on
the export urls
if there is a match they can contact
with the short term service to find
which ip address
is used to shorten that url
there is also another possibility with
the map coordinates
some of you may remember there was a
there is a platform named strava
which is used for tracking your running
and cycling statistics
a few years ago strava published the
coordinates of places in which strava
gets
its signals it was looking harmless
initially
however osint researchers are realized
that there are tons of strava users
in out of nowhere in the middle east
it was a clear indicator that those
places are hidden cia basis
so let's say you are collecting
coordinates that are exposed
via url shortener services and you
realize that
a coordinate in which is located in
desert
is appears lots of times it might be a
good indicator that that
there is something going on that
location right
so just keep that in mind
so as a conclusion we can clearly say
that
uh relying on obscurity when hiding urls
is a bad idea it can be exposed with
lots of different methods as we
explained in the
previous slides you need to set proper
access restrictions on them such as
ipvite distinct and maybe authentication
etc finding sensitive data via url
shortener services
can also be a good bug bounty method
right
back hunters may find private company
data and get paid with it
for example you can find some private
company data
in google drive or you can find some
devops processes and credentials on
trello boards
etc and companies uh probably
will reward you for that
so and that was all from my site thank
you so much for listening
i hope uh it was an insightful
presentation for you all
and i hope you very safe thanks for
everyone and if we have any questions
and i'm willing to answer them right now
thank you