hi everyone i want to thank you for
coming to my talk titled
let the bugs come to me how to build
cloud-based recon
automation at scale for the first half
of this talk
we're going to go through an overview of
how you can build your own cloud-based
automation
ecosystem so we'll go through kind of
the framework the methodology
lessons learned and provide an overview
of a lot of the cloud services
that you can utilize and then just talk
about methodologies that we can apply
um so that way all the challenges and
mistakes
and lessons learned that i've had over
the past year and a half
you can avoid that as you build this out
and then for the second half of this
presentation and talk
we're going to do some live walkthroughs
of the actual environment show how it
works how it
runs and at the conclusion of this
i'll make sure to release the majority
of the code that you'll see today so
that way you can reference it
and hopefully you can take it and build
out your own environment and
drive that success for recon
additionally
i posted earlier this week a
supplementary article that i'll
reference on the slide here
and feel free to pull up the link and
kind of follow along as we go the talk
doesn't go line by line with it but i
think if you want to delve in and try to
gain more information throughout
feel free to jump in there and leverage
those resources there's a lot of live
code snippets and pieces that may
add some background context
so i wanted to share a little bit about
myself my name is ryan elkins
i'm a husband i'm a father i have two
kids
and by day i'm an information security
architect i spend the majority of my
time
learning about cloud solutions how to
secure cloud and
um my primary focus is cloud and
um eventually once the kids are in bed
ten o'clock rolls around i like to try
to be a security researcher and hack
things
when i can when i have the when i have
the energy and so that's
that's really what led me into this talk
is that
limited time factor where a lot of times
the free time that i
do have starts at 10 p.m at night
however at the same time i still want to
be able to participate in
learning and bug bounties and in doing a
lot of the
the hobby security work that i really
love and
not to mention that it helps me out in
my day-to-day career as well and
we'll share some different lessons
learned but leading into that i guess it
reminds me of a story and i'll i'll call
this the
the blockbuster dilemma and i'll ask the
audience here
does does anybody remember going to
rent movies and actually walking into a
store or into a blockbuster and you
actually go and pick it out
well if not i'll walk through the
process and i think there's a lot of
carryover traits to
the at least my experience with bug
bounties and
so you're sitting here on a friday night
you just finished eating your your pizza
that you just had to carry out dinner or
pizza
and you realize hey we have four hours
to do something let's watch a movie
tonight so
you get the family in the car you you
drive to the nearest blockbuster
or grocery store wherever they have
movies that you can rent
and you go in you spend about 45 minutes
trying to get a movie that everybody
agrees on
you eventually compromise you pick the
selection you drive home
and then you make some popcorn you you
go upstairs you drag down all the
pillows and blankets you get everything
in front of the tv
you eventually start the movie about two
and a half hours later so you're down to
maybe an hour and a half
you get the movie started and you fall
asleep you wake up the next morning you
realize hey i missed the whole movie
i'm going to try to cram in the movie
before i have to return the movie that
day or you get fined and you eventually
end up missing the day by like 10
minutes and you pay for an extra day
that is exactly the challenge with bug
bounties and i feel like
we are reliving the times of the vhs
rentals and the dvd rentals
picture bug bounty this this was me
before creating and why
what drove me to build this ecosystem
that i'm sharing with you today
is you get the kids to bed it's 10 p.m
i decide i want to hack tonight i'm
going to work on a program
try to learn figure things out so i go
in open up the computer i log into every
single platform
see what's available i review all the
programs i look at my private invites i
look at the public programs i dig deep
trying to see maybe there's a public
program somebody missed you do that
every time even though
you don't always find something and then
and then you also look at what programs
have incentives and
eventually you get to the point where
you select a program so you have your
program now
you need to find all those ssh keys that
you forget where you store them on your
computer or the what password did i use
to log in
so you get all that together then you
realize hey i haven't even started my
vps server and do so you
start at the vps server and then it
starts up you fire up burp suite or
whatever proxy you use
you start manually converting these
scopes over into your burp suite
console you get everything ready then
realize well i don't want to just go for
that
root c domain let's we should do some
recon so then
you figure out you update a mass and you
go out and you
look up the syntax you get the syntax
you load it up
run it you kick it off you wait about 10
minutes you get
tired and you fall asleep you wake up
the next morning
you just realize that you just slept all
night while
a mass is running you have this big
arsenal of subdomains
but you're going to have to wait another
few days to be able to tap into this so
your whole night was basically gone
because you only have a few hours your
time is so
so important and for me
i was living that over and over again
where i would do this i would have this
recon and then the next week i would
just decide i'm gonna go for a different
program and do and
it was a it's really a systemic issue
that
probably a lot of us face and i hope i
hope some of this resonates with you
and some of the challenges so i needed
to figure out a different strategy a
better strategy
how do i solve this and i loved it i
thought
a couple weeks ago i know kodingo um
michael
skelton and heath adams the cyber mentor
were
talking on twitter about comparing poker
to bug bounty
and that really resonated with me
because it fit the exact objective of
what i've been trying to accomplish with
this automation is
i wanted to build this scalable
cost-effective cloud-based platform
that can do full recon automation end to
end
so that way it takes all this raw data
and transforms it into actionable
intelligence
and for me the poker comparison is that
the best poker players they don't play
every hand
the hands that they play they want to
make sure they have the best statistics
and the best chances at winning that
hand and they play it
in the same way the time that i manually
spend on a program or
doing analysis i want the highest
likelihood
of signal versus the noise that i'm
going to
find a vulnerability whether it be a
duplicate or not i don't care at that
point
i'm just happy to have proper raw intel
or actionable intelligence from the raw
data
that i can act on so that's the ultimate
goal
and i feel like this program that i'm
going to show and walk through
we we come pretty close to that and and
we'll talk through that as well
so as i developed it i thought about it
and i thought there's
there's actually i categorized into nine
tenets that i
felt important to make sure that i'm
always guiding towards and
i definitely didn't hit every one of
those and there's still iterations and
mvp and post mvp processes but
eventually
i want to stick as closely as i can to
these nine tenets and i think it's
important as you as you think about this
as well that they all
they all resonate and make sense so the
scales horizontally part this
solution had to cover every bug bounty
program in existence i'm not smart
enough
technically to be able to be writing my
own exploits in zero days
but i am smart enough to apply the oauth
top 10 and search for those and some of
the
common themes that we see and we read
about and we learn about
the problem is i need to i can't just
hone in on five
things and ever find it so i need to be
able to scale that
breadth of knowledge across every single
program at once
the other part was scaling vertically so
i want something that now
now once i have the data the
intelligence information the
fingerprinting the
the server information about every
single program i can continue to add
new checks things so the the scales
vertically are kind of your bug classes
that you're checking for
across the entire ecosystem environment
of bug programs
the other piece is i wanted to be
cloud-based i don't want to have to
stand up my own server racks and run
capital investments of hardware and
things in my
my home office i mean that that had no
appeal to me i want to be able to have
on-demand
scalability the same reason that
businesses are driving towards cloud
are the same things that i wanted to do
i need that compute power when i need it
and i want i want to make sure that i
have it at my fingertips so cloud-based
is extremely important with no
on-premise dependencies the other piece
is infrastructure as code so
being able to deploy your infrastructure
repeatably and source control and
version control and all that it's really
important i've failed
quite a bit in this area because a lot
of times i'm just working on something
and i want to make it work and i can go
in the gui and click so
that's probably the area that i have the
most technical debt that i need to
revisit but i'm certainly something i'm
working towards
robust documentation for me i could
write a line of code
and i could forget what it does the next
day so being able to kind of
understand why i did something a certain
way have articles about it explaining it
that way i can revisit this and not have
to continue to relearn
that and then it also helps with sharing
to the world and to the industry and
making us all better for the same things
by having
some statements and documentation around
it low cost is another piece
it it can get really expensive as you
start standing up multiple vps servers
and running containers and
cloud services so my goal is to keep it
within the price
of finding one low vulnerability per
month
i'm scaling right on the edge between
like a low medium
but my goal is to optimize this as much
as possible so
that way i can basically one low
vulnerability per month will pay my
cloud bills now
hopefully once we scale vertically it's
going to gain more expenses but that
hopefully means there's more success
being driven and it's not as big an
impact financially because it's
generated in a profit at that point
the other piece is aggregating the data
analysis and anomaly discovery so
i want this raw data and i want it to be
aggregated across different tools and
pulling everything together so i don't
have to individually go and look at one
tool's output and then the next one
but bringing it all together to make the
most relevant intelligence possible
being a security practitioner myself
security best practices are
also important and i think that our our
program owners
care about that as well i mean they're
they're giving us the opportunity to go
do recon
find potentially damaging multi-million
dollar
impacts to these companies based on some
of these sub ones pri
and criticals it's important that we
take our own due diligence seriously
around the data that we're collecting
and harvesting from these programs so
that may be performing least privilege
changing passwords regularly
doing encryption and transit and that
rest and just general
good security hygiene then the last
thing is version control the last thing
i want to do is spend
a year on this and make a mistake and
have no idea what i broke or lose the
code or delete something so keeping this
in github under version control is
extremely important and
super valuable
so i want to try jump into this
architecture what am i actually talking
about why are you spending your time
today to learn
and i i think one of the things that
i'll ask of you as you listen to this is
looking at this this is the
the actual diagram of the architecture
that's built out in the cloud and i'll
share the code and you'll see it all
working in different things through
follow-up articles and code that i'll
release
and we'll walk through a few examples
within here it's a fairly large
ecosystem
thus far so we won't touch on everything
in detail today but
as you think about this this even though
this talk is more
focused around bug bounty and recon this
applies so effectively to pretty much
any area of information security and i
would even extend that to
any area of i.t or technology i mean if
you work in
incident response or doing any types of
forensic analysis or vulnerability
management
or even if you're a risk analyst or
someone that's maybe not developing code
or and it's
quite as technical of a role these this
is basically just taking data
turning into intelligent decisions and
using orchestrated workflows
and that's that's basically the extent
of this
but doing it properly can make you
extremely successful
and leading into this i feel like just
in general across the industry if you're
looking for a niche
to really fill it's leveraging cloud
to further mature your secu your
company's security program
and i think if you go out and search for
articles you're going to see a lot of
information
and data about how do you secure the
cloud how do you secure cloud services
you're going to see a few things about
how do you hack cloud hack the cloud how
do you do that
what you're going to find very limited
is how can you trans
leverage the cloud to actually transform
and mature your own security program
your own security processes
so this is just taking one niche area of
bug bounty and recon
automating that with cloud services but
this could absolutely
apply to all areas and i would challenge
you to
start to use this within your own within
your own job roles as well
so what happens in this ecosystem is
i'm i'm using i'm actually tying into
all the major
bug bounty platforms and i'm using um
a it's an existing github repo by our
arcadi tedelman and he's actually going
through and
making a lot of the api calls and
scraping the programs and sites so
eventually i'd like to get directly
integrated into every single platform
itself and not have that as a dependency
but i'm using the raw
json output that he's developed and
published out to the community and i'll
reference it here for you and
um so i just i i thank him for the
awesome
project he has out there and available
to everyone and um i've been able to
leverage that
and through that i can load block load
every single project out there that's at
least in
the major platforms and then there's
always the situations where you need to
load
maybe an external one that i want to do
ad hoc or something
i also have the ability to make an api
post request that will load that program
so once you have
all the programs loaded it's then
literally just a
get request with the program name that i
want to initiate this recon cycle on
it steps through every single thing you
see on this reference architecture
it does all inmate automation it kicks
off
scans it does um domain enumeration
through
project sonar it has a mass module that
i'm not going to talk through in detail
today i'm not using it as extensively
right now
i'm more interested in hitting some of
the the data sources directly with their
apis
but then once it gathers all that data
from the the initial scope
it will then kick off a httpx scan that
gets kind of your urls and gets some of
the
base ports and the fingerprint details
and the response codes and everything
and after it has that it will circle
back and it will do a web crawl of all
those seated
base urls and then the nice thing is
iteratively whatever urls it finds it
does the json
the js link finder it looks for
everything api endpoints
analyzes all that pulls it back gathers
a list of every url found in there
whether it's in scope or out of scope
does a double check to make sure that
everything is back into scope creates
that list
feeds it back into the the recon process
does the recursive recon and then goes
back through
and it will keep cycling through until
it get until it gets
tons of information all the raw output
then as it's done and completed
it will do commands that aggregates the
raw output all together
and you basically get into structured
formats that you can
you can query you can search you can
sync it to persistent systems
and run tools like a sem grip or a sift
or a
regular expression whatever you want to
you could even do
i'm using nucle nuclei templates locally
against the
the file all the responses that are
stored across millions and millions of
web pages
so there's unlimited amount of depth
that you can do
once you have data basically curated
in a usable format
the overall infrastructure is broken
into really three main components
there's
program generation program operations
and program analytics
and that's kind of how i bucketed
everything that goes on
so this is an example of what you would
start with so if you want to do
recon on in this example you can see i
have tesla so i want to say i want to do
tesla this is
initial operation the nice thing about
this too is that if i want to just do
like a crawl i would say operation crawl
and it's going to crawl the existing
data i have for it or i could say httpx
and it'll
do an httpx scan against the urls that
we have for it but in this case this
just says
do the entire recon so it's going to
kick it off and at the outcome of
without touching anything else with just
this post request i'm going to have
dashboards like you see below where
i i can do have real-time analytics and
filtering capability around the urls
their status codes their responses
and this isn't going to tell me
specifically vulnerabilities at this
point
but this gives me in a way to filter
through and structure this so i can have
the most highest likelihood of signal
through
all the extensive noise that we've
discovered through the recon process
and then i think graph database is a
really powerful
visualization tool to quickly see those
anomalies that are sitting
and then you can also see relationships
and this is a neo4j
that's running within it's the neo4j
cloud service but
it's issued through as a third-party
google cloud api so i have that i'm not
using as extensively because cost wise
i don't think honestly i can afford it
to run it across all the programs it's
great for like one or two programs but
once you start hitting those
node limits that it just gets too costly
to expand that out so maybe
once we start returning having some
return on investment we'll we'll do more
there but i'm using
network x locally with the python
libraries and you can still
generate some raw analytics around some
of the individual programs which is
fascinating
so it really starts with the program
generation it's important to have all
the programs loaded ready to go and
that's where i i talked about
arkady titleman's um bug bounty targets
github repository that's just
awesome and you can go through but i
will highlight one of the challenges
that i faced early on is with scope
being so important that we stay within
scope the
bounty the program owners care about
scope for a reason and a lot of times
these companies they want to be able to
have these programs and
do this but they truly are sometimes
systems that may be
our legacy or they know that it's not
going to handle the traffic
it might be that they're just not ready
to have
everything within their ecosystem or
environment to be hit by
thousands of hackers across the globe
and i think that's fair and i think
at the same time for us as researchers
it's important to honor that
scope and make sure that we stay the
best we can within scope
and what i quickly found when you try to
automate across everything is that
it's difficult because the scope even if
it's in the same exact field
varies significantly so i've i've tried
to build and
iterate over different scope types where
you might have a url with a
wildcard at the end of it you might have
a domain with
the subdomain wildcard but does that
mean that i can also wildcard other
pieces of it
and then there's cider ranges and ip
addresses and you'll kind of see the
different variations here listed
and one example i learned pretty quickly
is making sure that github is not
included as the in scope whenever they
have a repository there because i
quickly learned that you start crawling
github you get a lot of
responses and it can really take a long
time for unnecessary scope so
i made that mistake once and have fixed
that pretty quickly so
but tons of lessons learned like this
that i never anticipated when i started
on this journey of
developing this environment and then you
can see if you want to load a new
program the post request payload is
right here where all you have to do is
provide the program
the scope in the scope out the platform
and the invite type so that way i can
filter and protect any private
um private repositories or programs
without whenever i'm doing demos and
things i can better hide those
so that's that's pretty much all it does
and it is all loaded and stored inside
an aws dynamo
db so it's structurally maintained and
then i have a time stamp field where i
can then
trigger and say once i have this and i
run recon i set the timestamp so i know
when the last time i ran this again so
if i want to start doing more
automation around if i haven't touched
the program in 30 days let's
automatically kick this off and those
are all future enhancements that i could
do but
just trying to attract the program so
you can see in the screenshot right now
i have 653 programs loaded
i know i have a couple other places to
pull programs from so that's
definitely not comprehensive yet but
that's that's certainly enough to get
started with
and then the operations piece is
everything is initiated via api
so there's there's really there's
nothing that you have to do
outside of an api curl recall request if
you have so
it's you you make the crow request to
the aws api
service you can set your own custom
cname i have api.brevityammotion.com
that's that's kind of where i host my
blog and other things and and i've
leveraged that so i
make the request there what it does is
it feeds in and you can proxy that api
call to what's called aws
lambda and picture the aws lambda
service has a
serverless function that is running
and you think about like python definite
or python functions
that's pretty much you want to leverage
these as small modular components that
do
one task and they do that single task
well and so i proxy that api call
and then the parameters i send in the
post or get request to get filtered
in as arguments into that function so
it's just a
quick way to make a function api based
and that's awesome i mean just think
about that power alone
where in your company that you write a
lambda or a python script or have some
some type of thing that you want that
had takes arguments and it maybe runs on
the server today
throw it in there proxy it through the
api gateway now you have an api call to
some type of data processing that you
could kick off and that that's just so
powerful in itself
to be able to apply to so many areas and
then once those functions run
they output all the data into
into s3 buckets which are object-based
storage that just scales
along with however much room that we
need the other piece is ephemeral
workloads so i'm i'm actually anything
that i perform
actively against the target so those are
the the web crawls and the
nuclei scans or anything like that that
is all
stood up dynamically on ephemeral
droplets within digital oceans i use
digital ocean
droplets for everything that i do and it
pretty much
writes a custom startup script
where i start up the droplet
automatically via api it feeds the
startup script to it
and the last command that i have in
there is shutdown so the startup script
also completes all the work synchronizes
the data
queries or i guess queries aws s3 pulls
all the data in
and then runs its processes and then the
last step is synchronizing the data back
to s3 so we don't lose the output
and then it shuts the instance down and
then i have a
a lambda that runs every five minutes
that's always querying my
my digitalocean account and anytime that
there's a
droplet that's shut down it will just
terminate it so i don't get charged for
longer than the
duration so worst case i have five
minutes of extra time
run time from every operation i do and
it's just
over and over sets up ephemeral for
every process then it's very modular
so that's that's how we do it i'll share
all the code you'll see snippets you'll
have
i think that's already in the
supplementary article so you'll be able
to
kind of see how i do it but it's it's
really that just-in-time dynamic
approach that i've been leveraging
the last piece is the program analysis
and i i know i already mentioned the
data is stored in aws s3 buckets but i
have
buckets for raw data i have data or
buckets for refined data i have buckets
for
data sets that i pull in from like
third-party sources if you think like
the max minds
apis for asn lists or geoip and all that
and then there's a curated slash
presentation area where
you can do the presentation pieces where
i might point
what you see here is a quick site
dashboard and i'll point to that data
and it correlates aggregates it all
scans it through and presents it back to
me and um
where i can have the the visualizations
and the summarized output of it so i can
make better use of my time and not dig
through thousands of
individual text files which i know a lot
of us face and struggle with today
all right well with the time left i
think we should try a live demo let's
let's try to walk through the the recon
engine end to end
in the workflow and as we go through it
i'll explain what each step is doing and
try to fill in all those pieces and i
think that will probably
wrap up the rest of this talk but
there's going to be tons of follow-up
information i advise that you
follow me on twitter check out the blog
and i'll continue just to post more and
more information so you can replicate
big chunks or if not all of this
yourself personally
so let's go ahead and jump into the aws
console here so
i know earlier i mentioned you can kick
things off for the crow request and do
all that you can actually test directly
within the ecosystem as well so i'm
going to just jump straight into the api
gateway this is
this is the api structure that i
currently have set up within here
since we're going to test out adding a
brand new program to here that doesn't
exist anywhere else
so far we're going to go ahead and check
out our post request
and we can kind of see the structure of
how this is set up where it's going to
go through the api
it's going to get proxied straight
through and hit this
back end lambda for a post request
and one thing i don't think i mentioned
yet is that
i do all of the developments within
cloud9 which is another aws service
which also makes it so i don't need any
crazy powerful laptop
all i need is something where i can get
to the browser and do that development
so i'll show you we'll hop into the
cloud 9 development environment here so
this is
this is what the ide looks like it looks
very similar to like a
visual studio code or something like
that but what we're going to do is i
have some test cases written out here
for for example so as i'm going through
deploying new code or anything
i can go in here and get some quick
examples so we want to do a post request
in here
so i know this is an example if i want
to test this out so let's copy this over
and let's put this into the api post
request gateway so we want we want to
test out this service and
if it's a get request we'd put it in
here you can add some custom headers or
anything like that
we're going to go straight to the
request body and paste this in now
this program already exists so just for
demo purposes this
shows something that's completely brand
new let's call this
recon let's actually call it defcon
recon village so we'll start that's
going to be our program name we're going
to look for
here's our scope in so we're going to
give it a wildcard to make sure that we
can do
some additional recon steps uh add
another site and let's
put a an out of scope site as well and
then we'll call it an external platform
and this is all my own so what we're
basically going to do is crawl my blog
for for sites so we're going to kick
this off here and say test we should get
a response in a second
which tells us whether it successfully
loads or not
let's hope for success program
successfully created
so it created the program and it's most
likely entered into our step functions
functionality here so let's let's go
into aws stem
step functions and this is our
orchestration unit so we can see it's
running right now so let's let's click
in here and see what it's doing
i love how you can get kind of a visual
status of what all it's going through so
as we
as we go through here it hits the first
choice
and it kicks over to program build based
on the operations we send it
so it built the program or loaded into
dynamodb
and then what it does is it checks the
build status it feels successful
it goes in and goes ahead and does a
sonar query against the
rapid7 data set of basically the four
dns services and so it it does an aws
s3 bucket query using the athena service
here
and it's in progress so it's pulling
those results back so if we're going to
go into the
athena service here we can actually go
and look at our
query history and we should see
the current query it's running against
the rapid seven forward data set
and so here's here's what it's running
exactly
and it just loaded i guess i clicked it
so we we have it here so this is what it
automatically generates pulling the
results back
and once it completes it looks like it
completed successfully
it would have stored those results into
an s3 bucket and now
it's doing running httpx so since this
is an
active process against the actual target
being my website
it's going to kick off a digitalocean
droplet to run this command and it's
going to build install it
everything from scratch so if we go over
here into my digital ocean account
you can see that this droplet just
finished
getting created the ip information and i
have a naming structure so i know at
least which program it results to so
while this is running it's probably
loading up everything
it's going to do all the installs we'll
keep an eye on that but we'll talk
through some other steps as it goes
so let's let's jump back to the step
function so
so far what is really neat about step
functions is
you can do something called callbacks
and one of the limitations with
the lambda service is that a lambda can
only run for 15 minutes
and you're charged for the duration that
it runs and so when you're doing
something like a web crawl or an https
scan
you really don't want your lambda
function waiting the entire time for
that to return back so what we did was
we called the lambda the lambda kicked
off the
program gen or the operation built the
droplet and then the lambda completed so
it closed so
right now we're not getting charged
anything within the aws ecosystem for
running
and then this this step function state
machine is waiting on what's called a
callback
so whenever i built the droplet in the
in the startup script i added a callback
task
that says when you get to the end of
this the last thing to do before
shutting down
is to make an api call to the step
function service in aws
and tell it that you've completed your
journey and then
success or fail and when it reads that
in it will it will determine where to go
next in the workflow so
this workflow is essentially in a paused
state just waiting on that callback to
happen so
we'll give that some time and see when
that happens
what else can we jump into here as we're
as we're talking about that let's
let's check out some of the the actual
lambda function that ran
for the program build i think that
that's pretty interesting so let's
let's pull up these are our lambda
functions here what they look like so
we're running httpx so let's
look at operation httpx and so
basically it's listening for event
context every single lambda has this
same kind of format
so it was sent in event into it
and you can see the events get triggered
as it goes down through so
the operation hdbex we can click here
and we can actually look at the step
input and see that this is what was
submitted to that lambda function so it
submitted the program variable
and the operation so when we go back to
that lambda
we will be able to see where we
basically go
we pull in some of the data parameters
and then we check does program exist
and does operation as long as we have
both of those things we know we're on
track
we have the right status it's going to
go ahead and run so it's creating some
initial file names of the domain storing
an s3
doing and then here's some other
commands where it says prepare httpx
generate the install script so let's
let's hop in and check out the install
script i think this is one of the
coolest things that it took a while to
figure out of thinking through
how am i going to manage scripts across
600 plus programs what if i want to make
a change to the syntax or what if
there's a project update where
i need to add a different command or
something like that when you're reliant
so what this does is it generates these
scripts
based on the runtime dynamically just in
time so i don't have like a bunch of
scripts just sitting ready at program
build
this actually gets generated on the fly
immediately before the program is
or before the operations initiated so i
only have one central place within the
code that i need to update these scripts
which makes it really easy to make
changes and modify and keep things up to
date as we go
so what we'll do is we'll jump down into
the code that does that
and what it is is you can see here like
for
generate script httpx what this is doing
is this is writing
this is writing the file using python
code
generating this file here so this is the
this is actually the operation what i'm
using so you can see the httpx command
here
my passing in my program name parameters
it gives me a program name defined
output so i know which one it comes from
in the json
and then i'm telling it kind of the
syntax of what i want from it
everything and then store your responses
so i'm storing raw responses locally on
that
droplet that's running and then
eventually you get to the point where it
gets down it gets everything finished it
processes it at one point i was actually
tarring all the responses up
and just storing them trying to save
space and compatibility but it became
easier
and more difficult and time consuming to
continually untar the files
do that especially when you're getting
into the millions of files individually
so it just does a
copy over to s3 this is the sync script
that it's just another script that's in
the supplementary materials you could
you could reference and see
it does that wait and then when it's
done this is that callback to step
function saying i finished everything i
wrote all my files back to s3
i'm all done here go on step functions
to the next thing
and when i write this file this is all a
file written to s3 so when this
when this droplet runs it downloads this
file to the server
and then it knows based on the variables
to run run this directly so you kind of
it's like a script to run more scripts
is essentially what it's doing
so let's go back and let's check the
status of our step functions here so it
looks like hey we
it succeeded it was successful and both
getting the httpx data and then
processing it so if we go back to the
digitalocean place we'll see that
hey check it out if you remember this
first droplet we created was called
brevity-httpx and it's gone so
that means that the lambda function
that's checking for the shutdown ones
already ran
deleted it for me completely gone no
history of it
and it just it's standing up right now
or our web crawl
of the of the new data that it's found
so
we'll let it go ahead and crawl and
we'll jump back to our step functions
and see what happened here
one thing i'm doing with the httpx data
that i think
is helpful and it's not out of box from
the command is when you have so many
different programs and you have so many
output files
it's one thing to add kind of the
program name to the naming convention
the output file
but what happens when you start to
aggregate the data directly within those
if it doesn't have the program reference
and i found that to be
fairly consistent across a lot of the
tools is that it will have the
information the urls the domains
but what happens when i do my aggregate
analysis and i see some random domain
i have no idea other than trying to
track down and figure out the source
file that it came from honestly that's
one of the most difficult parts is tying
it back
to the the original program so what this
process http x does is it
reads the json file within just python
code pulls the json and actually adds an
additional two elements so it adds the
program name and then it adds
the base url because it the httpx output
actually has the full url has the domain
it has the parameters
but it doesn't have that base and why
why is the base important the base is
important
for things like nuclei templates where i
don't want to just go after
something dot com slash and then my
nuclei
payload i want it to be something dot
com slash api slash
slash whatever and do that and whenever
i can do that generate the distinct
aliases are distinct url base urls
without any parameters or any additional
on there
it just makes it fewer and fewer bad
requests that i'm going to be sending to
these programs as i
as i do any type of fuzzing so as the
crawl goes it's going to go
through it's going to do a similar
process where i dynamically wrote the
crawl script it
feeds the crawl writes it back to s3
does all the processing analytics
and then one of the neat things that i
think is really neat is
amazon has a service called glue and
it's
basically a crawler where it can crawl
across an s3 bucket
so as security reachers or bug
bounty hunters we probably have hundreds
of output files from the same
tools over and over so i'll continue to
use httpx as the example
so what happens is i will aggregate all
100 plus
httpx commands that we have across
programs i'll put it into the same
directory structure within a bucket
and then i can set up what's called a
glue crawler
that will crawl this bucket and it will
automatically
extract the data so you can see here
where it goes through
it finds i have i've run the recon so
far on 91 programs i don't have all 600
of them yet i've been kind of going
slowly trying to monitor
two things as we go so right now i have
91 programs
you can see there's just over 10 million
unique urls across those 91 programs
that i have indexed
and then it will do a smart indexing so
it basically creates a
structured index table that points to
the raw
json output files of httpx with these so
so this is a and now i i can use the
athena service and use presto functions
and query syntax
and i can query programs now based on
the httpx aggregate
and you can see i i believe i have the
the program name these are the custom
the program
name and the base url i mentioned are
are not out of box from there but from
the processing i went ahead and added
those in so i can do
smarter queries against it so when we
hop back to it looks like step function
is still running we'll we'll give it a
little bit more time here
to run let's check on yep you can see
the server's still going
so as we're waiting let's jump into
athena and i'll show you what i mean by
some of those queries so
if you we pull in all the httpx data and
let's
let's use tesla as the example this was
a simple query i wrote but if you saw
like the
the table syntax there and you can i'll
expand it here
here's all of our all the different
things so if you're trying to
let's say you're doing fuzzing or brute
forcing apis you might want to look for
anything with the word
api in in the url like just like it's
written here
and then with a status code of 200 or
maybe you don't want 200 maybe you want
to try to send payloads that are going
to bypass a 403 forbidden or something
like that so
this query would run i hit run query and
it returns
it looks like two results we'll watch it
run here
in about four seconds it crawled 12 12
gigabytes of data in five seconds and it
returned
these two results here so let's let's
adjust this i i haven't tested this one
out so we'll see what happens if we do a
403 status if we get any more
or any less i don't know what will
happen here hopefully we have some
results that will be fun to talk about
but let's watch this run
all zero results that's no good
but you kind of see what i mean where
you can then start to think about well
what do i want to start doing downstream
now do i want to look for specific url
parameters or anything you write this in
here
and then what's cool is the at the
completion of every one of these queries
it outputs it into a csv file into an s3
bucket
so if you take that ephemeral approach
for a nuclei template or something
i can do the same ephemeral startup of a
droplet
install nuclei and then i can pull this
query that only maybe i write it so that
it only selects url so that way i just
have a list of urls that meet this
criteria
download it to the s3 bucket and
automatically pull that as my word list
that feeds in the urls
that it puts that targets for from an
intelligent standpoint so
i've just increased the likelihood that
i'm going to have a hit without having
to send millions and millions of
requests that are just
never ever going to to have a finding a
result so
i think i think that's pretty cool and
i'm really excited to dig in more and
more
doing doing some of that there
all right so when we hop back into our
workflow and step functions it looks
like our command's
finally completed so let's check it out
i think one of the fastest ways to do is
do the
the glue athena search so let's let's
see if the
updated table is actually successfully
updated if we do a
quick athena query here we'll we'll use
this previous one and let's
let's create a new query
and then we're going to say select the
url the content length from here
let's just get out the let's take out
the url api
while card and status and let's just
skip straight to program so we have
where program equals what did we call it
we called it
defcon recon village
order by content link that looks fine
let's let's run in just see if it worked
see what we have here
should take about five seconds and check
it out here's our program here's all of
our
our urls and ordered by content links so
we can kind of see what returned what
didn't
and i think the last thing the show is
just the recon dashboard so you can see
i can
automatically update this dashboard to
pull in the data so i have
was that eight just over eight million
rows imported here
into these analytics and i've i've
already custom sorted by program here
i don't think i have it filtered on
private and bytes public input so i'm
not going to hit the drop down
but you can see that i have 91 total
programs count of urls
this has 8 million urls generated in
there i have a dashboard of
i can look for like random response
codes that might be anomalies so
if i scroll down i bottom ranked or we
have some that's a 300 response code i
don't even know what that is but
it sounds like something we should
probably check out or a 202 or a 203 if
if i have one result out of 8 million
requests
it could be an anomaly or something so
then we can go through there and do
and you'll see the different status
codes that we have you'll you'll notice
pretty quickly these are out of scope
there's certain things that i'm not
going to attack or do payloads or
exploitation
but it's still interesting to see the
third party libraries that you might
point off to or do and
what i think will be interesting in the
future as we cross examine these using
graph databases and things are
how many commonalities do we have across
the programs as a whole
all the programs are they reaching out
to the same javascript libraries is
there something that if i could go to
this connect facebook stk.js
are there 60 programs out of the 600
that maybe have that same
relationship so if if i go from a
facebook bug bounding program standpoint
find a flaw in the sdk.js file i can
submit that to facebook but at the same
time let's submit it to all the other
programs and try to find those
double up on on some of the profits
around this so those are some of the
relationship
techniques that you could use and apply
so but as we wrap up today i
really want to thank you for checking
out and just taking this journey with me
and
i know i i don't think i mentioned
earlier but this i probably put in about
500 hours honestly into this project
there's
a lot of code a lot of lessons learned a
lot of trial and error a ton of failures
and i'm i'm actually starting to see
some success and this is finally getting
to kind of the
the exciting part after the grind so
feel free to reach out to me i love
talking about this i'm
excited to learn and meet a lot more
people share ideas and
and learn how some of you are doing your
automation so i
really appreciate your time today and
enjoy the rest of the
recon village and i think the team for
having me thank you