hey everybody i hope you're all doing
well i hope you're all staying safe
welcome to absec 101 at defcon 29's
absec
village over the next 30 minutes let's
take a journey from engineer to hacker
to appreciate the value that application
security adds to modern secure software
development
but before we jump into it though i do
want to call out that this is a 101
level talk
approachable to people who may be
unfamiliar with application security
it's what i like to call
cyber fried chicken
why do i call it that well let's just
take a quick bite and find out
just like you can eat a single chicken
wing instead of tasted an eaten chicken
it's just a small part of the whole
chicken it's part of the whole bird
you cannot say that you've eaten an
entire chicken you cannot say you've
eaten a lot of meat you've just had one
or a couple of bites
similarly
this talk will give you a broad
understanding of the field without
diving too deep or pretending to give
you the whole bird so in some sense
we're just dipping our feet in the water
also
i love chicken wings and i just needed
an excuse to pull that into the stock so
here we are
clucking along
in this talk we will learn what absec is
and how it is applied to secure software
engineering
we'll look at a fictional example of
insecure core development
and contrast that with sec devops or
devsecops
so we may all leave here with a better
appreciation for the role that
application security plays in modern
software engineering
but first
a word from our sponsor
have you ever felt like there were way
too many social networks to keep tabs on
like there needed to be a better
consolidation of our feeds so we can
spend
less time in front of our screens and
more
out in the real world
of course not
presenting
the newest anti-social network
tatter
tatter is launching soon on the web
mobile
ar
vr
and everywhere else your wi-fi network
reaches
sign up for an account or multiple
accounts today
tatter
it's where your friends aren't
now to be clear tatter is entirely made
up it's not a real company it doesn't
exist there is no alternative
anti-social network
called tatter
regardless of whether arrested
development would like you to believe
that through the usage of hacker traps
and other things like that there is such
a thing
in our case it's completely made up but
we can use it as a very useful case
study to see how the tattered team
in one iteration cut corners and
software development
and the consequences of those actions
and we usually only get this kind of
hindsight with an equifax breach or a
yahoo breach
let's instead learn from tatter and
their missteps
then once we have seen that play out
let's time travel back in time
and fix things by ensuring that tatter
follows abstract practices
and extrapolate that story arc to see
how it ends up
now
tattoos started as an anti-social
network from an ivy league college where
a bunch of students got together and
started writing code from their dorms
they were able to leverage their
connections with venture capitalists to
get funding to hire more engineers so
that the vc is good as i emphasized it
increased velocity
freshly funded and on a full belly of vc
money
tatter hired engineers to add more
features they added a dislike button
photo uploads advanced search and many
other features
in the race to launch publicly
many corners were cut
technical debt accrued
the focus was on velocity not veracity
mistakes were made
but they weren't caught yet
as tattered grew
they added a marketing department legal
department
and inevitably hr department
limiting their engineering workforce to
just developers
they just wanted to focus on getting the
product out the door
things were looking good for a product
launch soon the founders were happy the
vcs were happy
the engineers
let's just say they were happy because
this is all hypothetical anyway
from a soft launch to an open launch
tattoo saw increasing consumer usage and
engagement
in their first months they grew the user
base
all while it focused just on growth
however as these things tend to play out
one fine friday evening at four o'clock
in the evening
news broke that tatter was hacked
the news spread like wildfire picked up
by major news outlets
the tata team would have to pay up their
technical debt it seemed
as news of the hack spread and as time
went by
we are now at a timeline ahead of when
they were originally hacked hacked
so let's spend a few minutes learning
how tatter was hacked this will give us
insight into everything that went wrong
at tatter itself
and
to understand this better let's just
look at a high level architecture
diagram of everything that it involves
tattoos just had
a client so it was either a mobile
client using the app or the browser on
the desktop machine
communicating using an api
which allowed clients to interact with
the backend
in the back end we've simplified it here
as a database which is where the user
data is stored for the anti-social
network
now let's pick these apart piece by
piece to see the entire kill chain of
how they went from initial breach to
outright ownage
let's start with the mobile apps
tatter was meant as a mobile first
anti-social network
the focus of the engineering team was
around the mobile user experience with a
lot of attention paid to the visual
design
more than the entire architectural
design of the app itself
as a result many shortcuts were taken to
launch
and get the apps reviewed and published
on time
one such shortcut was that the api key
used to communicate with the back end
using the tattoo api
was stored in the code of the app itself
as a string in the resources file in
fact
so when the app was published in the app
stores
especially on android you know you can
download the apk and disassemble it to
look inside it
hackers did just that they got all of
the apks they cracked it open
they were able to extract the resources
from reverse engineering the app package
to find the api key because you can find
strings inside the strings.xml file when
you open up an apk file
so they had
one secret
additionally let's look at let's look at
the api
when signing up for a tatter account the
user is prompted to enter the email
address as part of the sign up process
we're all familiar with this is very
normal
however
if an account already existed that used
the same email address
tatter would prompt the user to sign in
instead
even at this point we're feeling
comfortable that okay it's information
disclosure at some level but
how bad can it be well
the username was pre-populated that
match the email address entered
now
even at this level if it feels like a
convenience you realize that anybody can
abuse the api and cycle through a bunch
of email addresses and find the
associated usernames
and to use this api you need the api key
hackers already had that from reverse
engineering the mobile app
so
they were able to cycle through a bunch
of email addresses find a bunch of
usernames
using the api which coincidentally was
not rate rate limited it did not lock
out after a bunch of fail login attempts
either
so using automation they could find
many usernames
and crack a bunch of passwords and
password cracking they could have used
password spring they could have used
credential stuffing
the point of the matter is
with the api key and in security in the
design itself
without rate limiting and things like
that they never had a bunch of usernames
not starting from the emails
and their passwords which they were able
to crack
now let's switch our attention back to
the website a given user's profile on
tata.com
could be navigated to using the username
so it was a url like data.com slash
users
slash username so slash jondo for
example
from the list they obtained by abusing
the api
now they had a list of usernames so
they essentially had a list of urls that
pointed to varied profile pages
and again abusing the api without rate
limiting
hackers were able to scrape the profile
pages of many users
including that of site admins and the
reason they were able to do this was the
accounts registered to
guessable email addresses like admin dot
com the username may have been different
but essentially were able to get that
from the api itself
and additionally the admin accounts many
of them used poor password choices and
they were easily cracked
so hackers were able to sign into the
admin accounts
using just the username and password
and view all profile metadata
now you might be asking
so what's the big deal if someone logs
into a nascent anti-social network with
username and password and you see five
posts
well password reuse is a problem so
let's talk about source code
a lot of the source code in fact all of
the source code for tattered was hosted
on the private github repo
and
the email addresses and the passwords
used for github was the same as the ones
they used on tata so they use both the
emails and passwords
between their header accounts and github
and in the case of their github setup
there was no two-factor or multi-factor
authentication either
so hackers were able to get access to
the entirety of the source code for
tatter product code internal
infrastructure code everything
and in there
they found
more nuggets they found database
connection strings they found cloud
storage account connection strings and
other credentials
so many other secrets in code and more
using the newly discovered credentials
hackers were ultimately able to breach
access and exfiltrate the cloud stored
customer data
additionally reminiscent of the yahoo
breach
tatter also stored customer passwords as
md5 hashes they did not use the most
secure hashing algorithm
so by exfiltrating the md5 hashes
hackers were also able to crack a large
number of customer passwords
which
we just re visited the problem of
password reviews
these parts were also invariably used in
multiple other sites
with these user accounts so you end up
having the username and passwords for so
many users not just on tattoo.com but
potentially their banking websites and
so on
this was this is huge
the case of the kingdoms was theirs and
tattered did not have clue
what was the outcome of all of this
for trivia the photo on you see on
screen is an image of the wildfires in
california from last year as seen from
the international space station
you see all the smoke emanating from the
forest which are on fire
going off the coast
into the pacific ocean
and this is what happens when a fire
goes unchecked till its impact is so
massive
now if you could anticipate that a
certain set of actions was going to
result in a massive wildfire
now let's say you saw somebody lighting
a match and you know they'd
start a campfire which you could foresee
would go out of control
and set fire to the forest
would you extinguish the flames when the
match was less lit
would you extinguish the flames when the
campfire got out of control or would you
wait till millions of acres were burning
it is a rhetorical question but it has
to be asked so we see tatter ended up in
a bad place
by russian new design
by using insecure development practices
poor implementation
and ultimately for the sake of our
discussion without an application
security program let's switch gears and
talk about the application security
program itself let's talk about starting
with a software development lifecycle
the sdlc we're all aware of it
we decompose the entire development
process into stages
traditionally in the case of a waterfall
model which
by the way the image you see on screen
is that of multnomah falls just outside
portland oregon
the sdlc traditionally comprises
an initial requirements phase followed
by a development phase then a testing
phase and then release
in this traditional model testing
especially security testing happens only
if the product has been built to some
degree of maturity and usability not
earlier
application security is a process of
finding fixing and preventing security
vulnerabilities at the application level
as part of the software development
process
this manifests as multiple parallel
processes that aim to find an address
issues early in the stlc
this is also called shifting left
because we're essentially shifting the
testing specifically the security
testing part
left in the stlc it doesn't happen
specifically after development it runs
in parallel with requirements gathering
with design with development with
testing there's aspects to each of these
so we can catch issues early
so we can extinguish the flames before
we end up with a wildfire
and here are some of the things that are
generally part of an abstract program
that we'll look at
kind of at a high level but should give
you an appreciation for the role that
appsec plays in software development and
an understanding of the processes
themselves
now let's travel back in time to when
tatter began as an idea
we saw how their lack of a knapsack
program and insecure practices ended up
so let's
try out in the time machine
let's infuse application security into
their engineering processes
and follow that development and see
where they end up
using some of the processes that an
appsec program uses which we just saw
we know there is a requirements phase
as part of the sdlc similarly an
application security program might have
a security requirements phase where you
have
personas you have user stories you
understand the security requirements and
lock down
security controls you understand to make
sure that there are security controls
i'll be very brief about this
but
the traditionally more prominent and
aware parts of application security that
people see especially engineers is i
think
things like thread modeling
so as part of the system design
since we're in tatter 2.0
tattered developers and security
engineers are collaborating on a threat
modeling exercise to identify the key
assets that need to be protected
once you identify the assets
you recognize trust boundaries between
the internal assets and external actors
and you ensure that appropriate controls
exist to secure your assets against
external threats
now since we're working
hand-in-hand between developers and
security engineers to keep the content
approachable to all collaborators
a diagrammatic approach is used in the
form of what's called a data flow
diagram a dfd
where the flow of data from external
users through internal infrastructure is
followed
as the data flows from external to
internal and potentially back you
identify
a bunch of security risks you identify a
bunch of threats you identify
controls to make sure those threats are
monitored mitigated reduce eliminated
and threat modelling is used to ensure
that controls exist to prevent things
such as spoofing
tampering repudiation information
disclosure
denial of service elevation of privilege
this the six that i have rattled off
you can remember it using the stride
acronym s for spoofing d for tampering
or for repudiation eye for information
disclosure d for denial of service and e
for elevation of privilege
once threat modelling is done as part of
the original design part of the stlc and
code starts being written
code reviews come in handy to try to
catch things early if they have been
missed so far or to catch issues in
business logic whether design secure but
the implementation is flawed
so code reviews are an essential part of
any collaborative software development
if you're writing code you're probably
hopefully getting that code reviewed
before you check things in
and when you have code you think is
ready to be checked in you seek review
and approval from your peers that's
normal for normal code review as well
a security focus code review is more
focused on the security controls it's
not that security engineers are trying
to understand the implementation itself
but anything specific to security
they want to make sure it's implemented
correctly it's implemented in a standard
way in a safe way
this is where you try to catch things
like custom crypto instead of using
standardized cryptographic algorithms
and libraries
it's a very effective way to get an
extra set of eyes or any new code that's
being added
now at this point
we have gone through the requirements
we've gone through threat modeling we've
gone through a code review let's assume
for some reason there's still things
that are missing so tattoo's gone
through things that hasn't done it at
1.0 but it's running 2.0 it's caught a
bunch of issues
but nobody's perfect or should i say
poor but is perfect so issues have
gotten in
and have been checked in
now this is where you use something like
a static application security testing
tool suite or a sas
so once the code's been committed to a
product's code base
additional sets of tools used to scan
this code
so a stash tool can be used to find
security issues that might have been
missed in code reviews so things like
secrets in source which we'll get in a
little bit
um and that you will talk about that
about how
you can use regular expressions and
entropy to try to find those things
insecure query construction where you're
not standardizing your user input we'll
look at that as well
and on screen you see the image where
i've used a security windows product
that scans github repos
and i forked the os juice shop repo and
i run the tool against it
you can see it flags a whole host of
security issues based on criticality
classifies them as critical moderate in
others
it's also found secrets in source
it has a bunch of other insights
it's very useful to incorporate a sas
tool as part of your
continuous integration pipeline so as
code gets committed in you're monitoring
the hygiene of incoming code
now what are some things that assass can
catch
one great example is
memory safety issues especially when it
comes to unsafe functions
now the function might
work logically but it might be flawed in
its own internal implementation and
these are things that could be a mess in
the human code review
so
as an example the c function string copy
strcpy which is used to copy a string
from a source to a destination
it is an unsafe function because it
doesn't check if the destination has
enough space to store the source string
and if the source string is larger than
the destination string buffer
it'll overflow the bounce and this can
lead to a buffer overflow which may be
exploited
so if such a documented unsafe function
is used a sas tool was will flag it and
there's steps to remediate this to use a
more safe function
so this is one class of bugs that can be
found and stopped
in fact mitral just came out with their
list of the most critical
bugs or vulnerabilities that have been
used in recent past and
buffer over buffer overflows or out of
bound rights was the highest buy a long
shot so if you can even cut down some
some percentage of that
it's better than nothing right you also
have things like input sanitization
where
you're seeking input from a user and
you're taking a space at face value
without
potentially sanitizing the input passing
that in as a parameter or an argument to
another function you're calling which
could lead to unintended consequences
this is how you end up with
cross-site scripting this is how you end
up with sql injection where you don't
sanitize the input you don't create a
parameter as query for example you don't
get rid of the html content that could
be passed in as textual input
which you then potentially pass on to
vulnerable functions
so catching this class of issues as well
because
sqli and xss has been
a fairly large scale
set of issues
for for a long time now
and
tools sas tools especially
are able to flag some of these
issues
we've talked about secrets and code a
few times now but why is this such a big
deal
for any or many intermediate processes
that an app does security is a big part
of those transactions and often the
security depends on the uses of a secret
of some sort so this could be an api key
an ssh key an odd token etc
committing any of these into source is
dangerous due to how easily they may be
found as we saw with tatter tatter had
it in their mobile app they also had it
in their private github repos
to store these secrets securely
use a key vault or a secret store which
allows programmatic access and retrieval
of the storage secrets versus hard
coding it in in the code itself
and the point here is there is no such
thing as a partial secret compromise
any accidentally disclosed keys need to
be changed
i will add even though it doesn't fall
under the purview of application
security per se the choice of strong
secure pass phrases over weaker
passwords
enough with the password one two threes
enough with the one two three four fives
let me ins
let's move to more secure passphrases
additionally in today's complex
interdependent software systems where
your code interacts closely with other
code from dependent libraries
it's important to protect the full tree
of dependencies
it's possible that the code you wrote is
secure but you are dependent on an
insecure library
software composition analysis sca is
used to better understand the
dependencies that your product uses
and this is also something that a sas
tool can find
so
once you have done
sas scanning of code that's been checked
in
and you have
through your continuous deployment
pipeline a bill that contains these new
changes or a version of it you can use
we can move to something that's a little
more dynamic in testing where you're
actually testing the product as it's
being run
and one such thing is fuzzing and let's
understand fuzzing with the joke which i
read a while ago
it goes this way
a qa engineer walks into a bar and
orders one beer
two beers
one hundred beers
negative one beers
one bear
an actual customer walks into the same
bar
and asks the bartender where the
restroom is located
the bartender halts and catches fire
fuzzing is the process of feeding a
system unexpected input
to see if can handle the unexpected
input appropriately
and if needed fails gracefully and
safely
so when you have a build of your
application under test
you can pass various sets of first input
and there's multiple sets of tools that
let you
create first inputs
whether it's an api or whether it's an
actual set of inputs itself on the web
app or any other app
for any code really where it's taking
input from a user instead of generating
expected input you can generate
unexpected input password and see what
happens
we've seen sas we've looked at
a static application security tool what
about a dash a dynamic application
security testing tool
since you have a build since you have a
version you can play with
you can scan it you can find out or try
to find out
using spidering all the different
so places that you can interact with it
you can try to
use yourself fuzzing but not
true fuzzing but there's a way of
passing in a range of inputs to the
application to see if it behaves
differently and on screen we have one
free open source popular
intercepting proxy wasp zap the zed
attack proxy
there's also
the proprietary but very popular burp
suite
and these are both intercepting http
proxies where any requests from browser
on that machine can be intercepted and
you can
at some level use it
as a netcat ask tool where you can pass
it into
the repeater
or the intruder and
mess with the parameters or the request
itself
now we have seen
starting with requirements threat
modeling code review
using sas using
dynamic testing and fuzzing
we're so far ahead of the curve compared
to tatter 1.0
and in this version of of tatter
in this version of reality tattered
launched successfully
its customer data is kept secure thanks
to early identification and remediation
of security issues
this was a result of a good application
security program
that worked with devops to prioritize
product security
and shift left successfully
tattered was praised for as privacy and
security and over time gained enough
users and revenue to allow its ceo to
purchase a ticket to space in the near
future
the outcome in tata 2.0 is that tata had
a good run the second time around they
had a successful laptop program
that allowed them to find and fix
security issues when the impact and cost
were still low
this was a far cry from the wildfire of
tatter 1.0
in summary
and by the way this is a great bit of
visualization from tanya's book alice
and bob learn application security
on the left you have essentially the
flow of the sdlc starting with
requirements designed through test and
deployment
on the right you have
application security processes that
occur in parallel with the stlc relevant
to that stage of the sdlc so you have
security requirements for design you
have threat modeling design reviews
for code you have code review and sas
for testing you have sas dashed pen
testing when you have more idea about
and a deeper understanding of the
product itself
so
by injecting and standing up in
application security program
parallel to your sdlc you're going to be
much better off from a security
standpoint
a couple of other things i wanted to
touch upon
all the issues that we saw tatter have
with their 1.0
are part of what is known as the os top
10
the owasp organization owaspec stands to
the open web application security
project
and it's an open source group of
security volunteers who come together to
identify
the top threats to web applications they
come to identify not just web
applications but apis and other
attack services as well
they identify
testing guidelines and the os top 10 is
a list of the top 10 critical issues
that that applica that web apps face on
the web today
and we saw things like um
injection we saw things like
authentication issues
they're all part of the wasp top 10 so i
highly recommend checking out the wasp
top 10 list to understand
the top 10
critical
vulnerabilities
that your web app or api or someone can
face
another project i'd like to highlight
from a wasp is the oauth juice shop
and the owasp juice shop is
a purposefully vulnerable web
application based around an e-commerce
marketplace so what it looks like
think of this as
like an online shopping website
that is it has specific vulnerabilities
for you to find
almost like a ctf
so here's an example
there is
what what they call a shenanigan
you have multiple types corresponding to
different
points of the os top 10 so you have
sensor data exposure you have
cryptographic issues injection in
security realization
and in the in the os dew shop app
you can
use it as your own ctf at home to find
these issues and what's even great about
this is when you find one you can also
look at the actual code that is
vulnerable so for example here
for this
i for this xss vulnerability i see that
there is a bypass security trust hml
function which should not have been used
because
that is the root of that vulnerability
and just for fun using the same
vendors sas tool
and for king zoo shop i ran it against
the jew shop source code and i found a
bunch of or the tool found a bunch of
issues that found secrets in code one
caveat i will call out about sas tools
is that it tends to have or tends to
surface false positives
where you think they are actual issues
but they are not and then there needs to
be manual triage
to find and
validate that they are
legitimate issues or not
because you don't want to bombard the
team the developer team with your 50
bugs
and it turns out maybe only four or five
of them are actually legitimate and the
others aren't
here's another screenshot of a couple of
the critical ones
i i see that it says http data to sql
database
points out the line of code in the file
open it up since it's a github project
i see that
a parameter from the request body is
concatenated with
or concatenated and added on to my
database query so i can use sql
injection to escape the rest of the
query and exfiltrate data or get access
to more data than was designed at that
point
it's also flagged bunch of secrets so
for example in code there's a private
rsa private key
committed to code which is not a good
idea we saw what happened header 1.0
again these are the kinds of things that
a sas tool can find out
so having seen everything that we have
seen for
an application security program and how
it parallels in stlc a couple more
resources i wanted to leave you with
tanya has a great book titled alice and
bob learn application security
it is a very well structured look at
abstech from
how you look at it from a design phase
through the development phase to how you
stand one up to how you apply to
different
types of applications themselves
the the tangled web is an amazing
amazing book as well which looks at
web apps html not html web traffic and
html some level http is what i wanted to
say
and when it comes to threat modeling
adam shaw stack's book threat modeling
designing for security is a very very
well structured book to understand and
ramp up and get familiar with threat
modeling as an exercise
i hope this session has given you a
better appreciation for the value that
application security adds especially
when run parallel to secure software
development
to learn more
please visit tatter.dev
it includes the notes we've used so far
including a transcript for
internationalization and accessibility
in closing i want to thank you for
attending my talk
take care stay safe and keep hacking
thank you
you