good evening good morning everyone and
welcome to if nothing goes right or she
left
um i'm very i'm happy to be talking at
absec village this year
i've recently been branching out into
the field of application security
um because i've been seeing some trends
and
and dealing with some things on the
right hand side
of um applications that has led me to
believe
that when nothing goes right we
generally need to look for the solution
on the left
now who am i in short i'm currently
an assistant professor at noroc
university in norway i originally came
from south africa
where i was a director in incident
response
i've also recently been working
extensively with medical manufacturers
specifically in the security space and
the application security space
the most interesting thing probably
about me is the fact that i am the
cyborg that i have a little cardiac
shocking machine that keeps me alive and
keeps my heart behaving
i'm also a good for defcon but the most
important thing
is i hack things i build things i
protect things and i
investigate them and that's kind of me
in a nutshell
so what are we going to be talking about
today what is this talk all about what
is she going to get on a soapbox with
what am i going to share well i thought
it would be important to share how i got
to the point that i
am today being um talking about
applications here so
yes application security from a forensic
perspective
i want to talk a little bit about the
infosec color wheel my opinions on them
which may not align with lots of other
opinions but this is how i
see the world i also want to talk a
little bit about the developer
perspective
the security perspective and the dfir
perspective
now everything was inspired by two
amazing books that i will talk about a
little later
and then out of this process spawned the
five logging philosophies that
developers can follow to consider
what to do with application logs
now my journey you know happened
by accident that i got into the world of
digital forensics that was before
incident response was really a big thing
in south africa
we were still dealing about you know we
pull the plug we do dead box forensics
reconstruct file systems
we go down to the hexadecimal values
that's kind of where i lived and
breathed
however for me there was always a
disconnect as to
the motivations and the why the why do
people do what they do
fast forward a few years incident
response became a
necessary evil and we started dealing
more and more with incident response
which tweaked my interest because
it's such a fast-paced adrenaline-filled
position
you never knew what was going your way
now one of the things that i
learned was that i didn't really
understand i could understand and
reconstruct from friends in perspective
what happened but i could never
understand the security perspective or
the how our attacker got in what i did
is i started learning more about red
teaming that was the first place that i
went and i was like
well i want to understand more about the
attacker mindset i wanted to understand
how they thought how they view things
i wanted to understand what steps they
take when
hacking an application or a device
now once i had that it surely had an
effect on my incident response i was now
able to anticipate where my evidence
would be
again i fell short on well what
recommendations do i offer now take into
account this is the very siloed industry
right in the beginning
you did you didn't do digital forensics
in
insurance response and security however
as things
progressed and shifted we saw that we
needed
a more blue perspective a security
perspective we needed to understand
how to keep them off how to isolate them
how to
you know keep them out now understanding
how the attacker thinks
what evidence is generated and it was
fairly easy for me to slip into security
when i say easy i mean it's not
impossible it was hard work on my part
but
i started to learn what controls can be
used
to keep a system from being broken into
now once this was done i realized well
now i know how attackers get in
so that i can anticipate evidence i know
what
remediation actions i can take with that
wouldn't affect any of the evidence that
has been created
and i thought i was done at this point
well big surprise i realized well hey
there's a reason that i don't quite
understand some of the artifacts that
are created
now if you're in digital forensics
you'll know that a lot of reverse
engineering
is done on airport to understand how
operating systems
work when they have been updated well i
did exactly the same
i started learning more about
application development building
and how software together trying to
understand the building blocks that i
am set to attack to defend and to
investigate
so that was kind of what led me to where
i am today
now the color wheel for this talk and
this is is just
to kind of frame where i'm going with
this conversation is
that red is our attacker but what's
important to understand about raiders
they only have to win once
they only have to climb through a single
window or break down
a single barrier for them to win
so they are always the team that wins
because it's
kind of a you only need one now blue
is the defenders of the role and they
have to
keep red out and they have to also hope
that no one has accidentally left the
window just the jar enough so that
someone can climb in
blue is also there to anticipate that an
attack is going to happen
they have to see into the future and
know that hey
you know there's a there's a window
that's going to be left open tomorrow
at 11 a.m now in this talk
purple is the dfir team and the reason
that they are purple in this instance is
because they make use of both
red and blue tactics to investigate and
examine and find the blue course or
something that's
you know happened in the incident
response perspective
we are there to inspect and find out
what's happened we're the ones that find
the clues like
like your crime scene investigators our
crime scenes our computers and systems
and networks
but we need to find out what was left
behind by red when neighbor again
now green is probably one of the coolest
teams that i
wish i could be part of and i wish i was
able to do the cool things i do
they're the ones that build things from
nothing they the ones that make amazing
things happen
so purple blue and red are there in
support
of applications and devices and castles
and
cool things that green team builds so
for this talk this is how we will be
framing it
and i hope you enjoy the trip with me
now let's look a little bit at the blue
team
the blue team is our defenders of the
realm
our protectors are knights in shining
armor
they are often in application security
um
specifically in those environments that
aren't very agile or very fast moving
cooled
in too late and not frequent enough
or too much now this is a very
contradictory statement
so often you will find that teams build
an application
and don't consider some of the
fundamental security about the
implementation phase
so they built their application to be
you know
good and they've built in all the
controls they could think of but they
just don't have the knowledge
installed in them to consider how this
piece of application integrate in the
grand scheme of things
and there's sometimes where security is
called in too late
however as you'll see later in the talk
there are solutions
into actually allowing you to build
better
with compromises in mind
now security is often seen by the green
team as those that bring
more more work for them to do they're
the ones that create more noise and more
work and more things the green team
that's already over stressed
and burnt out and overworked has to deal
with
but they are their jobs
whether we like it or not is to defend
and to protect and to anticipate that an
attack is going to happen
so let's look at the defender's view
because throughout this talk you'll see
one glaringly obvious string that
follows through that all of our
perspectives although
looking at the same information is very
differently
so how does the defenders or the blue
team view application logs
why are application logs important for
the blue team
well it's again the what who where when
and why
and those are kinds of things the dfir
practitioners live by
so the what they want to know what they
are concerned with what information's in
the logs and what information shouldn't
be in the logs
so are we disclosing secrets app keys
perhaps even sensitive phi or pii
they are concerned with the content that
should not be there
so the who they want to have
auditability now auditability in
in the security sense um or having an
audit trail in the security sense
is not exactly the same as having it in
other industries that
it's it's not about an auditory checker
list
auditability for the blue team is can i
see
who's accessed something can i see a log
on a log off
can i see um session management can i
see the workflow in terms of how users
access the system
so for them they want to be able to have
a security order to trail
now this is also where a lot of
applications fall flat
because we do note that there's been
successful logons but not always the
failed logins
are actually reported in the application
logs they are crucial for blue's ability
to
actually see into the future that
there's potentially mischief on the
network
and that's only one of the instances
when it comes to the who
that is important it is successful
um logons versus unsuccessful logons as
well
as where users potentially escalated
their privileges or
access so that's the complete security
audit trail
if you would like to know more about
that contact me offline and we can have
an
in-depth conversation now the where for
the defenders
is well where are the connections coming
from
whether it's a device a geo location if
you're allowed to have that information
or just the identifier of a user they
want to know that when they're looking
at it
at potential traffic that it's coming
from where it should or the connections
originating
from a device that's registered on the
application
now the when they are concerned with the
date and time step because it gives them
an
idea of when something took place now
it's much the same across the board
um it's just the reasoning behind the
wing we all want to know
and do timeline analysis but the
perspective from a blue teamer is
we want to be able to identify trends on
certain days
now the why uh security wants to
understand
why someone was you know was able to
access and authorize and i also want to
know
why things have changed because we want
to know
be able to track the workflow from a
security perspective that everything is
processing as it should
connecting as it should and people only
have access to what they should
and if they change there should be a log
that tells us about it
now the red team our attackers are
dragons the one that stomped down
um the castle walls they are also the
team that always wins
sad but true um the red team will always
get in
now they only have to find a single
window that's left open
a single door that's unlocked or a
vulnerability or they might just
literally brute force their way in i've
seen ping tests
where people would take down the
financial database while they're
breaking in
somewhere else so create noise in one
area of the castle and attack it in
another side
it's kind of the same in real life as
all when you look at war tactics so they
are also our chaos card
they kind of the the grenade that's
thrown into
into a room and explodes with glitter
and everyone's covered in litter because
everyone's going to get affected by the
spool out
when an attacker breaches your
application and access a system within
um an information that's stored uh
there's
there's many things that happen in the
fallout and often
the blue teams blame because they didn't
defend against the best of all
the fingers pointed towards you know the
purple team that
you know wasn't able to reconstruct the
event
and then lastly the finger gets pointed
at our boulders and saying but you
should have built it better
but my question is we always say you
should have built it better but do
does your green team have the education
the knowledge
and the tools to build it better and by
tools i mean
thinking processes i don't mean some
snazzy um
tool that you've bought that does
application security for you that looks
at code
i'm talking about the human element in
your team
now the rate team if the
there's a weakness or there is a
vulnerability they will find it and they
will exploit that it's their jobs
they are meant to attack they are made
to find their way in
now they also find application logs
super useful
but their perspective on them are vastly
different
so they want to know what information is
in your locks
and as someone that does that has done
some hacking
specifically um in the medical device
manufacturing field
that is one of the places where once we
reverse engineer and get
access to where we go look for
information to learn more about your
application
another what so the what is what
information can i possibly
gain that is recorded within the logs
so red teamers are very good at doing
log analysis because they want to know
exactly
how your workflow works they want to
know exactly what information it holds
how it connects how it communicates
how the user ids look they also want to
see
if there's potentially any print bodies
that contain information which is often
where sensitive data disclosure happens
within the application level logs
is when we just dump data out and we're
not aware and fully cognizant of the
fact that
there is an jwt token in there there's
potentially at keys and secrets
so your print bodies are one of those
areas that you need to be very careful
with the information you record
even if you think that it is innocent
and there's nothing that can be used
that piece of evidence could be the one
puzzle piece the regimen needs to
compromise the application further
now the the where they want to know um
where it connects to how it communicates
and
where it goes so what the workflow is
for sending data receiving data
receiving updates and things like that
so it's
it's again as you saw with the blue team
it's very different with the
where is we want to know from bluetooth
where is it coming from
red wants to know where is it going
another why
they want to know what sensitive
information
they could potentially use to exploit
the business now this is not your
internal rating potentially but a
malicious actor or an
insider threat that can use this
information to blackmail an organization
reputational damage when it comes to
incident response it's all very much a
shaming game which is something i do not
agree with
we should be creating an environment
where people feel safe to disclose that
they have
suffered a compromise it's not so much
about the fact that you have been
compromised i mean that's nothing new
we've all been hacked with all even
though we don't might not want to admit
it we've all suffered at the hands of
cyber criminals it's about how you
recover
and how you handle the fallout that's
the real important thing and i think
that's not celebrated enough
now the when um they want to know if
they can hide
information from you they want to know
if they can manipulate the logs and
clear it or erase it
now one of the things that i found that
you know
application logs often don't have is we
don't make sure that we can't do things
like log injection attacks
or you know we can if we have access to
a local device is there a potential
ability for us to manipulate the logs
clear a tamper with the evidence you
know the information
and overall the law of content integrity
so one dose does for instance not many
things right
but what it does do right is the ability
is the security event log
it's quite an awesome thing for a
forensicator to pull through with the
contains a lot of information
um specifically around when log
messages are deleted or changed or
tampered with
um it's got a fail safe that is built in
now i oh am i
always say to the building teams that i
work with that you need to
consider your logs as a source of
evidence you need to consider your logs
need to have forensic integrity
your logs need we need to validate that
they haven't been tampered with and that
the information we're receiving is
information regenerated and not that an
attacker generated now green team
our builders and our creators um
our awesome team um
this team is responsible for building
the applications or the systems that we
protect and attack but this team is
often under immense pressure
both for time and to perform
um when it comes to finalizing um
a release there's always stress
there's always um pressure that's on the
team
so they are the team that potentially
is the one that is suffering the most
their focus is on delivering the product
for the client client in a timely manner
their focus is getting um
the product out it's building something
that the client wants and sending it on
its way
this is also the team that outnumbers
every other second you know
every other team um and we'll look at
the numbers a little later
now builders or developers also love
their locks well the hope is they do
because if they don't then if you have a
really big problem
um i recently did a survey which i will
send out again about how much enjoyment
you get from looking at your logs
now i love looking at logs because i
feel that they are
the secret windows into how an
application works
now green team builds the logs
and i always say application logs are
built by developers for developers
and they also look at it from the what
when why who and where
now the what what has happened can i
tell that my application is
actually functioning correctly can i
tell that it's not
am i able to debug it am i able to solve
field investigations
that's kind of the what the when well
can i determine when an event is taking
place
especially when i have to debug it
especially when i have to do field
investigations
the why well i want to know as a
developer is my application
functioning if not why is it not
functioning if anything's changed into
an unfavorable stage
state i would want to be able to know
that it's done that
the who well we need to identify the
user that
the information is coming from or the
system that it's coming from or even the
component that's generating it
one of the things i always say to to
teams is when you are creating your
application logs a good way for you to
be able to do analytics is to have a
category field within your events
this is this category field is
associated with a section of code that
generates that event
and if you see multiple errors being
generated the same category that is an
indication that perhaps you should go
look at the code and see why it's
generating that
logs can be immensely powerful both from
all the perspectives if they are built
in a manner that
pulls in the information we've also been
able with the category fields to
identify when components are breaking
or when sections of code is functioning
correctly or optimally functioning
it also helps in terms of incident
response when we see a certain component
be more noisy
in the in a certain amount of time than
previously it also helps us to baseline
what is normal behavior and what is not
but this the perspective of the
developer is often to debug
to determine whether the system is
functioning correctly and to measure the
performance
another purple team which is the one
that i most associate with it's the
magicians in the realm
we often called in when red has climbed
through the window and blue
has been unable to defend we're kind of
the merlins of the world
but not as cool as merlin
we are there to look at the um
logs that the dragon hasn't produced and
draw our magic potion
so that we can see what was happening in
the past future and prisons
we often look at information and put the
puzzle pieces together
but often we are asked to make calls or
base our findings on no information
little information
often the logs are not enough or too
much
and what that means is over verbosely
logging is a problem for purple teamers
it creates
noise it creates confusion
so i always say that what we like to see
is a clear timeline a clear workflow of
how your application's doing what it's
supposed to be doing and how
it is logging the information um
we also the thrill seekers in the realm
because i also think we did kind of sort
of like to punish ourselves
because we know that the logs aren't
aren't gonna be what we need
it's kind of part of the game where you
know there's always a missing piece that
we have to
do this is why we layer our
investigation
we have various sources of information
and logs but
often the application logs is the one
that tells us what is going on
the most so if your logs are
built with the purpose of reconstructing
events
or workflows and cleaner not overly
verbose
it makes the jobs of purple teamers so
much easier
now we also love logs we probably the
people in the realm that loves the logs
the most
i mean i i kind of would do it like 24
hours a day i would
look at logs um and profile people
we also able to look at the logs and
tell
you know whether or not the team's
organized let's produce the logs um
there is a saying i have a saying clean
code produces clean logs
when code quality is good i love quality
is good because logs are a byproduct
of code that's generating the events
now the dfir perspective of an
application logs is we also want to know
the who when why
what and where but slightly different
than the rest of the teams
we again want to be able to identify
um authorization that's taking place
access that's been granted
as well as session management that's
been done
we want to be able to reconstruct who
accessed a cert you know
cert you know the application at a
certain point of time
it's all around the timeline of either
something that's happened that needs
digital forensics to come in
or an ongoing active breach now the when
is very important
when you are creating applications with
many moving components to generate the
timestamps and various logs it's
important to consider
that you have a universal timestamp
this means that it is best to stick to a
single
date and time slam format as well as to
stick to perhaps considering whether
you're going to lock utc
versus local time on the device but also
know that sometimes the device
has an additional log that you might
want to ingest that is now in local
system time
and the rest of your logs are in utc
time zones are tricky even for the best
of dfir
practitioners doing time zone time zone
translations on a global national
is very hard so it's always good it's
good practice to stick
to a standardized date and time stamp
across an organization
not just across a certain application
specifically if you have a security
operation center that is going to be
ingesting these logs
you need to make sure that your logs are
standardized
you need to set a structure a format an
approach for your logs across your whole
organization
now the why we want to know what has
gone wrong in your workflow
that has caused the attackers to gain
access to the network
um or we want to know if something's
changed why it's changed when something
changed from a favorable
way of working or the normal behavior to
abnormal we kind of want to be able to
tell from the logs why this has occurred
then again the what what steps were
taken what was
accessed um is what we want to know
because we all about reconstructing
what has taken place or potentially
where an attacker is going to
and that's all done by using kind of
profiling
the data and profiling the attackers
through
the steps that they take now we also
want to know the way
can we identify whether the attacker has
managed to gain access to other
services or systems you know that he
pivot
uh can we determine um where it
originates from and i don't mean that to
an ip address it's more just you know
identify a device that's potentially
connecting in that shouldn't be
connecting in
attribution is often hard the wear is
often hard
um it is it's not an easy thing just to
like csi makes it look that we press a
couple of buttons and ooh the hacker
is sitting behind his computer and we're
able to find him
um often the focus is for us on
who when why and what
the where is very hard to do attribution
to a person
but we do try and do attribution to a
computer or to a device or to a
user account now
when you put all these colors together
all these ways of thinking
this is kind of a road map that we need
developers or the green team to have
we need them to understand how they
tackle things we need them to understand
how the dragon's gonna attack it's
important also to know that your
green teamers are your best penetration
testers
green team makes up for some of the best
red teamers because they understand the
code that they've built they know where
they're potentially
vulnerable it's just about considering
that
going through a process before you
actually put your hands on a keyboard
and write the code is to consider well
what do i not want to have happen what
kind of attacks do i want to build
um you know build the capacity against
a good thing to run or a good thing to
have your
green team do is play a game called
cornucopia
it's something that i tell that i think
is so good because it's threat modeling
from a developer's perspective
it's an inverted stride it's an almost
project
um i will put the link in the chat for
you guys to find
it is really an awesome way to
think like a red teamer to build better
but we also need now that we have our
green team
understanding how they will attack the
application or
you know the code they are able
to actually produce
better code that perhaps avoids that
attack being
able to be carried out but we also need
them to think like the blue team things
we need them to anticipate moves that
are going to be made
and building the controls that either
tells us that it's going to happen
or that protects it from having to
happen so if a developer or your green
team thinks like an attacker and they're
able to defense
defend against the attacks they created
we already you know applying security
principles
before we've even designed any code
but we also need them to think like the
magicians
consider from a forensic perspective
what information do you need to generate
specifically around potentially the
vulnerabilities or weaknesses you've
identified
because then we're starting to do a
layered approach we know where we weak
we have building controls to defend
against it but we've also built in
actionable information that tells us
that something's busy occurring
or something that has occurred so again
the future past and presence is
important
now once i identified that this is kind
of where i wanted to go i
i realized that i didn't quite
understand how software development
worked at an enterprise organizational
level
so i got recommended two books by
friends and said
to read them and see if i understand the
world more
and absolutely these books by gene kim
is probably what is the foundational
information that i've i've worked on for
the last year
building into my standardization of
application logging that i want to build
better logs and applications i want to
build the logs that i need for future
breaches
i want to build the information that is
needed in
so i wrote the unicorn and the phoenix
project and there's five
ideals that are mentioned in the book
um philosophies ideals the first is
ideal is locality and simplicity
you know things need to be simple we
over complicate things as human beings
and often that's it from both security
red teamers green teamers and purple
teamers we all
over complicate things sometimes we
should just keep it simple
philosophy two is focus flow and joy
a happy developer is a good developer
is a productive developer it's kind of
where it comes from we need to have joy
in our lives and and these five
philosophies
are philosophies that security can take
on board as all they apply
to us as well the third philosophy is
improvement of daily work
right when you when it is easier and
there's processes in place for you to be
able
to securely develop software where
security is embedded
right from the beginning it's also less
work and less expensive than
actually bolting it on at the end
there's lots of organizations that would
say to me but we do devsecops
and when you start looking and working
with them you realize they do
they don't quite do that they do what i
refer to as dev
ops sec where security is still bolted
on the end which is a super expensive
exercise because now we're trying to
change things and patch things but
we really can't because we've already
hit it to the point where you know the
application's done we're just busy
dealing with some things it's easier to
build a boat sturdy than to try and
patch
holes while the water is trying to
stream in
the fourth philosophy is psychological
safety
we need to have an environment where we
can able we are able to discuss
that an incident has taken place we need
to be able to have
the safe space to say hey our
organization got breached
uh this is the root cause analysis and
it's kind of in the incident response
process one of the most critical thing
is
lessons learned and what i always tell
my clients is this is the point where we
don't
point fingers and often you'll find that
blue
points at purple who points it green or
you know any other color we're always
pointing the finger back instead of
saying well
we are what did we learn how do we go
forward
and actually have a safe psychological
space to have these conversations
we live in an information security world
where companies are ashamed on a daily
basis teams are ashamed on a daily basis
for when an incident happens
well it's no surprise that that a breach
has occurred because rate always wins i
think the focus should shift from
while you've been bridged delay let's
see
how you handled the incident let's see
how you conducted yourself
and often you know having the correct
narrative
on an incident and having the ability to
not
shame your teams when an incident has
occurred
yields and fosters the whole culture
that people can make mistakes and learn
from them and do them better in the
future
now the first idea from the books is
customer focus we want to get our
products out we don't want to have
you know products that never see the
market we want to produce
products that keep our customers happy
now these
five ideals kind of made me think of
well
we should have five ideals for logging
logging is a very important component
and shouldn't be
a byproduct of code that you've designed
it should be something that is thought
of you should be
identifying what types of events you
would like in your logs and building
your code accordingly
now based upon these in various
conversations with different developers
as well as friends in the industry these
are the five philosophies for logging
that we came up with
now again philosophy one is much like in
the book from the unicorn and the
phoenix project
keep it simple keep it clean and
detailed enough
now logs should be able to be
read by anyone that knows nothing about
the application and at least be able to
fundamentally understand what they're
looking at
if you struggle to do log analysis on
your logs it should be
you should know that there is a problem
that has been overcomplicated and overly
proposed
often when we log too much it is when it
becomes
impossible unreadable and difficult to
follow
you should keep it clean know what goes
into your logs
it is really a big problem when we start
looking at legacy logs
and as iterations of applications have
happened
the logs get super clunky and super
messy
and just keep it detailed enough be
purposeful with what you put in the lots
and how you put it in your lots remember
control the information that is printed
often i see print bodies being put into
logs
they contain too much information
but again we have the other extreme we
might not have enough
information it's about finding the
balancing act between
all the perspectives and knowing that
you are able to say that
you you are able for example to do a
connection without having to print the
body
that still gives you the ability to
determine whether the account was
you know able to be given access
now philosophy two is keep it tight and
create metadata and use it
um and one of the one of the things that
i've seen a lot in application
um logs is sensitive data disclosure
it's still something that is a big
problem for organizations
when you consider having information
such as phi
tokens those kind of things and consider
tagging it within your code um
perhaps assigning at a privacy level and
then when it is printed in the logs
differentiate what
what you want to have in your production
logs versus what you don't want to have
in your production logs
um it's never a good idea to have the
same information that you would have at
a debug level in production
but i mean that's something i have you
know that's something everyone knows
but how do you build in the controls
that accidentally you don't have the
disclosure of the information
and that is done by tagging your
variables of data
with the appropriate phr you know
privacy levels
so right in the beginning discuss the
different kinds of information you're
recording
then discuss what the privacy level
should be and what steps
will be taken with each associated
privacy level
as to whether they're going to be
redacted whether they're going to be
just printed
or whether they're going to be hatched
there are different ways of dealing with
that but consider data privacy as part
of
your log design process now philosophy
three is the one that much like
technical debt everyone hates keep it
clean and
keep it focused now i mentioned a little
bit earlier that as
applications develop so their lobs grow
so with every update or new iteration
there's new things are introduced into
logs
and we start dealing with log data we
start dealing with logs that
or legacy systems that are still logging
or a certain
component could be producing excessive
lines within
in within logs deal with
at least one item vlogging date per
sprint
clean up one thing because those things
add up if
every developer on a team just deals
with one leg no logging debt or legacy
log issues it starts becoming achievable
it's all about breaking it down into
small manageable pieces now philosophy
4 is assumed at some point you will be
compromised and log accordingly
now if you have done a threat model for
example using something like cornucopia
or stride
and you know where your application is
potentially weak
log accordingly if that is where you are
going to be attacked
or most likely attacked build in the
controls within the logging
that triggers certain actions that are
being taken
it also helps when you have when you
ingest these logs that you are able to
trigger on certain events and be able to
identify a bridge much faster
now philosophy five consider who has
access to your logs and how they are
stored and how they are transported
often these application level logs
reside on a third-party application you
have no control over
you need to do some mutual
authentication to
an integrity checks to determine whether
or not
the logs that you have been sent is
indeed what you've received
uh now there's various ways of doing it
this is kind of applying the forensic
process um one of the ways
we do it is we would have a forensic hat
we'll have an md5 hash
or show one hash that's regenerated on
the contents of a file
now if you change as much as a single
space or a single
full stop in the document the hash value
will be different
now yes everyone's going to scream that
there's hashing um
hashing collisions well there's a paper
you can go read about it
even though that is possible and can
happen hashing is still a very good way
to
do integrity checks on contents of logs
um or forensic images for that purposes
because the content still needs to be
readable and ingestible and possible and
makes sense
and when you are generating a collision
often the the data contained within
would not make sense or fit the format
of a log
so it's important to do those kind of
random checks to
just validate the integrity of the locks
in the end of the day when a breach
occurs your logs
are one of the sources of evidence we
look and
if there's no integrity to them we can't
verify that they are correct
how can you do your incident response
and say that the logs were not tampered
with
so this is important that you need to
protect it at rest
you need to protect the contents you
need to protect how it's seen so
consider these things when you start
designing your logs that you can build
in the additional features for it able
to be
done in a very forensically sound manner
as i said all roads lead to the logs
inevitably if we would like to do
incident response from any perspective
and big surprise
red blue purple and green all do
incident response now you're going to
say veronica well
how the hell is that possible well great
team looks at the incident response from
a perspective of
well what do i need to gain from the
logs you know what what
do i need to do to break in
then we have blue team that is from the
perspective of their incident is well i
need to look at the logs and i either
need to know something's coming
something's happened or something's busy
happening
then you have incident responders where
we do the traditional incident response
the breaches occurred we want to know
who what why and where
uh we want to do timeline reconstruction
as well as
hand over the network to blue teamers
with information on how they can better
keep this from happening again but then
we have green team that also has an
incident response
they do it from a debugging perspective
they do it from a perspective of well
something's broken that needs to be
fixed field investigations
so all the roads lead to the log slots
are super important and they should be
given the consideration
that any other feature is given on a
application
in fact i firmly believe that logging
should be a feature
within application development that is
considered prior to building an
application itself now
uncle bob that has been a big
inspiration to me
in terms of how he talks about the
profession the profession
of being a developer and heart code
should be
um you said be kind to your future self
when you go back and you have to go read
your code after a year because
something's gone wrong it is often very
hard
context shift back to that that is
similar for for indicators or dfir
practitioners
so be kind to your future for indicator
when creating your logs
often we have to go back with them often
too late
not soon enough and we have to
reconstruct an event that's taken place
and it's sometimes also time critical um
as a
live breach is ongoing so as much as you
don't like to go back and looking at
your code we also don't
generally enjoy going back to look at
lots that are difficult to work with
so i said that blue team wants to
identify faster and the reason being is
that the average days that it currently
takes to detect to reach 280 days
unless the attackers have made it known
earlier
this is the average globally which is
far too long
and it's my firm belief that if we have
better application logs are ingested
into a centralized team we will be able
to do detection earlier i will be able
to baseline what is normal
what is abnormal and be able to trigger
upon that
now 80 of all breaches in the last year
contain phi
passwords and secrets and a lot of these
were found in in the logs themselves
so again sensitive data disclosure is
quite a big thing and your logs are the
ones where they often slip
in in many organizations
now let's play the numbers game because
i said to you how do we fix this or how
do we make this all better well
26.6 million developers in the world
green team
is is largely big this is also
estimations based on some research
um that i've done now there's about four
million cyber security professionals
worldwide um
now i don't think this number is totally
accurate but even if it was
um more than that it's it's still a lot
less than green
and in this is blue red and purple are
all considered a cyber security
so that shows you that for the solution
to work we need to empower
green to think like blue attack light
red and investigate like purple
so how do we do this well it is by
taking all the colors
and putting it in a combination team
it's bringing the
blue perspective to defend to the green
so they are able to build in the
security controls that we need um
it's also looking at where their
potential will be attacked because
then they know what security controls
the building as well as building in the
information that is needed to do the
investigations
the now when green has the ability to
consider all these things you will
start seeing better application logs you
will see logs where
we have information that shows the
security workflow
we'll see additional logging around the
weak areas
so that we know that we have the
information to find
what has happened now
read understands how to attack blue
understands the defend
and purple understands to investigate
but we should be instilling this
information to developers
giving them the tools and the mechanisms
to build
what we need in red
um anticipates that green inevitably
will build something with a weakness in
if not today at some point it becomes
vulnerable
blue once you know blue expects green to
build devices that red can't break into
but we've already said red always wins
but from my perspective i cannot build
the logs better once an incident and
taken place if the locks don't exist
they don't exist i don't
in real life wield any magic that is
able to produce
information that's not there i am
dependent on green team as building in
the necessary logs that i need to
investigate their future bridge
so how do we do this we turn them into
forensic coding ninjas
we share within our organizations our
information
we help them to develop from just being
builders
into builders that are breakers builders
that are defenders
holders that offer indicators we do this
by sharing our knowledge and sharing the
workload
having conversations passing on
education
and it will do both all you know blue
team and purple team the world of good
to go understand the constraints and the
difficulties that green faces
the complexities of code you know
software development
um i certainly feel that i've come full
circle in my journey to understand all
the perspectives and understand that the
way to solve what we're dealing with in
the field
is to influence how applications are
built
and the only way that that can be done
is to influence um green team
to become the all-star team to become
the team that encompasses
everything that understands the
perspectives of everyone
so in short your logs should always be
simple and structured structured is
super important
we want to be able to grip it you know
do searches again to do analytics or
ingestion
and if you're having to write passes for
every application you are doing logs
wrong and you should consider changing
this immediately
your log should not contain more
information
than is needed it should only contain
the bare minimum that is needed for
debugging purposes investigative
purposes
as well as general workflow purposes
if it doesn't need to be in the logs for
any of those perspectives
based cuts out avoid print bodies as far
as possible
find different ways to write the same
event
you don't want to disclose sensitive
information
80 of all data breaches occurred because
sensitive information was disclosed
often there is accidental information
disclosure and it happens within your
locks
this leads to future breaches though
so always consider when you are
designing your logs
that at some point you will be breached
it's not a question of if it's question
of when and whether or not you can
determine that it's happened based upon
your logs
whether you will know that it's happened
unless the attacker tells you
all roads lead to the logs