hello everyone and welcome to my talk
um here at the appsec village
this presentation is going to be about
devsecops merging security with software
engineering
um i hope you enjoyed this presentation
so um just a little bit about myself um
i'm an information secret specialist and
senior threat researcher at trend micro
here in canada i have over 10 years of
experience working with application
security
i've been a devops and devsecops
enthusiast since 2017 when i started
researching about this topic and this
new
approaches and right now i just signed a
contract with a publisher to
write a book about the sockops and i'm
starting to write it
so yeah uh other than that
i have a blog personal blog at
katanosac.com
where i write some articles
about application security cloud
security or container security
and there is also all my contact
information there my social media
profiles and my all my previous talks in
my previous presentation and slides
since i
started uh talking about security in
2011
so so yeah that's it
okay so the agenda for uh for this
presentation right first
i wanna tell your story and talk a
little bit about my experience
uh of trying to uh
apply deploy uh apply security
principles and devops and devsecops
principles in different organizations
i worked in financial institutes
institutions back in brazil and i worked
in some tech startups
here in canada as well so i'll tell you
a little bit about that and i hope
that's
that's gonna be helpful for you uh we're
gonna try to understand like why there
is a uh
this um this kind of constant argument
between
the need for the word devsecops instead
of just using devops right and
understanding that and and
we're going to go into a little bit more
details there
um why why there is this need of of
devsecops right
isn't isn't just a different name for
the same thing and doing
a kind of uh just kind of a it's a new
buzzword that's been
um going on for a while now and and why
some
organizations um say that they're doing
devsecops
in and when actually they they're not
right
um yeah just creating some uh
also talking about like
how devsecops is much more than tooling
right and
and having those tools in your pipeline
although it's very important it's not
it's not that
that's not enough right so other
approaches
and other uh processes are also
necessary
and yeah just a little bit about that
and also
what what worked with me during my
uh my experience and my um
my approach applying those principles
in into different organizations and of
course
what didn't work uh as well so that you
know
that might not be as effective although
i tried
okay so before we start just getting
getting some things out of the way since
every
devops or devsecops talk needs to have
at least these three things right has to
have this
the infinity loop talking about
themselves
has to have this unicorn with the
rainbow and
has to talk about shifting left so we're
just gonna leave that here
uh before we begin so that you know that
okay yeah it's it's we're going to talk
about that
but it's not just that right
so let me tell you a story oops okay
let me tell you a story and and just to
paint the picture
for you as this was my experience uh in
a previous
employer right that was a tech startup
which was growing i think it wasn't
considered startup anymore they had over
um 1200 employees right
and out of those 1200 more than
300 of them were developers
right and they didn't have
any application security specialist
right
and so i was the first application
security specialist to join their team
and they didn't have pretty much
any kind of security tools into in in
their pipeline right they didn't have
any kind of
static analysis dynamic analysis or
software composition analysis
right there was nothing like that
there was also no secure code reviews
there was some code reviews but not
focused on security
per se and the only
um the only tool that i can say that
was related to security but it wasn't
properly set up and properly configured
was a web application firewall
in front of some of those applications
right
so you can see right away that there is
a big challenge here
right i joined their team and and i
needed to implement some
uh application security approaches and
projects
not just not just spin testing but also
training the developers on on secure
coding
and also there was
a project regarding pci compliance and
and that's one of the reasons why the
secure coding training was
was needed as well right so there was a
lot of challenge there
and and one of the things was
that that i did that was really
interesting was
okay right i i realized i can't do
everything on my own and i need help
right
so uh with that i started talking
with different developer teams
and also q a teams to understand
uh uh their process how they work which
tools do they use
right uh which just like okay which ide
do they use and
how how that works how they get a a
code out of the door right so how how
does
like they decide on each sprint what
they're going to work on
and and they have those tickets assigned
to the developers
right and so on how everything works so
the first months were
understanding that and i think that was
really important to connect with them
right but then i created
a secure coding training and
when i went to talk to the
their engineering managers right to okay
now i created this material based on the
oas top 10 proactive controls
and i i want to
like have some time and and have some
developers join at least on the first
session
so so that we can teach them and talk
about secure coding right
and so the training was very interactive
it was uh uh i had some some plans some
hands-on exercises so that they could
see
uh why the vulnerabilities happen but
also
how they they uh how they could fix it
right so that's one of the reasons why i
didn't choose the
oasp top 10 which is the most
traditional uh a top 10 document of oas
but i chose the proactive controls which
are much more
focused on the developer right much more
focused on
fixing the vulnerabilities or or even
avoiding the vulnerabilities from
happening
right if you implement them correctly um
and that's what i wanted like
i i don't want to teach a developer how
to
exploit a sql injection or exploit
across site scripting
i want to teach them and and show them
how to avoid them from happening in the
first place
right so yeah the training took uh
uh it was going the plan was for the
training to take four hours
uh so half a day and and right away
once i scheduled the training i received
some pushback
from the engineering managers right
they said no that's too long and and the
developers can't stay
uh four hours in this training that's
not relevant for them
and and that's not gonna work so
yeah the first that's that's i know i
understand that that's a struggle
right so i had to cut the training down
to two hours and and
because of that i had to remove most of
the
a hands-on part of the training right
um and okay like we did it with uh a
smaller group of
uh developers so that they would kind of
evaluate the first session
and see how that worked and and yeah
okay that that went well finally but
i just want to give you a warning here
and
make sure that once you join
a team especially in knapsack and
especially if you're trying to apply
uh some devops and devsecops approaches
that you need the support of the
engineering managers and the leaders
right
it's not going to work if you try to
push those changes because there are
sometimes they can be cultural changes
and
it's not the way how the the company
used to work and
and and you implement different tools
that might
affect their process right so
if you don't have their support it's
going to be really really hard for you
to
apply uh apply those and implement these
processes
okay so yeah this is the beginning of
the story and we're gonna talk more
about
later about what worked and what didn't
work
in regards to this process at the end of
this presentation
so what is a software engineering right
uh moxie marlin spike the
creator of signo said that software is
never finished right
you can't just stop working on it it's a
part of an ecosystem that's
that's moving right and for myself
i i totally agree with him and i say
that software development is like a
mmorpg right like
i'm a sieve muted player online role
playing game
it's an ongoing process right it never
ends there is always quest
to do there is always places to go
things to buy right so there is always
different stuff
in on those games that it never ends so
i think that
uh software development is just like a
mmo rpg
so why
do you need to worry about soft
engineering
if you work in security right there is
this great quote
quote that uh chris hughes
reminded me this this past week
from this book agile application
security which by the way is a great
book
that says that security teams that try
to reduce the risk
by minimizing change rather than
supporting the developer teams
right uh supporting the development
teams right
to realize their ideas in a secure way
are doomed to increasingly irrelevant in
an
agile world right so what does that mean
right
if you if you try to keep
working as a security person
on the old ways of uh blocking
uh blocking the development of new
changes blocking
uh the deployment of software if there
is some vulnerability that's not even
critical
or really relevant um what's going to
happen
is exactly that the security teams are
going
to be bypassed their security teams are
going to be ignored
and i've seen this happen right so i
i had a real life experience with that
and and then what in the end nobody wins
right
because security is not going to get
their way and
uh developers are going to ship a
software that's not safe that's not
secure enough because
it didn't go through any process right
so we need to understand that
and i think that's one of the big
reasons
why application security engineers need
to understand
code and need to understand how to code
so that they can
understand the developers mindset right
we
always talk about oh we need to have the
the attacker's mindset the hacker's
mindset
right but in this case here the
application security engineers need to
have the developers mindset so that they
understand their struggles and
understand that sometimes
it's not up to them to choose what to
focus on
right if they're going to develop a new
feature or if they're going to fix a
vulnerability
sometimes it's it's up to the uh
leadership
team or the project managers or the
product owners
there are the ones that decide that
right okay
good so what
what really is devops right what is
really devops
how does that work so
devops was created in as a
evolution of the agile methodologies
and it's it's not just tools
and processes right it's it's a set of
cultural changes and practices
that increases the organizational
ability to deliver value
and deliver code and applications and
services
at a higher speed right so in a way that
there is always
uh new features and and new applications
deployed in production
that they they bring value to your
customer right to your end customer
um but where where is
well like i don't see how
uh security can be uh
part of this this kind of this diagram
here right
security is not there because most
organizations that say that they're
doing devops
are not focusing on security right
they're doing devops they're doing agile
they're doing sprints
but most of them are not doing security
and that's why i think it's very
important to
mention that that for my perspective
that's one of the reasons
why the the word devsecops and other
words were
were uh coined as well to to create this
awareness right to create
to raise the awareness of where is
security
right so yeah where is security
many organizations are applying devops
without thinking about security and
that's
that's critical here right the goal of
devsecops is exactly to raise the
awareness and increase the need of
security
in this shifting culture using the agile
and the automation mindset right
so it's not just tools it's not just
culture
but also automation right the way and
and for
for you to have automation most of the
times you're going to need to code you
know you're going to write
some scripts and that's why it's
important for
security engineers to know and
understand
programming language and understand how
to code
so devsecops is an evolution of devops
right
uh to make sure that security teams
stop being the bottleneck of digital
transformation
okay so okay but
what is the difference between those
right it's still maybe you're still
confused and
and it's not clear what's going on there
are some companies that
can do security uh with devops
but that's not the majority of them
devsecops means considering like
application infrastructure security
from the very start right without
slowing down the devops workflow
and and what does that mean right you
need to understand how the pipeline
works how they're shipping code
so that you can implement security
features
at each step of the pipeline and without
slowing them down right you don't want
to block them you don't want to slow
them down
and especially you you don't want to
break the build right you want to avoid
breaking the build because that's going
to create friction
with the developers so introducing
security in the beginning
right in the inception of the
development life cycle and
uh automating testing within the develop
devops workflow
so let's say the developers are shipping
code and they're
already using a ci cd uh too
like i don't know jenkins yeah i mean
it's still used today but so how are you
going to automate your
security testing right one way of doing
that
is uh applying your scripts and your
testing
on the pipeline the same pipeline that
they're running their builds right
so you apply your security testing
together
with jenkins and jenkins and other cicd
tools have many plugins
that are uh that can do this for you
right they can
trigger some static analysis testing or
dynamic analysis or even software
composition out of this testing so
that's very important right
using what you have to uh
on uh to help you automate and increase
the security on the pipeline
of course devops has and devsecops has
many different names
right it started as like a rugged devops
right
and and any change over the years you
may find
uh uh different names for devsecops or
sec devops and nigel security as well
but to me they all mean the same thing
they all mean applying security
verifications on the pipeline in a way
that doesn't affect development
but also apply gives them some guard
radios and some
guidance right like
some protection before the code goes
into production right
we all know that identifying those
issues early and often
it's cheaper right and it's easier to
fix right so why
we're not doing that yet
so one of the things that i like to
mention here is
focus on some opportunities to improve
uh security
at some key points in the life cycle
right
um and in security or devsecops right
the game changes frequently right new
vulnerabilities
and new um threats must be addressed and
resolved on an ongoing basis right
it's very hard even for us that work in
security
it's very hard to keep up with the
latest stress and latest vulnerabilities
right
imagine the developers right that's not
their main focus and we don't want
developers to be an expert in security
right
we want them to be our uh uh
helpers and always always also our
champions
right so closing those security gaps
early and in the life cycle reduces the
the operational impact the risk of
exploitation
and the cost of responding to incidents
so yeah like having fast dust and
software composition analysis
is not devsecops right that doesn't mean
that you're doing
uh uh devsecops that's great having
those tools in the pipeline
especially if you you're addressing
those vulnerabilities and the reports
and of course validating them as well to
avoid
sending noise and false positives to the
developers
but that's not enough right devsecops is
is bigger than
that and and there's different
approaches that we can do
in the even in the early stages of the
life cycle before
even there is even some code written
such as threat modeling
uh to to start
some devsecops process and implement
some requirements
on on the process on the project
lifecycle
so why not just start left
right we talk about shifting left and
and
and all that stuff like okay yeah let's
shift security left
and although that seems
uh reasonable most of the times
if you start shifting security left by
the time that you get
at the beginning of the life cycle it's
going to be too late
process is going to be already changed
uh uh uh projects have moved
on and completed right so that's not
going to work
so i heard about this approach
i think was from uh stephanie from
secure code warrior
that okay why not just start left
right and that really really uh uh made
me thinking like
hmm okay yeah that makes sense
we should start our application security
approaches
together with the development of the
application
right if if the this life cycle
starts at the planning session and
talking about features and requirements
that's when should we should start as
well so maybe
starting with like a threat modeling
starting
participating on those planning sessions
and understanding
uh features that might be security
related or that might need some security
requirements
that's even better than trying to bring
security
uh uh left from like okay from paint
testing
to maybe dust and maybe sassed and all
that stuff right
so i i really like that idea
of starting left and not just shifting
left
and i think that's that's a very
important thinking here
that maybe we need to change to make
sure that devsecops is not just
a a joke to to people and not just okay
yeah i have those tools implemented and
i'm on fine i'm doing devsecops
right so yeah that that's that's what i
would like to
uh uh to recommend and maybe provide
some
uh uh feedback here get get some
feedback actually
from from you to see how how you think
about that like
is it better to just start left instead
of shifting life
right shifting left is not that simple
it takes time and effort right
by the time you have shifted left it
could be already too late
and shifting left can require purchasing
and and maybe the deployment of new
tools which
also takes a lot of time right so think
about that
so what worked and what didn't work for
me
when i tried to apply this uh devsecops
approaches in different organizations
right what worked what worked was
partnering with different teams such as
the q a team for deploying tools in the
pipeline
so for example as i mentioned earlier
we didn't have any application security
tools in our pipeline right
and we thought about it right and okay
maybe we should deploy something
and and should at least to have a
overview of
okay what is your or what is our attack
surface right
what is the kind of the projects and the
repositories that are most
at most at risk so
we deployed a tool called solar cube
to analyze the quality and the security
of the code
right and and so some of you may say oh
but sonarcube is not
a security tool right it's more like a
code quality and
code coverage testing coverage too right
you're correct but sony cube has
different
uh plugins and different features that
can detect
security vulnerabilities as well right
and so
by by deploying star cube you have the
support of the qna team
because they're also interested in in
detecting and fixing
quality issues in the code right so that
was
something that really worked for me and
i had the support
of the q a manager to do that and and
that went really well
another thing that worked for me was
bringing
senior developers to the abstract team
right
so as i said before when i joined one of
those teams that i
worked for uh uh in an organization i
was the only application security
analyst
right so like i wasn't going to be able
to to handle everything
and and we started hiring not just
internally but also externally to build
out the
the whole security team not just upset
but but the security in general
right so one thing that worked was
some senior developers that were already
in at the company
and they were working on their own
domains and their own code bases and
projects
but they really they really like
security and they're really interesting
security right
at the beginning we started something
like like everybody knows as the
security champions
right oh they stay on their own teams
but they are the the voice of security
on that team
let me tell you the problem of that
once the developer that's part of the
team and reports to
a team lead or or like a developer
manager
starts voicing problems
on the code of their own peers
they start getting like
being pushed around and being excluded
of other activities
and i know that kind of sucks but at
least that was
that was what happened with me uh and
and with my
uh with my experience actually so
what we we changed this uh approach on
the security champions because it wasn't
working
and also because who this uh the person
who decided
um which features or which tickets the
developers were going to work on
was the project managers the product
owners sorry
so unless they really said okay you work
on this security ticket
they couldn't work on that and although
there were some vulnerabilities from
previous fantastic reported to their
team it was
always on the backlog right and last
there was some
some really uh escalations to fix those
issues
there was always they always stayed on
the backlog right
and and and and kind of being ignored
there until
until there was some escalation so what
worked for us
was bringing the developers to the
abstract team okay
now you're not just security champions
but now you are part of the security
team you are the voice
of the security team and you have the
authority to say
that what you're doing is wrong or or in
a way of course it's not like pointing
fingers but
it's in a way that they need to fix this
vulnerability right this is critical
and even they can even help with um
like specific stuff actionable uh stuff
on on how to fix that issue right
because they understand the code base
they understand the code
they can even write a pr if they want if
they are available
all the the the developer teams need to
do is
review the pr accept the vr and that's
it
right so that really worked for us so
the security champions was kind of
something that we needed we needed to
adjust in our organization and maybe
that that's similar with uh your
experience as well
another thing that worked really well
was
to bring the support of the other
developers
other analysts and qna teams we did a
lot of exercises on
city apps we presented to them what is a
ctf
right we did some red team blue team
exercises with the developer teams as
well
so that they could understand how the
attackers
uh compromised some applications and
some vulnerabilities
but also how they could protect those
issues with code
or with tools right so that was really
uh uh that was really interesting to
create the
the to raise the awareness for the
developers
as well and now here we created the
kind of the attacker's mindset that's
that's that was the goal for for this
exercise
and what didn't work right so
i talked about what worked let's talk
about the bad stuff or what didn't work
and and maybe
maybe you can uh we can discuss what
could have done
could have been done better right one of
the things was
the lack of understanding and the
support of engineering leaders
in regards to security right as i said
uh in the beginning in related to the
secure coding training
right they didn't they didn't think it
was relevant for the developers and
didn't think was necessary to to have
the developers
spend uh uh four hours on a training
right and and and that was kind of
uh really frustrating to me
um so yeah you need to have the support
of the the leaders and the managers
especially on the engineering team if
you have if you want to build a good
relationship with the developers
and have their support as well
another thing was lack of the ability to
prioritize critical security issues
right as i said the person who decided
what was going to be worked on on that
street
was the product owner right and unless
we convinced the product owner that that
vulnerability or that security feature
was important
he or she wasn't going to uh they
weren't going to do
to implement that and to prioritize that
it was always going to stay on the
backlog
right so security maybe can have a uh
a discussion with that in about that
in regards to okay who should be
responsible about prioritizing security
right one of the things that we're
trying to implement
but but we couldn't do it in time
was deciding okay
now if 30
or or x amount of percentage of your
sprints
should be related to security right
maybe 30
okay so if you have any security
vulnerabilities that are
kind of high or critical or if you have
security features that can be
implemented
let's say to uh prevent brute forcing
attacks implement
captcha or something like that then you
need to focus on those
for the 30 of your sprint right so yeah
if the sprint is two weeks right you
need a couple days there
to focus on security stuff
right but that that didn't get
implemented in time
so that didn't work and
another thing that didn't work was app
sack or application security or
infosec in the beginning wasn't involved
in
as an active participant in the plenty
of new features right
so we didn't know what the developer
teams were working on
any major products or features that they
were planning and developing
so we're kind of kept in the dark there
so we had to we had to like
kind of it was was almost like we had to
listen to our developer friends and talk
to them to try to
to try to get those informations about
what was what was happening and what was
being developed and if there was
something that could be
uh potentially critical in regards to
security like a new payment feature
or the the company accepting now
uh gift cards and stuff like that right
that could be
really affecting uh uh the security of
the whole application
right so that's that's something that
that's probably
uh um that that's really important to
have and and maybe having a security
team or a
an upset person joining
the the sprint planning sessions of
those teams or at least
being involved and being uh
being informed about those new features
that especially the critical ones
that's really important
uh i forgot to mention but one of the
things that did work for us
was working with uh
third-party uh vendors and and
yeah third-party vendors that provided
some
uh some paint testing as a service so
since we didn't have the team
and enough uh people to do pen testing
on many different applications
we had uh someone help us with
uh with those testing that that we they
would report those vulnerabilities
and what we did was based on that
reporting we would add
the the vulnerabilities as a jira ticket
into our ticketing system
and place that ticket into into the
the backlog of of the the specific
developer teams
right so that was also something that
worked
so to summarize that and and i think um
i hope that brought you some uh
interesting remarks here right um so
try to make security as much part of the
development process as writing code
so understanding the tools that they use
like i said
uh it can be jira which ide do they use
is it yes code or is it intellij right
how do they work on their own tickets on
their own bugs even if they're not
security bugs right
and then you can se follow the same
process
for for vulnerabilities or which or
security bugs as well
right make security transparent and part
of the developer
workflow right without adding more
barriers now
so don't don't force don't force
uh the developers to use security tools
don't force the developers to be to need
to access
a console off of the res to get the
results of a security testing
um that's not gonna work right that's
not going to work
so there is one uh uh and one great
quote
from uh disney denis cruz from oasp
that he he said that in the upset latin
latin america in 2011 right so 10 years
ago
and that really caught my attention
nobody likes bugs
not even the developers it makes them
look
bad also nobody likes to be
told that they're wrong especially in a
report report format right a pdf format
if you send a pdf to the developer
i'm sorry but i can't help you right
that's not going to work
so the idea here is to become the
developer's best friend
and they will help you without even
knowing right
kind of the security champions
initiative as i as i mentioned
right and also him remember what matters
to the business
and balance security with
features okay that's it for this talk i
hope you have enjoyed and i'll be
available for
questions thank you