hello hello hello and welcome to free as
in beer
creating a low-cost static analysis
program
you can grab the slides for this
presentation afterwards at this url
so take a screenshot or a photo if you
would like
to do that
so who am i i'm aaron browning
i'm a senior security engineer at slack
in security for
over six years now you can contact me at
erin at frowning.wtf or
on twitter at e frowning yes that's upon
my last name
hi my name is tim ferracci i'm a staff
security engineer at slack
you can hit me up at faraji tim
gmail.com
or linkedin tim ferracci
so i mentioned that tim and i both work
at slack
and we're hiring we hire remote
tim and i both have a lot of experience
implementing static analysis programs
personally i've implemented a static
analysis program for
1200 developers across multiple
languages
for a bunch of different compliance
standards
a little bit about my background i've
run
a lot of static application security
testing programs
implementing them from the ground up or
going into a company
looking at it and then making
modifications or replacing
the current implementation i work in pci
hipaa and federance ramp compliance
space
i have experience in commercial and open
source solutions
for static application security testing
i've seen almost embedded out almost
every static application security
testing
software for in-house
cloud mobile and
more i want to tell you about some of
the problems with static application
security testing with vendors
one they're slow most of the vendors
take 30 minutes an hour or more
worst case scenario i've seen it's 48
hours
if you love comcast data caps you're
going to love scanning per gigabyte with
vendors in their cost model
or their cost model per app or their
cost model for defining what a developer
is
on average about 200 000 dollars
yearly sales negotiations enough said
mystery secret scanning sauce do you
know exactly what
complete transparency what is being
scanned and how it's being scanned
no i would love to see that and i think
everyone should it's a tenant of
security
transparency let me tell you about a
better way
static analysis is a 15 year old solved
problem that anyone can do
and you know i want to show you some of
the things that i'm looking for
in a good solution one open source
easier implementation
now integration is hard if it's open
source
it's easy we control it we can do
whatever we want with it
dev wants scan results fast
scan results should be 3 minutes or less
if it is any slower it makes devs
sad they want to push code
fast lots of language support product
that we use
has tons of language support nine at the
time it's writing and slack created
additional language support that's
coming out
and we want to leverage power the open
source community we are smarter
as a community than any single vendor
i want to find vulnerabilities now
the vulnerabilities that i've seen from
commercial and open source are very
very similar in fact they're so similar
it's really hard to tell the difference
between the two if you did a scan result
side by side
that's my years of experience
and then you want to get that compliance
check box
the dirty little secret is any security
scanner will get you that compliance
checkbox
but i prefer a good security scanner
which is why
we think you should check out semgrep at
the time of this presentation
semgrip has taken no money from us nor
have we given them any money
or editorial rights to this presentation
we just think it's a cool program
that you should check out this is why we
chose to create our own static analysis
program
around this tool
let's get right into it in order to
create
an easy to scale static analysis program
we need the following things first of
all
we need a static analysis engine
that can support multiple languages
second of all we need the ability to
not only define our own rules but
to finely tune them
and third we need to be able to build up
our own
infrastructure around these scanning
tools
after analyzing a variety of closed
and open source tools we decided to
implement our program around an open
source tool called
semgrep it is a generic engine
which supports multiple languages
exactly like we wanted
the other cool thing about semgrep
besides
it being language agnostic is that
you can write your own rules for it
fairly trivially
the rules are yaml files they're easy to
write
and also easy to modify there's a whole
bunch of open source ones you can
download that are based on
common vulnerabilities
i've written several now and
it takes me longer to write the
developer guidance than the actual rule
which is incredible you can also get
really down
into the nitty-gritty of
pattern matching so you can exclude
things for example that
let's say it's a command execution for
command execution
you can say okay if it's taking a static
string we're going to say that's fine
if it's taking a variable let's worry
about that
and you can actually express that in the
semgrep rule sets which
so many fewer false positives so many
fewer false positives
so here's a quick example of how the
rules look
you can see there's a pattern area
developer guidance
and you can include
a whole bunch of other information if
you need it for your
statistics or reporting
the company that makes semgrep is r2c
and they have been so helpful as we've
been implementing this program
their tool has really met the
flexibility requirements we wanted out
of this program
so we have some grip we have this
generic engine
how can we use that to scale up our
static analysis program
well i think everybody has seen this
diagram
right when you find a bug in code
the later in the development process you
find it
the more expensive it is to fix we
applied this thinking to the
implementation of our static analysis
program
we want to push a lot of the
heavy decisions around
the design and implementation
and rule set as early into the process
as we could
so that we wouldn't have to make
expensive changes
further down the line we also wanted to
prevent
having to deal with false positives
and other similar issues again and again
and again
here is the nitty-gritty of our design
first of all we need easy to maintain
architecture
for the infrastructure surrounding
semgrep
we decided to write python we have
python code that runs
the docker container we have python code
for the configuration for each language
we have python code for
generating hash ids for performing
comparisons everything's in python
and here is how we add new code bases
with one line of code we have an enabled
file
for each language we currently support
this enabled file is newline separated
repo names
so if i wanted to add a new repo for
going
i would go to the enabled file for
goatling
type in the repo name save it commit it
we're now scanning that repo
we also wanted a graceful way of
handling false positives
that is low maintenance and allows
developers to report false positives to
us
instead of a database we chose to use a
json file
for super easy maintenance we don't have
to deal with database backups
we just commit the json file straight to
our project repo
and if a developer finds a false
positive or something they think is a
false positive
they can make a pull request adding it
to this file
when we onboard a new code base we also
add an empty false positive file we do
this
to increase the anti-fragility of
our infrastructure around semgrep
otherwise
our code may still try to run even if
it's not removing false positives
which we don't want so that file must
exist
we label each finding with a unique
identifier
we create those identifiers they are sha
256 hashes
which contain the following information
the file path where the vulnerability
was found
three lines of code before and one line
after
of the line the finding is on
the rule name that was triggered
and then we we hash all that and that
makes our identifiers
we don't use the line number it's too
fragile
if somebody adds new code to a file the
line number will change
and then they have to go through the
false positive process all over again
we don't add that into our hash instead
we put that in the message associated
with the hash in the false positive file
file path can also be a little fragile
if somebody's going to re-architecture
their application or their code base
files will move around
however we needed a way to differentiate
findings where the code may be really
similar
so we included the file path what we're
planning to do
for pull requests in the future is
analyze them for
files that have been moved for files
that have been moved
we can then regenerate the false
positive id
and add it automatically to the false
positive list
this doesn't happen very often but it is
an edge case we plan to support
one of the benefits of using identifiers
like this
is we can do comparisons we can compare
any two result files
in any way imaginable this is our actual
comparison code it's super
straightforward
so currently we perform a daily
comparison
we take yesterday's scan result compare
it to today's scan result
and alert if there is a new finding
then we also do branch comparisons which
run on our pr
system we run different rule sets
between
the daily scan and the pr branch based
scan
in the daily scan we run more intensive
rules rules we're testing
[Music]
rules that still need to be tuned
because
we don't want the scans to take too long
for the developers
the pr scans are intended for direct
feedback to the developers on really
dangerous functions
tim will now run you through the pr scan
and the processes we've developed around
them
the pull request scan is a bit different
than the daily scan
the pr scan hooks directly into our ci
cd infrastructure
in github what happens is when
development triggers
a pull request a jenkins job
is triggered which downloads semgrip
downloads slide specific semirep rules
and downloads the repo
the repo is then scanned with these
rules and a result
is re return back to github in form of a
test
pass or failure these tests
are fast and they're meant to be fast
they're meant to run
the most important security rules
we have we are targeting three minutes
for our results
if they take longer than that then we
try to either adjust the rules or make
our infrastructure faster
it depends now
how these rules work in the pull request
scan is
a diff is done between the scan results
of the last master scan
and the pull request if new
vulnerabilities are found
the test is failed if not it passes
so these results end up in two different
locations in our cicd infrastructure
one is in directly in the jenkins
console
it's json it's not very fun to read not
very fun formatted you gotta copy paste
it in order to format it correctly
but it's there and it's fast to click on
and view
however just as fast and easy to view
is the html view it's the exact same
results
but in a nice digestible human readable
format
what's in this vulnerability output well
the vulnerability
the hash the time of the scan run you
know
the rule that triggered some education
lots of education slack specific
education general best practices
education
suggestions and how to remediate
and the goal of this all of these
vulnerability outputs if there are
vulnerabilities is to give
steps something to do that is actionable
that you
can work with either it is
hey this is how i fix this or this is
who i reach out to
in order to ask questions to get help
with this
also on this vulnerability output we
have a
very simple command line interface that
is on every
developer's laptop so if there is a
vulnerability
and development says oh this isn't
actually vulnerable or
this is a legitimate use then
they can go ahead and trigger a
pull request for security to review
security will review that pull request
and go hmm is this a legitimate
use case of the exact function
perhaps well it's an internal script
and it's only running on the box and
user controlled
yeah they should probably be able to run
exact because
their use case is fine what's nice about
this
when the scan results occur is they
produce
alerts these alerts can integrate
directly into slack
so if you have a development feature
and some alerts come out you can throw
those alerts directly
into the channel so people can take a
look at it immediately
security can also hook into specific
alerts we want to see
with specific channels that are perhaps
very sensitive infrastructure we want to
know what's going on
all of the alerting all the
vulnerability output
all of this is captured to bring us to
metrics
now we have initially 55
repos that we have scanned at slack of
those 55 repos we've had 648
vulnerabilities
now depending on who you are you may go
that's a ton of vulnerabilities wow i
can't believe you had so many
vulnerabilities
or oh man that static analysis really
didn't find very much
you know ain't doing a very good job
now in my experience
running through this it's doing a really
good job
we're finding similar true positive
numbers
running our open source sub grip
uh rules as i am
comparing it to commercial offerings
of those 648 vulnerabilities 12 of them
are actually true positives
of low to high that's not very many
vulnerabilities per repo
it is really really low
uh let's see here it's about one
vulnerability per every four repos
so one of the repos which
was in development for a very long time
it's a much older repo
at the company and we've been a company
for about what
seven eight years this repo has it had
eight vulnerabilities eight
out of the eight plus eight years it's
been in development or
or whatever well working with
development
you know trying to vet these they're
going oh man that's a lot of
vulnerabilities
or like no not really i mean given what
we see here
is for your cicd
development process you would only be
impacted potentially once a year in
order to review a vulnerability
i take this as part because slack has
good developers we have very good
developers here
the other metric that we have here is
the average scan time
we've had of all of our repos has been
3.2 minutes
now some of them are slower we have some
that are particularly big and they're
skewing it up
but that's a very good scan time very
very very good
again we do want to get this slower
sub three minutes we might even have
some stuff in place
that we can even half this time
depending on how we architect our
scanning
but we're not there yet and hopefully
one day we will be
a little bit about the soundgrap rules
the same grip rules
this is an example of a go laying
some grip rule in particular looking for
the exact command pattern
the semcrep rules are actually really
really easy to implement and create
it's a very symbolistic pattern it's a
better version
of regex it's just a very simple pattern
it's called pattern syntax
alright so in these particular examples
there is an ellipsis operator which is
the dot dot dot
and that's looking for any uh
anything that is thrown into that
function
so we're looking for exact demand
and we're looking for it doesn't matter
what
anytime anyone tries to throw anything
into exact command
we want to be alerted on it we talked
about this before we have
developer education this is important
um i know at companies we do this
uh stereotypical hey you're onboarded
we're gonna teach you developer
education we're gonna have a yearly
interaction what the loss top ten is
uh i mean it's useful right but
i learned the most when it's in the
moment
we really try to make sure that all of
the documentation
at slack is actionable has education
explains to development what this is and
honestly we don't know which type of
person we're going to get
that's going to be submitting submitting
code that could have a vulnerability
it could be a very senior person who's
just like oh yeah oops forgot about this
i know all about this i'm going to go
ahead and fix it
to people who are very green and they
would
you know love to like look into this and
read about it and it's really helpful
for
them in their career
so another great thing is because we own
the infrastructure
we can make it faster we can we have
some stuff that we're
thinking about here paralyzing
the rules downloading multiple different
multiple copies of the code to be
scanned but
also running on different boxes with
different flavors of the rules
so you know scan results that
would take maybe an hour it's going to
take you know
minutes um those three minute
times we saw earlier um either running
on beefier box
or paralyze them uh just
just some stuff that we're thinking
about doing in the future but just
haven't gotten to
some grip supports 21 different
programming languages
however at slack our main code bases are
written in hackling
some grip doesn't currently support
hackling
so you may be asking okay so some grep
supports all these different languages
but it doesn't support the language i
want it to
well you can add languages to samgrep
but
how difficult is that well
an intern can do it for two interns
who are almost done with their computer
science degrees
these are our wonderful interns this
summer david frankel and nicholas lynn
who have been working tirelessly to
implement
hackling into semgrap
they're both very cool people and we've
been
very lucky to have them
nicholas came up with this project title
project sushi aka static analysis using
semgrep
with hack integration and he made the
r2c logo
a little bit more delicious first it
helps to understand
a bit about how static analysis works
all languages have grammar english has
grammar chinese is grammar programming
languages have grammar
you can actually write up formal
specifications for
what the grammar is for your programming
language
and then you can take existing code in
the language
run it through the grammar and create a
representation called
a syntax tree there are two main types
of syntax trees there's
concretes and tax trees which are direct
representations of grammar
in tree form and there's abstract syntax
trees
which are simplified representations of
code
the concrete syntax tree will have so
much information in it
like everything the computer needs to
run the program
for our purposes we don't necessarily
care about all of that extra information
we just want to be able to understand
the semantic meaning of the language
an abstract syntax tree takes that
complicated
concrete syntax tree and reduces it to
a useful representation of the code the
teal
is the same code as the orange
it's just been reduced to a more useful
format for analysis
how do you get from a grammar to
a syntax tree well you have to parse the
code
just like if you were to read a book you
have to
teach the computer to read the language
you want your syntax tree to be strongly
typed because you want
to be able to say okay in this
expression
the grammar here is this is a variable
this is a method
this is an argument you you want to be
able to know
what each piece of the language means so
strongly typing them
is a huge advantage for when you want to
get more semantic meaning
out of your programming language
we'll do a deeper dive later so once you
have a grammar
you need a parser the parser will read
the source code
and turn it into a concrete syntax tree
and then when you want to go from a
concrete syntax tree to a more
simplified
abstract syntax tree you need another
parser
we've been using tree sitter for grammar
tree center is a wonderful project by
github it allows you to specify grammar
in javascript and
r2c has a bunch of ocamel
that takes the grammar rules
from tree sitter and
gives you a generated parser that you
then have to go through
and teach the grammar so we have to
modify
the o camera parser file to produce the
correct inputs and outputs
based on a generic grammar
because semgrep is language agnostic
when we have a cst and we're going to an
ast we need to use constructs that
semgrep can understand
so some grip will know what a variable
is
but you have to tell it how the variable
is specified is it prepended with int
or is it implied like in python
the other big initiative is once you
have the hackling parsers
well
you can't do anything with them if you
don't have rules so
we've been working on creating a rule
set
for hackling that looks for
common problems we've seen from our bug
bounty program
from internal reports from our own
experience
of course like olaf's top ten you can't
forget
the wasp and as i was mentioning earlier
these rules are actually fairly
straightforward to write
it usually takes us longer to write the
developer guidance
let's put this all together we have an
enabled file
when you get a new code base that you
need to scan you plop it in the enabled
file
there is then we create an empty json
file
for false positives
then the scan runs we
review the results mark down false
positives
look for true positives
bam you're now scanning that code base
even if you're still dealing with
processing the results of the scan
you're now scanning which means
you're in compliance
thanks to the snow team
static analysis non-web app as we've
been calling the project
r2c for all of their help with creating
the parsers
our wonderful summer interns nicholas
and david
of course slack because without slack we
wouldn't have
gotten to work on this program and the
apsec village for having us
the sun turtle appreciates you and
finally
thank you to our lovely co-worker
antonio who
wrote a good chunk of the initial
grammar for hackling
and has guided both our team
and our interns in implementing hacc
this summer
if you want further reading you can hit
up these slides
and also check out some crap we've been
really happy with it
thanks everyone and we appreciate you
taking
the time to come to our talk let us know
if you have any questions in the discord
and we're happy to chat with you
bye