what up hackers we doing some sci-fi
stuff today
thanks for having me i'm glad technology
has brought us together and for those of
you who are in person at the con i hope
if you see me you say hi
if you're developing software you likely
have a complex dependency graph there's
simply no way to do what we do without
the work that has come before us
so how do we deal with the swell of low
quality insecure code that is just
waiting to be plucked in your project i
say embrace the crazy world of unchecked
unsecure code and help it grow
let's talk about why and how
okay
let's level set we all stand on the
shoulders of giants
whether or not you're developing
software
you are using software that is either
open source or has open source libraries
in it
and some businesses market themselves as
defenders against they defend you
against open source
so it's really hard though to imagine
where we'd be without the sharing of
code
and of that knowledge
and in fact some of your employees
whether you know it or not may be
contributing to open source on their own
time
and
as a business it may not seem obvious at
first but this is definitely a win
and we can talk a little bit about how
you can bolster value in this
relationship that some of your employees
may have with open source
we'll talk a little bit more later about
bolstering that relationship
and
leveraging the value there
but for right now we'll talk about
security because that's why we're all
here right
in the past infrastruct professionals
have had to
act like mad dogs to get traction with
securing important shared computer
resources in some ways this is carried
over into
modern day application security through
a dogmatic approach to software
development
this has left some newer security
professionals with a myomic myopic take
on software development
are you
affected by this myopic
attitude
let's test it out if i was to say
not all code needs to be secure
what comes to mind do you instantly
think ant wrong or we can't even
consider that to be the case or don't
tell developers that for heaven's sake
if you thought
something like this or
anything in the realm of this this talk
is for you
i would like to talk about things that
can be overlooked
we'll talk about how vendors prey on
this dogmatic approach
and
we'll uh see how it's to the detriment
of you and them
also you may be
leaving free knowledge on the table if
you have this attitude so
let's get grounded
let me be clear if the internet can
touch an instance of it it better be
secure i'm not trying to be bombastic or
create click bait
by saying not all code needs to be
secure i'm not saying that all codes
should be insecure that's simply not the
case
what this statement is
meaning is that third-party code may or
may not fit your application threat
model
by assuming all software should be
secure you shift your ownership of some
of your some portion of your threat
model to an unknown party
and that unknown party you will probably
blame later when things go wrong let's
be honest
so who am i that i'm saying all this
stuff why should you listen to me how do
i know what i'm talking about
well
you may already recognize me as being
unleashed
remember this one good times
i
professionally worked my way into
network engineering from frontline phone
support role at an isp in the late 90s i
moved into software engineering after i
found out i enjoyed working on software
and later i made the steps to actually
get better at software engineering i've
been a professional developer for nearly
20 years now
and i've always been a a
community wallflower
in the last couple years i have been
trying to go legit as some can attest
this is difficult
also
hire me i specialize
in a particular platform
and you can find out more on my linkedin
profile
tell me you saw this
over the years i have been involved in
various developer ecosystems
i've watched some ecosystems mature
and then fracture and some of them die
some i watched
just totally fail i was involved when
some large enterprises responded to the
epiphany that open source works
and then the awkward efforts that they
made to make it work for them
so i've seen
good decisions i've seen bad decisions
and
i've witnessed breaches
supply chain attacks have recently been
the talk of the town
are you tired of hearing about it
but it is important
you should probably check out some of
the other talks given at conferences
around
and i think there was one just earlier
in our very own apsec village
supply chain attacks might be simply a
suspect chip
on a circuit board or more likely a
compromised build server injecting
malware or it could simply be a private
package name on a public package
supply chain attacks aren't new
and you may remember the lifepad debacle
just a couple years ago or maybe some of
you remember some older examples
some vendors blame it on open source
but if you look closely the most
successful were aimed at proprietary
software and used exploits of
proprietary software
that is not to say that open source
wasn't involved or that it's not a risk
in some ways there are definitely risks
on both sides
do you understand though what your
software supply chain is
and what all is involved
where does your software
come come from that you use in your
supply chain
do you know all of your transitive
dependencies
those are dependencies of dependencies
do you have an inventory of all the
component components of your dev ops
tooling that's part of it too
because
software supply chain is a complex topic
and we won't fully be able to explore it
here today
simply reviewing the names of what might
be involved in a kubernetes cluster
would take a whole conference of its own
so
the point is
that to get a handle on software supply
chain in general we need to look beyond
the primary libraries that make up our
apps
because it involves
everything that is involved in getting
our application our code
to production
but right now though
i'd like to talk about some roles
and relationships that can give you a
grip
on software supply chain
just like not all dependencies are open
source
not all supply chain issues are security
only
the first place we should look is at
getting a hook into the developer
mindshare
by understanding a little bit of how
they relate to security specifically
whether or not a developer knows the
term they develop under a certain
assumed threat model to a developer
threat is a term that is broader than
security and extends to other
undesirable
input output and computational side
effects
generally what infosec refers to as
security is a subset of quality criteria
to a developer
unfortunately some developers see
application security as obnoxious or
paid sometimes hostile qa people
if this is the case enabling some
understanding of the developers
perspective of a threat model can go a
long way to bridging the purpose of the
two perspectives that is the developer
and security
because educational systems often
neglect security
a developer may have a threat model
that has not been
thoroughly acquainted with
some aspects of security concerns
early on it may be mostly influenced by
issues they discovered in the wild
so
how does a developer threat model
develop
that could be a key to understanding
the many aspects of your software supply
chain
for some developing started with
curiosity
how does this thing work
how can i make this thing do that thing
a person will call them alice finds out
that they can use software to make
things happen then
they figure out that they can do other
things by writing software to satisfy
their curiosity
then alice finds another person who is
experimenting
with the same things
that they were
we will call
them bob
and so
they shared code in hopes of
growing and learning from each other's
perspective
and alice might tell bob
don't do such and such because it lets
out the blue smoke monster
alice learn through experience
and
experimenting and bob returns the
gesture and tells alice i did this that
you said and it didn't work in my case
you need to change something different
this
learning exchange
may or or may not include security
the information being exchanged
is just not considered to be explicitly
security or quality at this stage
the code at this stage is not anything
you'd want to put into production
however
someone might come along and do just
that
we'll call him elliot
he sees some functionality
and maybe he wants to be the first to
put it into a product
so
he plucks it and puts it into
his software running on the internet
and can we guess what happens
he gets hacked
who is he going to blame when it's
clearly his fault for not paying
attention to the code that he picked up
like
chewed bubby bubble gum from a sidewalk
if he complains
and blames them publicly this bad
experience may discourage alice from
doing any more shared learning
at best
it's a burden
that they didn't ask for
at worst this threat model could get
personal
and turn into a deal breaker
sending them
looking for a different hobby
and that's
obviously something that we don't want
we need more
programmers
however elliott sees his mistake he can
responsibly disclose how to enhance the
code and then alice and bob
and anyone else joining in later
can see this and learn from it and
create better code for elliott to use
coincidentally
this doesn't need to be so formal
but i hope you can draw the lines from
this circumstance or simple example to
more formal circumstances
and
perhaps draw ways that you can encourage
developers
to increase their quality and security
in the things that they do share in
order for them to share them
in an elevated
context
okay
i know you've been dying to talk about
this
let's talk about that insecure code
for a minute
let's look at code
in this way it can be likened to
clothing in the past i've been known for
going to lots of concerts one of the
things people like to do at concerts is
dress up sometimes they dress up fancy
sometimes they just want to look nice
and appropriate and sometimes they
experiment with more wild clothing
sometimes they just want to get
attention
you have probably seen some pictures
from burning man or corcella in your
social feeds yeah we're talking about
those people some people use these
places as a safe place to experiment
and maybe develop their own style
they can get to know what they are
capable of pulling off what looks good
on them
however
if you're going to climb mount everest
you should not pick from the outfits
that they wear at coachella because they
do not match the thrift model of your
journey
if you expect a summit wearing some
little sparkly fanny pack around your
neck in a swimsuit
you're probably not going to have
success you're probably not taking it
seriously enough
in the same way
you would not choose code that is not
fit for your purpose
if your purpose is a production
application
you would not choose coachella
however
if you want to show the effects of
prolonged exposure in sub-zero
temperatures
you could
take that outfit and start up mount and
everest
this would be like into
much of what's done here at defcon
you see a lot of this seems like a bad
idea let's see what really happens
this is really the only way
that
bad quality or insecure code should be
permissible in teaching
that is to demonstrate how and why you
don't do some things
one of my padded irritations is when a
person teaching
coding says
don't do this in real life and proceeds
to explain a real life scenario with
some code that
is sub quality and insecure
the only time the example code should be
incorrect or insecure is to show exactly
that
insecurity or how it's not correct
example code should always be secure
it's just like the old
adage if uh the old illustration if you
want to be able to identify counterfeit
forgeries you study the real thing if
you study with a forgery your model will
be inherently flawed
and us technical people in this age of
machine learning understand this
if you want to create the hot dog not
hot dog app
you feed your model hot dogs
not everything else
there are some people who love to
experiment
they don't want to code in their day job
all they want to do is express
themselves and
learn something in a stress-free setting
they have no intention of taking that
next step to production
or to offer their software as a service
innovation takes many forms and comes
from many places
sometimes these experiments innovate in
ways that impact privacy or security
but this shouldn't prevent
the experiment from happening
how much better when this sort of thing
thing happens this innovation out in the
open where anyone can learn from it and
if need be they can learn how to
mitigate it
nearly anyone can place packages on a
public feed i should know i have a few
out there
as security professionals we need to
recognize that the fundamental purpose
of public feeds
is to make experimentation and creation
easier and faster
i've not found any myself but i would be
interested in hearing about mechanisms
built into package feeds to be able to
differentiate experiments from
production-worthy code
many
feeds offer version suffixes
and they've found some adoption but
if you're not intending on releasing
your code you're not going to go through
the trouble of putting
beta on it either
[Music]
right now public feeds contain a gambit
of quality and security
the uninitiated developers are likely to
grab any old package from their feed
that makes their job easier
if the feed is blocked in some misguided
attempt to control the flow of outside
code into the application developers
will turn to their favorite community
site blog or even twitter if they're
desperate
each platform and framework may have
multiple feeds and sources you can't
block them all
so how do we manage even if we come to
grips with the fact that insecure code
needs to exist
what can we do to keep our own software
supply chain up to our standards of
quality and security
so let's review some of the things that
everyone is doing
or should be doing
the fundamentals
update often automate scans run them
often code review pen test work to
remediate quickly
maybe an internal package feed is a good
way to stay on top
of your dependencies
i've seen a process implemented around
the process of getting new packages into
the internal feed the package being
approved by a qualified developer then
all builds are forced to use only the
approved feed and anything not approved
breaks the build and usually prevents
the merge
it seemed to work out pretty well
one place i worked at
someone thought that it was a great idea
to package name camp on the public feed
buy me a beer later and we can discuss
all the ways that that can go wrong
another place i worked for bid their
developers from any outside code
and this created an issue with learning
and problem solving and in just regular
flexibility it was really a pain to
block all the outside code and so you
know eventually it failed sometimes you
just need to look at the documentation
what about forking a project to preserve
its state this is no different than up
never upgrading a package the risk is
high that you'll just get stuck with
flaws and vulnerabilities don't do it
if you can get your static analysis
security testing software composition
analysis
and everything from the same vendor some
people feel that having it all in one
package
is all that they need to cover all their
bases
and
we've talked about some of the ways that
that's not the case something that's
often overlooked though when you get
things from a vendor is and it's rarely
reported on is how the vendor treats
open source if they are doing scanning
and they find a flaw are they
responsibly disclosing or are they
secretly passing on a patch to their
customers and never giving back to that
project this by the way may be in
violation of some open source licenses i
don't know i'm not a lawyer but it's
just kind of a jerk thing to do
when a library gets a flaw are you more
likely to abandon it instead of
responsibly disclosing the flaw
we might be able to work with
communities and
develop a rapport and fast turnaround in
order to satisfy some of those needs
according to github's 2020 state of the
octoverse vulnerabilities often go more
than four years before they're disclosed
so maybe if we disclose them more often
ourselves we'd be doing the responsible
thing and we could
improve that
statistic not every issue needs or gets
a cve sometimes the only place you can
know of a security issue is in the issue
check tracker on the project itself
with open source you can watch these
although it might become quite a chore
to keep up on all those packages how
many packages do you have in your feed
again
so
do you ever feel like the kaiju just
know your name and are coming for you
sometimes trusting just seems way more
simple so maybe how can we do some of
that
phil hacked a former github employee
outlined four questions to ask to
establish authorship in a post entitled
trust and nuget back in 2013
it asked
who is the author
is the author trust trustworthy
do i trust that
this software really was written by the
author
is the author's means of distributing
software tamper resistant and verifiable
phil has written several times about
package management security i suggest
you take a look at his blog even
outlines some suggested steps to change
maintainers
such as consider a change of ownership a
simver breaking change on popular
packages upper offer reputation
information about the new owner
work with open source foundations to
find and vet maintainers
don't leave the burden on the maintainer
or they might just delete it and walk
away
you may think that just paying for a
package allows you high
level of trust
the expectation really is the difference
with a proprietary feed because of this
i would still start with the same level
invest of investigative trust as a
public feed it would be good to
determine
what would prompt you to revisit this
and plan to repeat it down the road
something that has been suggested by
vendors is rating systems for packages
however one very important thing is left
out by those suggesting such a system
no one asked to be rated not every piece
of code is created for altruistic
motives or even to be consumed on
enterprise products
i'm not sure if you noticed but people
really don't like to be labeled
so here are some guidelines if you want
to start a rating system
a rating system must be opt-in
if you start slapping labels on
unsuspecting code you are telling them
that they have no choice but to maintain
your standards
ratings must be simple i shouldn't have
to do math or look it up on a matrix to
make sense out of it
be specific about flaws flaws and
vulnerabilities and link off to owasp
and mitre descriptions
each site flaws shouldn't
cited flaw should be given a specific
platform standard well-documented
mitigation
also mediation should allow for
explanation and manual approval and the
reason behind that is just because you
think it's wrong doesn't mean it wasn't
done on purpose and false positives
happen
the positive side of open source is that
the source can be examined but how often
is this really done
if projects were given the opportunity
to
opt in it would also give us a place to
start doing secure code reviews
and testing them for quality and
security
[Music]
at the end of the day there are really
only two groups of code
if the maintainer is able to put the
effort in and the project is able to be
publicly approved
proved to be adequately verified we
could indicate that it is being
maintained as an adequately verified
product avp also by the way my favorite
comic book franchise
this way is it's made explicitly
separate from the coachella without any
action required on the part of the
experimenters
offering a community a way of
differentiating their product may
enhance the motivation to provide higher
quality and security developers may want
to go the extra mile to be recognized
feed providers can enable
trust building by passing this
information on in a simple way
so that perhaps an offering perhaps
offering an avp specific feed even
companies can get involved by offering
resources for carrying this out they can
offer compute resources as well as
reviewers perhaps earmarking time of
their developers to devote to the
community
do your appsec personnel
get trained to do secure code reviews
there's no doubt this is a good skill
for them to have
however
i've heard from several that do it that
reviewing all of the code a couple of
times every a year just doesn't really
scale
and then if you think about
a situation
that's like if you
were in some place that you didn't know
the laws and you're being stalked by law
enforcement that kept arresting you
without explaining why
the simple solution to reduce your
violations would be to tell you what the
laws were
in the same way
giving developers the chance to have the
knowledge that they need makes sense
you can make policies and standards easy
to access even better make them easy to
quote in merge request comments
contrary to many invosex stereotypes
developers like to learn
make sure they know where they can find
best practices offer to help with
clearing the way to implementing
processes to enhance quality and
security and let them own it
during one plane trip i sat beside
someone that was criticizing the pilot
tell me if i'm wrong but they if they
want to have safe pilots they train the
pilot not the passengers
so while it seems harsh if you're not a
developer you're not writing code you
are a passenger the highest impact on
quality from training is going to be the
obvious choice of the developers
when they are trained offer to help them
to put the procedures in place to make
them successful with their new knowledge
have you thought about identifying key
people in the open source community
perhaps that you could
identify for sponsorship
you might be amazed by the result
when developers learn they often get
excited they might return to the group
start using what they
learned and share it with others
but just be careful not to douse their
fire with your grilling
it has never been a better time for a
company to get involved in their open
source communities
it would be great to see communities
shift their attitudes about open source
to large-scale commun communal
ownerships
managing proprietary third-party
dependencies is different than open
source
it is obvious we need to scrutinize our
code dependencies whether proprietary or
open source
the key takeaway is to train our pilots
how to stay away from coachella
managing our software supply chain is
just as much about inventory as it is in
participating in its open source
communities
take the time to understand where your
developers came from and where they're
coming from and enable their ownership
of quality and security
and
embrace the crazy because you can't stop
the code thank you
you