hi i'm adam shaw and i'm here today to
talk to you about a deep dive in supply
chain vulnerabilities
and how sec devops can save the day a
little more about me
i'm the director of enterprise security
at contrast security
application security and product
security is under my purview
so i spent a lot of time thinking about
supply chain vulnerabilities
and ways to mitigate them
especially in today's threat landscape
i've also spent about a decade of making
and breaking applications so i have a
good experience kind of on both sides of
the coin
with development and security there's my
twitter handle on my website if you'd
like to follow my work
or get in touch afterwards i'm also on
the appsec village discord
under the same handle clever and wise
hit me up if you have questions about
the content i'm producing
or if you want to chat about my talk
i'll be
available after the talk as well some of
my hobbies include family time
home lab home automation i enjoy
participating in ctfs
and hackathons i also
enjoy working with the community i
helped start a
uh a local cyber security conference
in omaha nebraska called colonel con
love for i'd love to have you out uh
also i helped run a local defcon group
but let's talk about supply chain
attacks what are they
besides being the 2021 buzzword bingo
center square supply chain attacks are
a cyber security attack vector where
malicious actors
basically target less secure elements in
the supply chain
so if i read that you know it's it's
attacking the low-hanging fruit
um recently there's been quite a few
large supply chain attacks
just today jfrog analyzed pi pi software
and found eight packages
that have been downloaded over 30 000
times that contained malicious elements
also in 2021 kasaya helped ransomware
1500 businesses because of its own flaws
in software and lack of patching you may
have also heard of the code cove breach
where an attacker exploited a flaw in
the container build process
and modified part of the bash uploader
uh leaking environmental variables
secrets and more
and that affected 29 000 clients
and 2020s big one
uh solar winds it led to a massive
attack on american companies
um and the attackers basically got into
solar winds
uh changed some code maliciously hid
their tracks
and then the routine update that
solarwinds
sent to all of its clients
then added all the malicious code and
backdoors
to all of their clients it was a huge
attack at 2020. these are recent attacks
though
um when i look at like the timeline of
supply chain attacks uh one of the first
ones can be traced back to
1983 when ken thompson
the co-author of unix he created a c
compiler
that backdoored every program so these
have been
you know in the in the news and
uh infosec communities have been aware
of it since 1983.
so why are they still so prevalent today
right why is
supply chain attacks you know such a big
deal
now well it's that weakest link part
right um when we look at the difficulty
of an exploit
uh operating system attacks have always
been more difficult to exploit than
somewhere along the line in the supply
chain but
it isn't until modern day times where
supply chain attacks have actually
started to really
build out its impact level
today's software you know takes an
enormous amount of
dependencies i think that something like
98 99
of software has dependencies of some
sort
and now um supply chain attacks
have the ability to have a large impact
on communities
so um yeah that large impact
there's dependencies in the code uh mpm
yarn maven pi pi nuget uh all kinds of
things
and there's microservices in the
architecture
and then there's vendors in the
environment and then hardware and
everything
and you didn't write any of this
uh that uh trusting trust
uh paper that ken thompson wrote um
in there he says you can't trust
anything unless you write it yourself
but in modern software
the amount of stuff that you write
yourself is so little
compared to what you're pulling from
other people that
it's pretty much impossible you know you
didn't write your vendors you didn't you
didn't write the
hardware you didn't write all your
dependencies
but talking about this it makes you
sound like oh my gosh
you can't find pepe silva right
well this scope is just way too big to
discuss all of this
so we need to we need to narrow our
scope a little bit we need to
bring it down so today's talk
we're going to focus on javascript
attacks um
many projects use javascript and build
dependencies
there's a wide range of an experience of
experience and use
so i hope that a lot of people viewing
this talk
can relate uh also um most of the
defensive strategies that we discuss
can be used in other supply chain um
mitigations so i'm hoping that this is
uh ubiquitous enough that people can
take
pieces of this and apply it to other
different
uh software or supply chain levels
and then lastly one of the reasons why i
want to focus on js attacks
was in may of this year the node.js
vulnerability reporting program
shut down and they closed all of their
existing reports
with an ask that researchers reopened
their findings
through a private company and uh
nothing against the private company but
um
a lot of researchers aren't going to
double dip their the work
or they've moved on and so i worry that
there's
you know hanging node vulnerabilities
that just never got
uh never got triaged or any attention
because they were in the
backlog of the nodejs vulnerability
reporting program
and now they're uh now they're
you know closed so
uh let's talk about the problem with the
javascript supply chain though
this is a fact modern software contains
only about 22
custom code uh that's modern javascript
uh applications so 78 of it comes from
open source dependencies or link
link or language libraries which is an
incredible amount when you think about
it
over almost eighty percent of your code
you didn't write
and there's a problem with that um
it's almost impossible to audit all of
the code and dependencies
uh managing vulnerabilities in
dependencies is
is cumbersome and nearly impossible
right um
and updating dependencies means
continuously bringing in new code
that hasn't been evaluated yet yet so
when you have code that
when you have code that goes in a
dependency that asks for an update
you have to bring in code that you
haven't actually
like looked at and so it's risky to just
update even in other words
when you look at a dependency here this
is an engine i o dependency this brings
in
18 other dependencies so it's it's
basically
a human pyramid right it's it's as
secure or as safe as its weakest link
i mean look at that that guy in the blue
hoodie he's definitely falling
and that pyramid is gonna collapse
so let's examine some case studies and
learn from mistakes
that can help us inform solutions these
are some of the most
famous famous cases supply chain attack
attacks in in javascript applications
uh well not this one uh this is the this
is
uh popular it's a typo squatting
vulnerability
um i uh i made this for a ctf
two years ago i cloned a react js
color picker and then i added
some code around it warning people about
type of squatting vulnerabilities
and i even have a flag
i discuss different types of
supply chain vulnerabilities and yet it
still gets
eight weekly downloads to this day
people are still downloading this
because they um
you know they spelled color with the
british way and they forgot an e
and on the word picker and people still
end up downloading
this color picker and even though
in the read me i talk about typo
squatting
and i say this is not the package you
think it is
and people are still downloading it
which just blows my mind
that's that's basically what type of
squatting is is uh
somebody goes and they downloads a
package
that they didn't intend to
uh let's talk about a malicious update
this is a
this is um a very famous vulnerability
that i thought
was insane
with the amount of thought that went
into it by the attacker
event event stream was a very popular
npm package
which exposed a lot of node helpers for
working within streams
it had 1.9 million weekly downloads
there's two characters in this story
there's dominic
uh and right nine control or the bad
actor
uh dominic is a 10x engineer he's
uh you know he's built out 600 or
something
npm repos and
packages and uh he has
you know a crazy level of um
community help that he's built out so
uh dominic created event stream
uh in 2011 and it had been kind of
in maintenance mode for two years before
september
18th um right nine control
made a few commits on github basically
adding unnecessary stuff to the repo
uh he then emailed dominic and said
can i have permission to publish from
this repo to npm
uh dominic went ahead and granted
permission
um then in the middle of september 2018
he made a few more commits he actually
like deleted some of his previous
commits it was
uh if you were actually watching these
commits it was kind of obvious that he
was just
um doing whatever he wanted and it
didn't
make any real impact on the code uh
and then he emailed dominic and he asked
for ownership of the github repo
and the mpm package now dominic
wrote this in 2011 this is seven years
later
and dominic had already moved off of it
you know he had been two years where he
hadn't made a commit
but this package was still getting 1.9
million weekly downloads
so dominic went ahead and transferred
the ownership he doesn't
get anything from it he hasn't used it
for years
um so he went ahead and transferred the
ability to publish
to the npm repo to the bad actor
he also attempted to move the entire
github repo
to the attacker this is where the
attacker
he really big brained it he uh he
actually
forked dominic's repo so dominic
couldn't transfer the github repository
to right nine control because uh he had
a repo with the same name so he couldn't
transfer it
over to him and so dominic still
maintained
the repo under under his user
um right nine control had ownership of
the repo
um as did dominic but uh
if people went and looked at it it still
looks like it's under the same trusty
trusty person who's you know always
maintained it
because right nine control basically
blocked the transfer intentionally
uh then september 9th uh right nine
control
that's this is where he kind of went
nuts he adds the dependency flatmap
stream
into the github repository and he
published it to the npm package so
uh flatmap stream was a
a dependency of event stream and event
stream
was a dependency of a lot of major
software at the time
so that's why it had such high volume
so now when you look at this dependency
map um
you know flat map stream is now the one
of the
um dependencies of event stream and
flat map stream turns out actually had
vulnerable
malicious code in it and this code would
look for
um certain crypto wallets on your
file system and if you had said crypto
wallets
then it would try to break them and
send uh your cryptocurrency and your
keys
uh back to an attacker back to a server
which the attacker controlled
so right now he adds this vulnerability
and he uh he adds the malicious code and
he publishes it
then a week later he actually removes
the dependency
and implements it manually which you
think okay well maybe he figured out it
was malicious
here's where here's where the difference
matters he publishes it to mpm as
a major major version number
so by updating the major version
right um he effectively locks anybody
who uses a carrot
or a tilde to um describe their version
number
in their package.json into the
vulnerable version
but if they went and evaluated the
current version it would look
legit it would look like it was um
it would look like it was good
uh code without any without any
malicious code
um because he implemented flatmap stream
manually
and removed the dependency which had the
had the malicious stuff in it
so all the people who
who all the people who um download the
latest version
um were fine but all the people who had
been using it for quite some time
are basically locked into a vulnerable
version so it's another way that he's
obfuscating
this ability to detect this
vulnerability
so yeah like i said if you go check out
the latest code on npm
the crypto crypto mining code isn't
there
uh if you check out the latest code on
github the crypto mining code is in
there
also github still points to your
reliable dominic
who you know right there is where he
talks about how he tried to transfer it
but github aired because he already had
a fork on it
so dominic had tried
already to to send this to another
person
so his tracks were covered he was mining
wallets for
two months before any discovery on uh
any application that used event stream
uh the discovery is kind of interesting
it was discovered because the attacker
used a deprecated library
which is hilarious to me because he's
basically attacking the supply chain
and he was using a deprecated library
which popped a warning
and some dev uh saw the deprecation
warning and tracked down and thought
what is
producing this warning and he figured
out that it was this
flat map stream and
then and then he thought what's flat map
stream doing
and that's where he noticed that some
strange activity
that turned out was the uh crypto mining
and discovery took a huge effort uh
you had to get github to remove the
commit you'd have mpm remove the package
in each organization to to ban the user
um and uh and then npm even archived
event stream
and uh prevented any future updates from
it
they rolled it back before flatmap
stream
and they prevented any future updates
from ever being committed to it
so this brought a lot of attention to
process right
uh a lot of people didn't know that
what's published in mpm doesn't have to
match github
uh you know that in any way so what's on
mpm
could be fake uh what's on github
could be totally different a lot of
people
you know thought well how how could you
transfer the ownership
to this person uh when i
use this and the honest answer was
that dominic hadn't used it for two
years
and a interested party came and
knocking and offered to do some free
development work
so it's things like these that really
you know
lay out kind of the difficulty around
uh around protecting against supply
chain attacks
so then recently
more recently you might have seen this
one this is a uh
a really interesting attack vector
alex burson published an article about
dependency confusion
where he and another researcher noticed
that paypal had non-public dependencies
uh most likely internal called out in
their package.json
so basically these dependencies in red
right here uh they were internal
dependencies
um but if the
um so the build server when it was
looking for these dependencies it would
go to the public npm
and if it failed it would then look
internally
so they set up their build server
incorrectly it was first looking
publicly
before internally and when it failed it
would go to an
internal dependency but if it
if it didn't fail it would download from
this public
npm repo so they went ahead and
registered
those names on the public npm repo
and they were now able to get
um get into the build server itself
and with with node um
uh with uh node scripts you can actually
use something called pre and post
install
so you can actually run uh
run your own code on this build server
just by them pointing at one of your
packages
which is crazy and the amount of effort
that
the amount of effort it took is very
minimal right
it's registering something with it
with that name uh my co-worker matt
austin
uh we read this article and we were
talking about it and
he went and um pulled uh
microsoft teams
and looked at the uh and he looked at
their
um their electron apps dependencies and
he went and he registered
uh some npm dependencies and he was able
to get hits
from microsoft's build servers which is
is crazy to me that this is still super
um exposed and super vulnerable uh and
really a problem
so let's talk about some sec devops
solutions here
how can we install guardrails to help us
mitigate some of these vulnerabilities
uh first sec devops let's acknowledge
that
we talked about the 2021 buzzword bingo
square
sec devops is probably the 2020 or 2019
buzzword bingo center square
what is it well it's the process of
integrating security into the
development
and deployment workflows uh basically
security processes
and checks um enter the lifecycle at a
very early stage
uh to keep up with the rapid devops
release approach
one of the things that i always preach
is that we want to be
partners not hindrances in apsec and so
uh partnering up with devops and trying
to
engage in this rapid release approach
really helps solidify that partnership
between apsec and development
so some of the things that um
you know your organization can do uh
ahead of time and really just as part of
the process here
is squat your own enterprise uh that
sounds funny but
um honestly it's it's it's really easy
to do
and it's uh it's probably going to save
you
on a lot of type of squatting type
vulnerabilities
one of the things you do is you purchase
all potential typo
domains for less than 10 a year you can
buy 20 or 30 domains and don't forget
combinations
of alternate uh tlds like
dot com.net.org you want to grab all
those
dot gov is another popular one uh
these are these are things that you want
to go and grab and own
right and um this isn't part of the
supply chain but you probably also want
to set up
email servers and have those email
servers
you know points to your internal
security team you don't want people
accidentally
sending out billing information
to a wrong domain you want to know about
that
treat it as a teachable moment
don't don't just register the domains
though redirect them
point them to the main domain to help
alleviate confusion
uh you don't have to pay to do that that
can be done without hosting
uh again this is a super cheap option
um just a guard rail that you can put in
place
and then register namespaces on public
dependency managers
uh this is important um because uh
uh if you do that you can help prevent
some of that dependency confusion stuff
that we talked about
um it'll if you register the names in
similar spellings
remember typo squatting uh of internal
libraries and packages
on all public repositories uh even the
reserve name
even if they're never used it basically
prevents an attacker from tricking a
developer
and into using that an identically or
similarly named package on a public
repository
uh speaking of um you want to activate
name spaced modules
many package managers including maven
and mpm support
namespaced modules which prevent the
same name from being used
by two different resources another good
way
to protect your enterprise
and then lastly treat your list
of internal packages as sensitive
information the success of these attacks
depend on knowing the name
of internal research resources and so
organizations can't be cavalier about
that they need to protect it
and you can't just push all of your
internal
uh internal packages into a
production client owned you know package
that's easily uh decipherable um
so this is the kind of stuff that you
want to keep um keep
as private as possible uh
some ways that you can guard against
militia malicious updates
use a lock file lock files ensure that
you get the same package version
on every install so if you're secure
today you'll be secure tomorrow
this would have really helped with that
event stream vulnerability i talked
about
again it was created in 2011
and in 2015 it really entered
maintenance mode
and if you had been locked in to
that maintenance mode version when uh
when flatmap stream was added at 336
version 336 you would not have installed
it
right so you would have never you would
have never had to even install it
so um that would have really uh
saved you know save the day in that
scenario
uh something else that i ask uh is that
um
developers think before they install
dependencies right um
many organizations have limited
visibility into the sources of their
software
which is a problem when you're a big
organization
or when you're an organization doing a
lot of m a lot of mergers and
acquisitions
knowing kind of the source of your
software is
vital you need to understand the
dependencies
and track them at a very granular level
and this can be difficult but
um if you don't start now then you could
regret it
it could be you know in two weeks or two
years
but um the best time to start thinking
about this and about where your software
comes from
is now and not tomorrow
and you can put in organizational
policies and checklists
for your devs um just
as a good way to uh to
help guard against some of the uh the
problems with
uh supply chain attacks some of the
checklists would be
um do you need to install this
dependency
is the package well maintained do it you
know
do they have like actual issues that get
closed
do they have open security issues uh
does the package match the package
manager source right like
are they publishing the same thing that
shows on their
public repository uh can you write it
yourself
um this is a big one i'm not sure if
many of you
have heard of the term or the the
program left pad
but a developer another huge community
developer
had you know 140 packages on npm
and uh mpm decided to give
one of his packages names
to a company that wasn't even going to
use it
because they were the company and it was
a trademark thing
and he uh he had created the package i
think before the company was even named
and um he was mad and he
actually removed all of his all of his
uh modules that he built well one of
them was called left
pad and all left pad did was nine lines
of code
and i think it just added spaces to the
left of something
uh to the left of something that
converted it from white space to spaces
or something like that but it was it was
nine lines of code
and it probably didn't even need to be
nine lines and um
but he had made it at some point um and
uh somebody used it uh at some point
it got some spread and eventually um
babel the main transpiler of javascript
of modern javascript applications
uh had left pat as a dependency so by
removing his
uh repos he effectively killed babel
and uh and every modern javascript
application i'm talking ember
angular react uh you know all of the uh
common uh javascript applications at the
time
they all like just broke they could no
longer
take um the framework and convert it to
vanilla javascript
for use and it um it was huge it
affected all of the javascript world and
community
and it was just because this developer
got mad and he removed
what essentially amounted to nine lines
of code uh
that somebody when working on babel
could have just
made natively and it would have been uh
never an issue so i always tell people
about it
this when i'm when i get when i educate
them with this checklist because
that's something that uh you know
easily could have been written
themselves but by
inheriting a dependency you've inherited
another risk
right you've opened yourself up to more
risk by having this dependency
and um another option is uh if
you're a little unsure about this
dependency
is you can evaluate it as is
and then fork it or clone it and then
just point to that directory so
if you're not sure it's getting proper
updates and you like it
as is uh you can host it yourself
and just point to that instead of the
public one and now now
now you have complete control over that
code that code is now your code that
you're
using and you're not taking any updates
um the one of the reasons why this is um
this is there this is a good solution if
you're willing to also
you know check the updates that it does
get
because if there is a security update
you want to you want to make sure that
you uh
add that to your side um and so
there's guardrails that you can set up
here where you're hosting it yourself
but you're evaluating every update
and then pushing it to what you've you
hosted yourself
that's another option uh another
thing that you want to do um you know
part of the cia triad
is you want to protect the integrity of
these dependencies
uh one option is you can use a sub
resource
integrity with cdns basically
directing the browser to only pull a
specific resource
if the hash of the file matches the
cryptographic hash that you've specified
so what you're doing there is you're
saying uh
i expect it to be this and
if it isn't the browser is not you know
the
the project won't pull that that
resource in right which is
which is what you want you you if this
is different than what you said it was
you don't want to pull it in because
it's been modified and it's potentially
been modified
maliciously uh
also you want to continuously monitor
your dependencies
you can use tools like mpm audit
retire.js dependency check
and dependabot these are good ways to
continuously monitor
your dependencies but take but there's a
stipulation that goes along with this
make sure you understand the results
right um there was a great article
recently
um i forget who it was but uh where they
talked about how
and the mpm audit results are very
misleading
um because several of the results uh
involved dev dependencies basically
build dependencies that never made it to
production
so when i'm deploying an app that uses
javascript
and i'm using some packages to build
stuff it doesn't matter if the
package that's that's building something
has a cross-site scripting vulnerability
in it because that's on my build server
and all i'm using is i'm all i'm using
it for is to generate another package
and deploy it and if it doesn't get
deployed with it
it doesn't matter to me so you need to
understand these results and
really like really read through them and
understand exactly what you're
um what they mean but
you can always uh train people to do
that
or you can um you know add some sort of
rules or run books around how to
determine whether or not
a finding is a finding you need to be
concerned about
or if it's a finding that you you can
kind of put on the back burner
so that's just the stipulation with that
one
uh always uh prefer signed packages uh
from official sources
um because it reduces the chance of
having
modified or malicious content uh and
then
something else uh protect your
deployment accounts right because
uh when you're pushing stuff uh to
mpm or um whatever you could be
uh you could be someone in a supply
chain right like think of solar winds
they they were in the supply chain of
all of the companies that got attacked
by the
by by their update really that by the
malicious code put into their update
and so protect anything that you're
deploying as well
um that means two-factor rotating
service keys
uh also use like role-based
authentication on who can push artifacts
you know really narrow the scope of who
has the ability to do that
and uh put that into a trusted group
and then make sure that there's um you
know a process in place for rotating
service keys
and then if somebody leaves the
organization make sure that there's a
way to
uh rotate their keys or
you know delete them from the group
properly so that they still don't have
the ability to push something
out into a package manager which would
be a big
all right problem those are some of the
ways
you know that that this matter um that
you can protect against it
but honestly like we talked about how
this has been around since 1983.
why is it still such a problem
and what can we do about that well
it's still pretty ongoing because of two
big reasons i think
uh there's a lack of awareness right um
without
appsec educating the product and the
engineering teams about the dangers
of supply chain risks it can be
difficult to get the priority it needs
right and so i think that one of the
things that you can do as appsec
engineers as
as internal appsec teams it's really
build out that education make people
understand
hey this is super risky and it's gonna
it could have a large effect on
us and here's what we need to do to
safeguard it and if we don't do it
that's that could be a problem
and make them understand i mean uh
without that education
uh some people are unaware that just
pulling in gobs and gobs of dependencies
isn't
the best idea you know a lot of people
think oh i can code so much faster
if i pull in these 30 dependencies and
you know i'm pulling in things that'll
do arithmetic for me
or generate html or something you know
silly
uh that's not a great thing
right so there's a there's a
cost analysis that goes with that i mean
what's the time it would take somebody
to implement something themselves
um but you know they probably can do it
and you need to start factoring in risk
with your cost analysis
and that brings me to my next point uh
developers have extremely tight
deadlines right
we live in this uh we live in this
community in this world where
software nowadays has tight turnaround
times
um tech debt is growing uh
developers um you know are getting
pressure from product
and uh public stakeholders uh
you know you've got your investors and
some public companies have shareholders
and they have to produce results and
it's just there's
a lot of pressure and so as absec
team members right we need to have um
we need to be good team members and one
of the ways to do that
is to fight to get them the time
to code securely and app second
development should be
should be like a cooperation not a
competition
so um you need to be good partners
on good partners in your company and um
that's something that um i think really
like
is a key component with some of the
supply chain risk
because uh if you're a good partner and
you're helping
um them uh get the time they need to
code something more securely or code
something themselves instead of bringing
in a dependency
um it's a win-win scenario right they
don't want to produce bad code
everyone wants to code securely and uh
you want the company and the product to
be secure
so remember that you know that's that's
that's one of the big takeaways your
team
and here are the guardrails that you
want to implement with your team
here's some of my sources thank you all
for your attention
like i said i'll be i'll be on discord
if you have any questions
hit me up you can also follow me on
twitter
and give me a message there if you want
to
touch base i really appreciate it thanks
for the opportunity to apsec village
i know that going virtually with uh two
weeks left
or three weeks left is uh extremely
difficult
so um thank you all to the organizers as
well
and uh i look forward uh to
you know getting back to normal and
hopefully being back next year in person
thank you
you