hi there
i'm cheryl viswas you may know me as
encrypted online
i am honored to present at absec village
welcome to my talk today signed sealed
delivered
abusing trust in software supply chain
attacks i hope you enjoy
a little bit about me i'm a threat intel
specialist
with a major bank here in canada shout
out to my awesome team
i'm also a founding member of the diana
initiative
where we support diversity and inclusion
in our field
much love to the terrific staff
volunteers and supporters who helped
make another
amazing event happen this year i
sincerely hope we can all
meet together next year
and now for the obligatory disclaimer
the views expressed here today
are mine mine alone and not those of my
employer
okay let's get started
in this talk i want to explain what
software supply chain attacks
are and the growing threat they pose
we'll look at how code dependency is a
factor
compounded by mistakes and
misconfigurations
then we'll look at a brief history of
some major attacks
and the state-sponsored actors involved
to talk about who's been watching and
what they leverage
finally we'll wrap up with what we could
do better
and some suggestions
we don't fully appreciate just how
interconnected we are
and how that translates into software
code dependencies
it took an event like the solarwinds
orion attack
to rattle the bars on that cage and wake
us up
to what's been going on for some time
the reality
is that software supply chain attacks
are not something new
they've been around for many years and
we've been watching that check engine
light
but not really addressing the issues
my hope is that the knowledge i share
here today
gives you some new insight and awareness
so that you can go back
and better secure your projects
what a year even before we got out of
the
gate the tone was set solarwinds was
truly unprecedented it's the kind of
event that we
talk about for decades we build off of
it and we learn from it
however as you're going to see there
have
been a lot of software supply chain
shenanigans just over 2021.
these types of attacks are an abusive
trust
their compromise right at the source
the insertion of malicious code into
legitimate software
and then distributed on mass
to trusting customers this can create
access points into the networks of those
customers
up to the tens of thousands as was
demonstrated by solarwinds orion
and as we've learned nobody is
untouchable
everyone everyone is a potential target
software supply chain attacks take time
resources
and skill to plan this is the purview of
state-sponsored threat actors notably
russian or chinese-backed groups but
both cyber criminal
and state-sponsored groups target the
technology industry
because these companies are relied on by
many organizations and individuals and
that can have wide-ranging
impact attacks on tech companies can
enable
third-party compromise of enterprise
customers
via updates through software supply
chains
for example china has actively targeted
tech companies in taiwan with these
types of attacks
because they see them as major
competition
now another important disclaimer i am
not a coder
or a developer and i'm not going to
pretend to be one
i am presenting this talk from my
experience as a threat
intel researcher and what i do plays a
key role because
we watch the geopolitics play out and we
reach back for
historical context on patterns of
behavior
because history repeats and actions
have consequences ian pratt
the global head of security for personal
systems at hp had this to say
whether they are a direct target or a
stepping stone to gain access to bigger
targets
as we have seen with the upstream supply
chain attack
against solarwinds organizations of all
sizes
need to be cognizant of this risk
there's nothing like getting some
historical context
so let's start with this in 2009
operation aurora was a wake-up call when
chinese state-sponsored group apt
17 targeted google adobe
and other tech companies for their
source code management systems
to alter the source code and this is at
the heart
of a software supply chain attack if you
control the source code
well imagine what you could do
this event led google to implement
zero trust to track lateral movement
and to put in place better
infrastructure
interestingly enough google was not part
of the mass
fallout during solarwinds orion
and then in 2017 there was nopetja
an elite russian hacking team known as
sandworm
part of the gru military intelligence
service
compromised and took over software
updates
of a mandatory accounting package known
as
medoc now this software was used
throughout the ukraine
at tax time and they didn't have the
plethora of choices that that we enjoy
it was pretty much that or bust
russia has used ukraine as a cyber petri
dish
for years to test weaponized cyber
weapons in what
has been a rather hostile relationship
russia has control issues
so sand worm used midoc to
distribute a destructive malware known
as not pecha
which was supposed to be a punitive
measure against ukraine for daring to
leave the mothership
but instead that spread to infect
major companies like maersk as
unintended consequences
and the costs amounted to more than 10
billion
dollars globally in january of 2019
a sophisticated supply chain attack was
discovered targeting
the asus live update utility
now this is a pre-installed utility on
most asus computers
to automatically update certain
components like the bios
uefi drivers and applications
it's very convenient something that end
users would
implicitly trust and there are a lot of
asus end users
in fact according to gartner in 2017
asus was the world's fifth largest pc
vendor
that would make it an extremely
attractive target
for apts advanced persistent threat
groups
who might want to take advantage of that
user base
in this case that apt was chinese threat
group
apt 17 also known as barium who happened
to have been behind
a number of software supply chain
attacks
nick weaver a security researcher at uc
berkeley
international computer science institute
shares this
you're trusting every vendor whose code
is on your machine
and you're trusting every vendor's
vendor
we're grappling with the growing sense
of complexity in tech
it's not just what corporations deploy
in their own environments
but how they are incorporating
third-party tech
according to windows snyder the product
assumes
all the security risk of all the
components that it incorporates
so let's have a talk about our code
dependency
sorry
the fact is that applications
increasingly depend on
external software to work proprietary
code
open source components third-party apis
modern applications are just too big for
one developer to do
on their own and software reuse has
become the norm
popular open source projects are used as
dependencies
and these are attractive targets for an
attacker
to add malicious code to and then claim
the users of those dependencies
we need to appreciate the increasing
complexity
of library use in applications how the
functions of one library
rely on a number of other libraries in
order to be implemented
this creates complex dependency trees
that developers won't remove or update
because of the possible
downstream consequences
so let's take a look at some numbers
the average app has 118 open source
libraries
the average library is 2.6 years old
the average java app has 50 open source
vulnerabilities
99 of organizations have one or more
high-risk java licenses
and what are the false positivity rates
using legacy
sca tools
java has 23.net 13
and node 69
per github security researcher maya
kazurowski
85 to 97 of enterprise
software code bases come from open
source components
with the average project having up to
203 dependencies
and given how sophisticated those
attacks can get
any projects that don't incorporate even
basic protections like code signing
put themselves at real risk because
we're using other people's technology
it's a trust issue
let's take a look at the four elements
of a software supply chain
there's what you write custom code
that's developed in-house
there's what you build with software
development tools
there's what you buy off the shelf
software as a service apps
and there's what you use third-party
libraries
so we need to understand areas of risk
around open source libraries and
frameworks which can come from
active or inactive libraries or library
classes
the age of the library licensing
and of course vulnerabilities but that's
hard to do when you
don't have visibility into the libraries
and classes being used by the software
or when you're only relying on static
scans from
legacy security code analysis tools that
go out of date as soon as that library
changes or updates
this is from the 2021 contrast labs
open source security report there's a
lot here to take in
but the key points are these one
if you're using open source code you
need to be paying
more attention than you have been what
we're doing
and what we've been doing to identify
and remediate risks
it's not comprehensive enough we're
missing stuff
and the legacy security tools in use
aren't securing us we've got to level up
this is a lifetime a timeline a lifetime
no it's a timeline of software package
repositories
that have been in supply chain attacks
it's really easy to just
get code from various projects online
and then incorporate that into
other software here's the risk
some of those open source projects are
widely used
but not well maintained or they've even
been abandoned
code reeves is a deliberate strategy
it's seen as a key differentiator for
efficiency
innovation speed and application
development
but at the cost of being vulnerable to
compromised
off-the-shelf components now some
repurposed code comes from
internal repos but a lot of it comes
from open source libraries
third parties for an attacker
compromising a software supply chain can
be through
manipulation of the application source
code
manipulating the updater distribution
methods
or by replacing compiled releases with a
modified version targets can range from
either
a very specific and limited group to a
very wide
number
stuff happens
and that brings us here the continuous
integration
continuous development pipeline which is
a best practice
for devops to deliver code to change
frequently
and reliably ci is a coding philosophy
it's a set of practices to drive
development teams to
implement small changes and then check
and code
to version control repositories
frequently
it's a consistent automated way to build
package and then test applications
it's good except you're using
other people's technology and
mistakes will be made
in november of 2020 the fbi issued a
flash alert
warning that threat actors had been
actively targeting
vulnerable sonar cube instances since
april of that year
to access source code repos and
exfiltrate proprietary
code from us government agencies
and private organizations the fbi found
multiple potential computer intrusions
that corresponded with leaks
associated with the sonar cube
configuration
vulnerabilities that had been under
exploit since april
in august of that year unknown threat
actors leaked
internal data from two organizations
through a public
lifecycle repository tool they scanned
online for exposure via a default port
9000 a publicly accessible ip address
and default admin creds
several of those sources had those
as hard-coded credentials which opened
the door
to accessing other resources and
expanding the breach
per sakhar sarakai who wrote in the
mcafee blog from july 31st of 2020
regarding the sonar cube source code
leak
because mistakes will be made
so let's talk about several major
software supply chain attacks from the
past
and about lessons that maybe went
unlearned
if that looks busy it is
the identity theft resource center or
the itrc
stated that 137 organizations
reported they were impacted by supply
chain attacks
at 27 different third-party vendors
that's an increase of 42 percent from
the last quarter of 2020.
attackers will manipulate software
dependencies and development tools
to compromise data or systems before
those reach the recipient
and as we've seen they go after source
code
in the case of solarwinds microsoft said
that attackers
gained access to some of the source code
for exchange
azure and intune having the source code
gives adversaries the advantage
because then they can go look for
undiscovered vulnerabilities
and be the first to exploit them the
attackers also go after
certificates stolen code signing
certificates let them
evade detection and deliver malware
payloads as though those come from
a legitimate source and certificate
abuse
is an ongoing component in all kinds of
attacks
and let's talk about attacks
there is vendor compromise that's the
most sophisticated
there's exploiting third party
applications so
finding and exploiting vulnerabilities
especially zero days
these bugs can enable an attacker to
manipulate the software to do things
that it was
never intended for including
credential theft or accessing
information
there's exploiting open source libraries
now these can also have vulnerabilities
to exploit
but unlike software packages
it's harder to update and maintain open
source code
and because that source code is openly
available
attackers then can easily study it and
practice on it
for example the apache struts 2
framework
that had 13 critical zero day
disclosures
just over 5 years
there's dependency confusion we're going
to talk about this a little later on
and then there's hostile takeover
an example is from 2018 known as event
stream
and involving node.js
when the original owner was approached
by another community member who was
willing to take over the management of
his project
well they handed it over in good faith
they transferred all that control
in a few weeks malicious code was being
downloaded
via automated build tools and then
deployed in
numerous software applications this code
was designed to steal
credentials from copay bitcoin wallets
it used encryption to hide its
functionality
and the stolen credential data was sent
to a remote server
fortunately some mistakes caught the
attention of managers of the node.js
repo
and they thought that there was a bug in
the package and they looked further
so this is a good lesson about the
abusive trust
the assumption that the software supply
chain is trustworthy
and that anyone can create or contribute
with no
formal vetting process
this is a good depiction of how an
attack uses
dependency trees i'll just let you take
a look at it and
not get into the weeds
this is from something known as the
backstabbers knife collection
and the link is there
solar winds december 14 2020
a lot of us remember where we were that
day
what we were doing and what we wound up
doing
solarwinds orion is trusted network
management and monitoring software
used by the government fortune 500s
microsoft fireeye and nobody knew
anything was wrong it's been described
as
the most significant attack of our time
what's distinctive about it is the
degree of stealth
and the ability the total ability
of the attackers to conceal their
actions they had wicked good opsec
and the length of time it took for
discovery
this was an operation that took time and
precision to do the reconnaissance
sophistication to tailor it and patience
the attackers targeted and compromised
the software build environment
and code signing infrastructure for
orion they modified the source code to
add a back door
and they signed it they leveraged the
existing
software release management system and
then they used
stolen certificates to laterally move
through
chains of trust
dependency confusion as promised
this is a very interesting
attack security researcher alex berson
shot a bright light on a scary
possibility
he took a hypothesis about a supply
chain substitution attack
where a software installer script is
tricked into pulling a malicious code
file from a public repository
instead of getting that intended file
with the same name
from a trusted internal repository
then he went on to target apple
microsoft
tesla and 32 other companies so that he
could execute
unauthorized code inside of their
networks to prove this worked
it did and within 48 hours of
publication of this
a lot of other parties thought it was a
good idea too
so without permission they targeted some
more companies in march of this year
and those included microsoft amazon
slack
lyft zillow and more here's the issue
existing dependency scanners typically
don't detect if a dependency executes
malicious code
since these tools are limited to
identifying dependencies with known
vulnerabilities
now this prompted a study by researchers
at red hunt labs and what did they find
93 repositories out of the top 1000
github organizations
are using a package that does not exist
on a public package index
this can be claimed by an attacker to
cause a supply chain attack
169 repositories were found to be
installing dependencies
from a host that is not reachable over
the internet
126 repositories were
installing packages owned by a github or
get lab user
that doesn't exist and of the top 1000
organizations scanned
212 had at least one dependency
confusion related
misconfiguration in their code base now
this matters because
according to the researchers much of our
open source ecosystem depends on
these particular companies and we know
that these repositories
have a lot of users so
if any of the projects get affected
there is a high probability
that potentially millions of users could
be at risk
xcode spy we know that attackers
are targeting developers they're
targeting those shared sites
the repositories where code is uploaded
for use by others
in march of this year a new malware
variant was observed
targeting ios developers in a supply
chain
attack to install a back door on
developers computers
this is a malicious project and it's
taking
advantage in abusing this free
application
development environment it's abusing a
common behavior by developers
to share their projects online with
other users
and the threat actors behind this abuse
that norm
and they used a legitimate development
environment created by apple
to fool their victims into adding an
online project
to their app that would in effect
compromise their systems in
a supply chain attack
it's an abusive trust
codecov this is an online platform
used for hosting code testing reports
and statistics
it provides developers with tools to
help them quantify
how much source code gets executed
during testing
now they serve over 29 000 customers
globally
and many of those are enterprise level
they include
companies like godaddy atlassian royal
bank of canada
procter and gamble but the impact went
beyond them
to thousands of public development
projects things like
ansible kubernetes pi test
and victims included twilio
rapid seven and the e-commerce platform
mercari on april first of this year
kodkov reported that a supply chain
attack
had actually occurred back in late
january
what happened was this attackers
leveraged an error in the process that
creates kodkov's
docker image so this allowed for them to
extract credentials which protect the
modification of the bash uploader script
that's a tool that's used by customers
to send their code coverage reports
to the platform there's a lot of stuff
included
in those code coverage reports now the
script was modified
to deliver details from the customer
environments to a server that was
outside of codecov and then attackers
could export credentials tokens
or keys that passed through codecov's
continuous integration environment
and then they could use those to access
services data stores
or even application code
what were you doing on the july long
weekend
yeah a whole lot of us were doing this
kaseya is an i.t solutions developer
for managed services providers and
enterprise level organizations
over 40 000 reportedly use at least one
cassaya product
they offer something known as vsa which
is a unified remote
monitoring and management tool for
networks and endpoints
yeah it sounds familiar doesn't it which
happens to have a high level of trust on
customer devices
so much so that attached clients will do
whatever they are told
and therefore that's likely one of the
reasons that casey was targeted
so on july 2nd rival ransomware
operators delivered a
fake vsa update sending a ransomware
payload from
compromised vsa servers to vsa agents
that were running on
managed windows devices let's
talk trust issues initial compromise
through a trusted channel that further
exploited the trust
in that vsa agent code
initial access came from exploited
zero-day vulnerabilities
in cassaya's vsa software and it
leveraged those against
a number of msps the attack was
triggered
via an authentication bypass
vulnerability
in the casea vsa web interface
and then this allowed for the attackers
to circumvent authentication controls
they got an authenticated session they
used that to upload
upload their malicious payload and then
they executed commands via
sql injection achieving code execution
in the process
the malicious update was dubbed the
cassette vsa agent hotfix
so trust issues well that hotfix was
already good to go thanks to
anti-malware software
exclusions that got set up for cassaya
at the time of installation
and then the attackers attempted to
disable ms defender
using a signed certificate shown here
the current tally is about 50 casseya
direct customers
and anywhere from 800 to 1500
small to medium-sized businesses along
that chain
who got infected some of these weren't
even cassay's customers
the thing is from an impact standpoint
these are smaller companies
they don't have the financial or
personnel resources
that enterprises do to deal with
a major ransomware attack
so let's talk now about who's maximizing
these opportunities
of all the state-sponsored adversaries
chinese cyber espionage groups have been
and will continue to be i think the
biggest threat to tech
because they conduct economic espionage
and intellectual property theft
technology companies as we know are rich
targets just on their own
but these groups leverage them to infect
supply chains
and go after those customers they are
definitely the leaders of the pack in
terms of
the number of attacks and capabilities
now this chart is from
the report broken trust that was issued
earlier this year by the atlantic
council
and in it they reported 36 other cases
of
intruders successfully targeting
software updates from about 138 recorded
supply chain attacks and vulnerability
disclosures
of these 36 15 had similar access
to build or update the infrastructure
and half of those 15
could be attributed to nation states
these are some of the most well-known
supply chain attacks involving
chinese state-sponsored threat actors
while russian adversaries are formidable
oh yeah
solarwinds orion you really don't want
chinese apts in your network they play
the long game better than anybody else
they're masters of stealth and they are
ruthlessly dedicated
to eliminating the competition
we'll talk about ccleaner from 2017.
this one has been linked to apt-17 we
mentioned them earlier
and more specifically a subgroup known
as axiom
again historically engaged in supply
chain attacks
the sea cleaner attack showcases
technical knowledge
preparation and patience the timing was
strategically
advantageous because the original owner
piriform was in the process of selling
to avast
anybody who's ever sold the car or more
important
let's say you're selling your house you
know what goes on
there's a lot of distraction and
confusion
so it would be easy to miss things in
the network
that didn't belong there the attackers
took their time
to move laterally in the network during
off hours
to avoid detection and within a month
they had installed a
modified version of shadowpad backdoor
malware
to escalate their privileges and then
they distributed a cryptographically
signed version
of the modified ccleaner product
no one suspected anything
so let's take a little walk through that
a text step by step
for initial compromise they accessed
unattended workstations
of a ccleaner developer
just this one actually which was
connected to the piriform network
via teamviewer teamviewer for the win
remote they've reused credentials
we've talked about this many times
haven't we from previous database
breaches
and they used those to access that teen
viewer account
they delivered malware then using
vbscript
and they went on to develop their
malicious version of the ccleaner
product
using rdp they opened a back door on a
second
unattended but connected computer there
they dropped a binary and a malicious
payload of second stage malware
and this was delivered to 40 ccleaner
users
and then they compiled a customized
version of the shadow pad
backdoor to allow for further malicious
downloads
and data theft in preparation for the
third stage
where they installed their third stage
payload
that malicious version had multi-stage
malware
to steal data and send it back to the
command and control
as for action on objectives well all
that data
probably went into future espionage
projects
in 2018 apt 17
well sorry 2017 apt-17
also known as barium delivered shadow
pad malware
to in fact enterprise networks via a
product known as
netsurang now netsurance specializes in
server management and security
connectivity software
that was used globally by major
organizations
we've heard about that stuff before it
seems to be a popular choice
a module hidden in a code library was
making
suspicious dns requests and sending data
back to the operators who then decided
if they
wanted to go ahead with it if they did
they would send a malware payload with a
full back door
to take over the system
we talked about shadowhammer earlier
that was the asus live update attack
2020 able desktop
so this involved the chinese apt group
lucky mouse
who went after chat software used by the
mongolian government
agencies as well as some private
businesses
they hijacked updates in the software
supply chain
now the attackers didn't need to steal
or forge an updated signature
because able's updates were not signed
in 2020 there was operation sign site
this targeted the vietnamese
government's certificate authority
so this was a targeted software supply
chain intrusion
that could affect a wide range of both
public and private entities
who used the digital signature software
which provided
certificates of validation and software
suites
to handle digital document signatures
that software is widely used throughout
vietnam
it's even mandated in some cases
key point here is abusing
and targeting something against
a service oligopoly
it's not clear who exactly among the
chinese state-sponsored groups was
behind this
but they're possibly the group known as
ta-428
who happen to have a track record of
targeting
east asian countries like mongolia and
vietnam
for these supply chain attacks
now this is a very interesting attack
with
similarities to napeta
it just lacks the destructive malware
component thankfully
golden spy malware was embedded in
required tax
payment software that's issued to
corporations who would like to do
business
in china and the chinese banks require
businesses to install something known as
intelligent tax in order to pay the
local taxes
this is produced by the golden tax
department of the isino corporation yep
it sounds familiar doesn't it
the malware installed a back door on
systems
which enabled a remote thread actor then
to execute windows commands or
upload and execute any binary
that could include ransomware or trojans
and the malware provided system level
privileges
with the capability of executing any
software
on any system where it's installed and
interestingly enough
just because you removed the tax
software you did not remove
the golden spy malware back door it
connected
to a cnc that was distinct from the tax
software
so what have we learned
well there is a new executive order
which is a great starting point
but as you've probably seen this picture
is far bigger than we realize
there's a whole lot of dependencies
what can we do
well package management is an ongoing
challenge
that we can do better a registry of
record will help us to
improve reliability in that chain
code signing is important
following the practices of making text
secure by default
using something that we're talking a lot
more about known as a software bill of
materials
eric sheehan of semantic who happen to
be integral in helping investigate and
decode stuxnet
likens this to finding a needle in a
haystack
having the right security telemetry and
visibility
at the right control points in your
organization
so here are some takeaways
we need to be paying more attention to
the open source code that we use
it's time to change what we've been
doing to identify and remediate risks
because we're missing things
those legacy security tools in use
they're not securing us
here's something new i'll share with you
and this because
it's time to level up
sigstor is from the linux foundation and
it seeks to empower software developers
to
securely sign software artifacts things
like
release files container images binaries
bill of material manifests etc the
signing materials get stored in a tamper
resistant public log of
all activity for accountability of
every submission and it's free for use
by all developers and software providers
and you can see the link there
package hunter is a new tool from get
lab that you can use to detect malicious
code
and also behavior that you're not
expecting
in your dependencies because as we know
there could be a lot of malicious
packages in the wild
this is free and it's open source
so i will end with this i love podcasts
and one of my go-to's is risky business
they had an excellent segment about
operation aurora
with mark rogers from octa whom i've
referenced a few times in here
and i will again now
if you can gain a position of trust
you can exploit it think service or
non-person ids
accounts that nobody really scrutinizes
with enough privilege to leverage to do
e-discovery
and to find all the stuff you need for
an attack
because your chain is only as good as
its weakest link
and there are more ways to abuse the
chain of trust
than people realize let me leave that
with you
thank you so very much for attending my
talk
for the opportunity to share this with
you
those are my details there and thank you
to apsec village