hello and welcome to my presentation
on supply chain attacks on install
scripts at absec village at defcon 29
uh before i start i would like to thank
all of the organizers and attendees for
making this talk happen
um here's basic outline of my talk
a little bit of introduction we're gonna
talk about some different supply chain
attacks
i'm gonna talk specifically about the
curl bash pattern
uh talk about how we do verification of
scripts that use that pattern
some tooling that i developed some
interesting things that are found along
the way
and we'll finish off with takeaways and
hopefully their questions and answer
session
so as an introduction what prompted this
journey
so in april it was discovered that a
bash script used
by codecov.io which is called coverage
tool has been compromised for at least
three months
here's a screenshot from the codecovale
compromise page a lot more information
available there
but essentially somebody gained access
to the script
and since early january it has been
compromised
they injected code and as a result
codecov.io users thousands of
organizations tens of thousands of users
and they're running malicious code
within the environment
when i saw this originally i kind of
was it looked interesting to me and one
of the things that looked interesting is
is this something that's limited to this
particular
vendor or is this something that we use
every day and i went out and started
looking at other places where this
pattern is used
now that we know that we know that this
thing is possible
that a compromise of this nature can
happen
how do we go about making sure that it
doesn't happen again
so i'm going to be talking about supply
chain attacks but specifically i want to
focus on
what i think is a corner of the supply
chain
which has not been looked at before
which essentially is
these shell scripts that use what i call
the curl dash pattern
and talk about some verification methods
some approaches also some tooling
interesting findings uh before i start
just a little bit of an introduction i'm
a software developer have been software
for most of my life
um did security always on the side for
the last couple of years i've been
privileged to work in security full-time
but i want to be very very clear i am
not here representing my employer
i'm here in personal capacity i
presented for a bunch of other
conferences
uh there's a bunch of anti-spam things
out there that i've been involved with
some rfcs security.txt is something that
i've been recently working on
i'm helping out at um overflow who is
the original author it's about to become
an rfc and there's also some other rfcs
that i
have done in the past um here are some
cvs that i've discovered
uh also i want to be very clear is two
things
one is i'm not blaming anyone there's
gonna be a lot of
vendors and open source projects
mentioned in this talk but it's not
their fault
security is something that we all have
to do and it's not an easy thing to
implement
many open source projects are often
working for free
the second thing is also my senate
disclaimer
as you can see the screenshot below for
more games don't do anything without
talking to a lawyer
there are some techniques here and that
i'm going to be presenting
that may lead you on a chase to do
something you're not supposed to be
doing so once again
don't do something without talking to
someone in the legal profession
before we start talking about the
particular
pattern of curl bash and install scripts
let's talk about software supply chain
tags in general
modern supply chain due to its nature
includes a lot of different phases
there's the design of the software the
development of the software while the
software is built it gets distributed
and then you have to get it from
wherever it originates from to the
organization that's using it
and even after that there's updates
because
a lot of things are no longer done
in-house and there's a lot of shared
code
out there modern software includes
third-party components that come from
both open source and commercial sources
in addition even without that developer
tools infrastructure csd tooling a lot
of things that are
around the business of building software
also uses a lot of third-party
components
even after software is built and is
ready to go out the door
social distribution and updates a lot of
the infrastructure there
also uses third-party components there's
cdns there's web services buckets
developers and security teams often
don't know what's inside these
components how they delivered and how
the legit
uh whether the source that originates
from matches which you actually have in
your hand
and anywhere along the line any of these
pieces can be compromised
this is what many people imagine the
solar supply chain to be
you have a beautiful castle in reality
it's more like this cartoon
underneath a beautiful piece of software
that could be an open source component
that somebody has been maintaining on
their own
for many years for free and what if that
component is compromised and what if the
maintainer of the component
has been changed and now works for a
nation state
so the reality today is that even though
it may look like a beautiful castle
underneath there's a lot of pieces that
may not be so beautiful
now what are software supply chain
attacks
a software supply chain attack targets
any third-party components
infrastructure
that using software development delivery
it's sometimes it's third-party
sometimes it could be something that
a company owns but somewhat located
somewhere else
usually what an attacker tries to do is
to replace
a legitimate component or inject
malicious code
sometimes that may take a form of actual
code
sometimes they may target infrastructure
instead of the software itself
what is the goal of these attacks the
goal is
that they are trying to get a foothold
inside organizations that are using the
software that's being built
and then usually that gets followed up
by something much more nasty
once they get the initial foothold
through the door through the compromised
software or infrastructure
there's usually things like ransomware
stealing data or
something of that nature that comes
along the way in the recent
years a lot of these attacks due to the
sophistication
required to pull them off have been done
by apts and nation states
now some examples this is just a picture
from that landing console report in 2020
they tried to go out and catalog how
often did this happen
the 115 attacks that they have in the
catalog you
follow the link to the paper uh various
different stages
as you can see here 32 are in the
beginning
where the software components originate
them some of them
those six compromises the case you got
open source
repositories never compromised you got
some sort of account access attack
where i'm assuming probably developer
account that gets compromised you got
absolute compromise
you got updates that's the flavor of the
months updates are often also
compromised
and you know things of similar nature
here are some examples that have
happened recently
not all that necessarily malicious cdngs
cloudflare sadn that serves open source
components
research has shown that it can be
compromised the bug was fixed
some research from alex berson recently
also showed
how you can compromise modules that come
in from public repositories
codekov as i mentioned happened also
solarwinds
obviously is the big one that happened
december of 2020.
there's have been multiple attacks
involving xcode projects because xcode
can execute binary code
developer downloads some xcode project
they saw on github they get compromised
um famous twilio sdk compromise because
of a
s3 bucket that was misconfigured not
patea malicious piece of software
allegedly from the ukrainian government
released the virus also too many to
count malicious modules
in open source repositories this happens
all the time
npm gems python repository java
to a lesser degree happens all the time
mage card attacks
the magic card attacks is when
javascript gets compromised in a web
page
sometimes those compromises happen from
a piece of javascript that comes from a
third-party vendor and obviously many
many more
what can we do so before we can even
talk about
solutions to supply chain attacks one of
the things that's very important is
awareness
do you know what's inside your software
what's in your infrastructure what's
inside your tooling
your csd system your artifact repository
cdn's
the awareness of all the different
pieces that are used to build your
software
or all of the different infrastructure
that supports it and once you know
what it is do you even know where it
comes from if you have a release process
which i hope you do
who's in charge of it what's involved in
getting raw source code
into an actual release these are not
easy problems to solve
very often you could go down a rabbit
hole you could have a component that
uses a component that uses a component
same thing for infrastructure today's
infrastructure is no longer just a
simple web server
many times there are third-party pieces
their cdns
their package stores there's all kinds
of cloud stuff
and then if you're an organization that
publishes and updates
software do you know what's involved in
your update process
are those updates digitally signed
the reason why this talk is called the
poison diary is because for those of you
who
are harry potter fans this is a poison
there is a
magical artifact called the poison die
the diary
that was running essentially
magical code turned not to be malicious
and
the point of the diary was
you should not trust anything that can
think for itself if you can't see where
it keeps its brain
if you are not controlling the
particular component
that you're using in your software
infrastructure or if you're not
controlling the particular
infrastructure itself or the cicd
process how do you know what's going in
there
how do we secure the supply chain this
is a very very hard problem
the basics require which are the
requirements are pretty simple
if you're a consumer of components
you just need to make sure the component
you are getting matches what the
original
project or commercial vendor actually
produced
in addition you also have to make sure
that whatever they're giving you
has not been injected with malicious
code if you're a vendor
or if you're an open source project
maintainer one of the goals that you
have
is to making sure that whenever you
publish something it actually matches
the original source
at the same time you want to make sure
that nothing was injected into the
original source
but these are not easy problems to solve
here are some approaches that are found
in the wild
so some people choose to answer security
third parties
for example in my research which i will
show later
there are people who rely on direct
github downloads
meaning their script that they're using
is located in github source code that
bypasses the entire
infrastructure used for builds and
you're getting this script essentially
directly from the source
some people choose to rely on package
managers because packet managers
have a more narrow release process
people who
are putting stuff into package managers
are supposed to
be you know have some semblance of
control over their software
there's also a push to do some
cryptography checksums which we'll talk
about later
there's always digital signatures so for
example gatekeeper
and mac os there's athena code in
windows using certificates to sign
things
in linux ecosystem you could use bgp
signatures
uh pgp signatures straight up uh we'll
talk about that as well
uh publicly ecosystem there are ways to
get a cold signing certificate from a
public certificate authority
just like we do with ssl and by you
trusting that ecosystem
you end up trusting a certificate there
are attacks in that as well
there are some other ones you could do
an antivirus scan
unfortunately that doesn't necessarily
find something that's new
you could disable automatic updates then
you get to lose
if there is some sort of a flaw in the
software you have
there's going to be delay till you apply
the update there's primary defenses you
could have for example
your ci cd system locked up you cannot
go
call out to the internet that makes
things harder you could do detonation
where you could try to download the
update
and see if it does something malicious
kind of behavioral
pieces there's manual review which is
you know sometimes easy sometimes hard
but requires humans
there's some experimental approaches
that haven't taken off yet
for example reproducible builds there is
a push
out there to be able to take a piece of
software and build it
and see if that piece of software
matches the official release
because of the differences in how builds
are done it's not an easy thing to pull
off
and obviously there's blockchain
so now that we talked about supply chain
tanks in general
let's talk about the particular corner
of the supply chain
that i have been looking at for the last
couple of months this is what i call the
curl bash pattern
what is the curl bash pattern here's an
example from homebrew
if you want to use homebrew you
essentially use curl to pull the
insulation script
and you pipe it into bash this pattern
use curl bash or wga bash or some
similar approach
to download a script from the internet
and execute it automatically in your
shell
without anything in between it's a very
convenient way to install
software especially if you want to do
something like bootstrapping
there is a long-standing debate is this
safe or not
my honest opinion it is not safe because
you are executing code that originates
from an outside system without looking
at it first
historically security concerns with this
approach have been downplayed
and hear sound excuses that have people
have used
oh don't worry about it a user will
review the script before running them
or these scripts are not used often like
how often does somebody install homebrew
third one a malicious actor can
compromise other things just as easy
and i will talk about some of these now
reviewing scripts an average shell
script contains
unix commands here's an example from
xkcd which i'm sure many of you can
relate to
use commands use very complex command
line options
so here's an example using tar can you
solve this problem can you type an
entire command
without googling i cannot and i bet you
many of you cannot also
the same thing applies to the shell
scripts many times a shell script
is pretty obtuse so if we cannot rely on
users
to be able to read a shell script here's
another example
kotkov that i mentioned earlier the
script was used by thousands of
organizations
tens of thousands of people the way the
issue was found
is through checksums it was not found by
somebody
who was reading a script so you have
three months of usage you have
tens of thousands of humans machine
cacd pipelines etc not a single person
realized that they were using malicious
code the only way we found the problem
is because the attacker forgot
to update the jackson file another
example here's
pip this this is one of the ways to
install pip it's not the only one
the way i found this is because i was
monitoring for changes in the pip script
over time
and i couldn't view the changes it turns
out they tell you that
straight up pip includes binary data
in the end it's base 85 and go to python
so if you expect your users to actually
read this it's not readable
i actually went through the trouble of
writing a small little utility trying to
see what it would take to actually
decrypt this
um you encode it and look at it it
produced a whole bunch of different
files which were not easy to review
another example going back to codecov
since the incident has happened kodkov
has come up with a new
idea instead of using a shell script we
will now use a binary
so before if you were expecting people
to review
a shell script now we have to blindly
trust someone with a binary and once
again like i said earlier
i'm not blaming kodkov this is a common
issue
with many many different projects where
we have to go and blindly
trust a binary here's another example
one of the concerns that people
mentioned is that hey it's not a problem
because it's not used very often it's
usually used by humans
it's usually not something that gets run
on a frequently basis
here's an example if you could look here
at
line 2 13 and 9 214 this
installs rust from a shell script this
is part of a docker file
that is part of a project part of an
open source project called the pandabot
again not blaming the pandabots but my
point is that
historically these scripts do not always
end up being used by humans
they end up being incorporated into cicd
systems
and it's not just about scripts if you
look further down line 221
and 222 you could see a similar approach
where we are
downloading some other thing from
hashicorp it's not a shell script but
the concept is the same there's
automation happening here
and if somebody were to interfere with
these files
a lot of projects that are relying on
this automation will become compromised
like i said in the earlier we don't want
to trust something
unless we know where it's coming from
and this is very true for these shell
scripts because a lot of times
we these shell scripts are not something
that you
have an opportunity to review you don't
look at them so why are we trusting them
how common is this one of the other
concerns that was raised and when i had
discussions is
is this really something that's common
so i went out
and i started coding a list this is what
i this is one of my projects which i
will
show you later i want to show you know i
want to see how common is this thing
so unfortunately this is fairly common
the good thing is
that this a lot of times is not the only
way to install something
a lot of times there's an alternate way
which is more secure the
downside is if you want to verify
whether these scripts are actually legit
it is not
an easy problem to solve but the bottom
line is do you even know
if this pattern is used in your
organization
here are some examples um here are some
projects that use this pattern so that
you know and i'm going to show you some
examples later on
azure you want to do the azure cli now
to their defense is not the only way to
install azure
but if you want to install it from a
script this is how you do it
dell firmware update you want to update
this particular server
the way it's done is using a butch this
particular script
um firebase there's a client of google's
firebase
you want to install the tool this is how
you do it again coming from a shell
script
obviously homebrew that i mentioned
earlier pip is another example that i
mentioned earlier as well
rust rust is also you know it's a fairly
new
programming language becoming very you
know very popular we install it is using
one of these scripts now
if you are doing install scripts and
this is how you bootstrap
if i'm an attacker what can i do so
scripts
just like every other piece of software
comes from the original source code to a
distribution point
in codecov's example it ends up on some
web server um
unless you're doing direct downloads
from
a core a source code repository like for
example in the homebrew case
it's coming directly from github it's
going to go through some third-party
distribution point
you could target anywhere any
infrastructure along the way
web servers or cdns so cdngs
third-party cdn hosting javascript it
copies the javascript from some source
puts it on cdn
on the cloudflare recent research has
shown it can be compromised
cloud buckets you see something in the
cloud bucket you forget to configure it
properly
can be compromised twilio sdk that
compromise that way
codecof one of the underlying issues was
some sort of a
google cloud issue with the bucket build
systems
solarwinds if you look at the reports
solar windows compromised by something
being injected into their build
environment
original source code repository php
recently had an attack directly on that
git server
you don't necessarily have to go after
those systems directly you could go
after the people who have access to
those systems
in my opinion these are novel attacks
why because these things are not common
shell scripts are not as common as for
example npm module
but you get a bigger bank for your buck
if you're an attacker just think about
what it would happen if ross installer
or homebrew compromised even for a
couple hours
now that we know that these scripts
exist and they represent
some sort of a secure risk one of the
other things that i
want to research is if i want to do the
right thing
and to verify that these scripts are
legit how do i do that
let's say you want to do purification
so i looked into how that can be done
this is not an easy issue especially
with cryptography
pgp becomes very complex i did some
tooling
um i also when running it for several
months i reached out to multiple
projects and commercial vendors
most unfortunately refused to implement
the request because they did not
consider the risk to be significant
enough
like i said earlier i do not blame them
we are only aware of
a handful of attacks on these particular
scripts
there's other issues that also came up
but at the end of the day if you want to
be secure
what do you do easiest thing don't use
them
don't use the script in many cases
there's another way to install something
instead of using a script you could use
a os package you could use
a language package some offer official
releases and binaries
those are not perfect they can be
compromised but they're not is
necessarily
compromised in the same way a lot of
times they're harder to compromise
there's also a formal release management
scripts often don't have formula release
management another version and they're
not released
former release management helps because
you have a narrow
pool of people who could do that as
opposed to everybody who has access to
the infrastructure
if this is not again something that
could be convenient in a lot of cases
shell scripts are very convenient
another option you will go directly to
the source code
like the homebrew case i showed earlier
you could check the script directly from
the source bypassing all of the third
party infrastructure
this relies on lions's law enough people
see it someone will notice something
the flip side is that when you have a
source code repository with many
committers only one commander needs to
be compromised if it happens let's say
at night when the other committees
aren't there
it could be you know fairly dangerous
uh changes to source code all take
effect immediately unlike when you have
releases which takes time
um this can happen as you know as soon
as as soon as as soon as the attack
happens
um you can review the source manually
you can save a
good version um but a lot of these
scripts are not versioned
and therefore essentially what you're
pulling is the latest and greatest
and one example that i found um again
this is an example of home brew
again not blaming them this was a
mistake back in 2020
in 2012 one of the developers made a
mistake and for about 10 minutes
the homebrew script pointed to a
different repository
so as the result for 10 minutes somebody
was downloading something else that was
supposed to
the developer originally finally
reverted
next another method checksums many
projects publish exams
so for example here's the case with code
cop they were publishing a checksum file
if you think about checksums are useless
for security because an attacker can
modify them but
as the cold calf case is shown it is an
extra barrier
because the attacker has to be aware of
it and go ahead and check
you know change the file you could also
choose to host the checksum to the
separate server you could put them into
the code repository
now that that has to compromise two
servers the same thing goes if you want
to compare the script
let's say you have a cdn you want to
compare against the original there's
another option
there's also an option of doing this in
the browser um
sri if you do checksum in the browser
where
you know cdn suddenly changes that
really works well
but it only works because um you in the
browser
javascript components are usually
versioned
you could go a step further we can do
cryptography
so some oss can let you sign
up things with a cia certificate
chocolate uses this approach
that uses the c infrastructure in the
case of windows in the case of mac it
uses
apple's infrastructure there are issues
with certificates you could get a stolen
certificate you could get a legitimate
certificate but it's still better than
nothing and it's not
it's not bad um because the pgp you
could use pgp you could create a
separate signature
but it's not ca based so in a ca case
you're relying on
the fact that cas have some sort of
checks in pgb's case
the brown you run into ski distribution
if you want to verify that a project is
legit
how do you distribute the key if you put
it on the same server the attacker can
change it
if you have a pgp server those could be
poisoned
i have not seen yet attacks that involve
compromised keys
but it doesn't mean they cannot happen
here's an example of chocolaty as you
could see
the chocolate script itself in the end
has a couple of network signature
here's an example of nyx os they're
using pgp
and when you are installing the original
install script which is
on top you can go ahead and pull the
specific
keys and signatures um they're using
as you could see a key that's i didn't
buy an id
by fingerprint from a key server and you
can actually verify that the script is
legit
in summary best thing is to avoid shell
scripts
and use package managers but it
necessarily means
as a safest option it's going to depend
on a particular use case
you could check sums you could download
from source you could do comparison
against version another version it helps
as long as the attacker forgets to
update both
files or if it's hosted on both servers
the priority cryptography works
better than most of these approaches it
is hard to set up and use and has its
own issues
like i said key distribution attacks for
pgp
ca certificates have issues as you could
get an illegitimate certificate or you
could steal a certificate
most projects that i looked at do not
provide any kind of signatures
bottom line is organizations need to
make their own decision if you're using
these scripts
and you want to verify them it is up to
you to make a decision how to do it
properly
and we still need more work and more
research on this particular area
let's talk about tooling and i also want
to share some things that i found
so tooling one of the things that i
wanted to find out if i want to do
verification of the scripts
in automated fashion how do i do that i
also want to see how often these updates
happen is there a lag
between let's say a signature being
updated and the actual file
i want to monitor scripts for changes i
was curious about what kind of changes
happened install scripts
the existing tooling wasn't really there
i happen to write some of my own
but it is kind of experimental uh the
best way to do verification let's say
using pgp is to use a tool like
gpg directly those tools are not always
easy to use
one of the things that i ran into is it
changes machine state
when you download keys it changes the
key ring on the machine you're using
um i didn't want that i want something
more clean there are ways to do that by
using temporary directory but again i
wanted something a little more cleaner
i also want to have some configuration
so i create a bunch of experimental
tools
um the first one is just a github
project listing various different
projects
um using this this particular pattern
there's a blog post this talk obviously
also like a suite what i call ice trust
it's a tool
uh that lets you actually verify
downloads some dashboarding around it
and also there was another tool
release auditor i found a bug in github
uh which i'll talk about in
a little bit but it's not really
connected to the rest of the tooling
what is the ice trust eyeshadows the
python module lets you verify files you
already downloaded so basically you're
doing your curl you get
wget and then you are using this tool by
either comparing
two different sources of script or
you're doing a checksum file or pgp
this is what i call canary mode where
you could have json configuration files
that can be run on a regular basis very
useful for
an open source project or a vendor
that's publishing software wants to make
sure that monitor
make sure that the stuff doesn't change
um some dashboarding example
like nice visuals to see if you could
see like multiple products at the same
time there's some twitter account that
also
gets results under the hood uses github
actions
you could look take a look at the code
github itself is used to
store changes here's just
an example i showed it earlier this is
just a simple
list of projects in the project what the
script is
if there's any particular integration
linked to the original issue
if it's known um ice trust istrust lets
you
essentially do things that you could do
in gpg but a little bit easier
if you could do gpg you should do it
yourself for example here you do your
curls
or you download the files you download
the signature and then you essentially
run
my tool against it with the key that's
known
if you want to use the key file itself
you have to download it separately and
you can do it that way again you could
do this with gpg
but i decided to do it this way because
i wanted to
script it in order to do this in a
larger scale
here's canary mode canary mode is where
you have a json file json file
essentially includes information about
the project instead of doing the curls
that matter and the curls yourself
the tool will do it for you and it also
outputs some
data dashboarding this is a very simple
github actions
github pages jackal configuration takes
the json file
and what essentially this does is it
looks at
verification results on a regular basis
you can also follow the links take a
look at
the file being changing um here's an
example
of file being changed yes codekov you
could see that for example
uh two days ago the version was changed
um
using the regular standard github blame
and something a little more fancy is i
trust
uptime it is called an open source
project called apa pub time
it lets you essentially look at it kind
of like a status page
type thing um it also as you can see
here
it also lets you it opens up it issues
automatically to kind of track
historically what was changed what was
you know verification failing
uh twitter account just tells you you
know if i'll kind of change verification
fail verification passed
um how do now what i do with this with
these tools
so i'd recommend 25 projects to use this
pattern there's a whole bunch more that
i reached out
um still going through the disclosure i
can't talk about them yet
monitoring signatures on five projects
like i said most projects don't offer
cryptography
um also looking at where there's two
different copies looking
one against the other monitoring some
for changes
runs every six hours piping into all of
these different places
eventually looking at incorporating to
my own projects but
right now it's more of an experiment i
have been running this since
april what did i find
only my narrative projects offer a way
to do cryptograph cryptography
through pgp most project events are not
interesting to fix they don't consider
this issue to be important enough
although i suspect if these attacks
become more common that may change
there's usually better alternatives in
most projects but not all of them
raw github downloads is a fairly common
pattern
that's a lot how a lot of projects kind
of bypass those infrastructure
challenges
most of these scripts don't really
change very often or change in very
minor ways
some cases verification is not trivial
you have to parse them to get a version
number from the version now we have to
go to branch
there is a lag that happens sometimes
from the time a script is available
until the signature gets calculated
i suspect some projects haven't gotten
their release management done correctly
yet
the downside is that that happens and
you're doing verification automatically
the automation fails
there's often no formal release
management version for most scripts
the good thing is no supply chain index
i haven't found a single
um active attack as a result and it is
possible to
do monitoring on a larger scale
some incidental findings um above in
github you could go and modify a release
without anyone else knowing about
it i let github know they don't consider
it to be a bug i wrote this tool called
release order you could take a look at
it
um another surprise was firebase cli
from google for whatever reason it's
called google analytics
could be a privacy issue for those
people who are concerned
some scripts don't match the source code
when i want to do comparisons
there are some slight differences in the
source code it's a bit of a problem
because it's kind of hard to tell why
they differ
the research is still ongoing now what
can we take away
from this entire um you know journey
we need to have more awareness about
what's in our software
tooling and builds remember like i
mentioned earlier the poison diary
you need to know where the tools we're
using with
they keep their brains essentially the
usable subscripts in my opinion is
dangerous
they're so safer alternatives but you
have to validate your own risk
accordingly
not everyone agrees the same way and i
you know
can't say across the board this is
dangerous in every single case
it is possible to verify some scripts
however it's not a trivial case
you should prepared you must be prepared
for things that can break
if you are publishing scripts or
software yourself consider monitoring
them
you know for changes that could be an
additional control
a formal release management is important
uh these scripts don't necessarily have
formal release management in most cases
less people have access more you know
there's less things to compromise if
your former approval
if you have you know two people
approving it um
there's a lag between when the source
goes modified till it's published
use native tooling my tooling is
experimental you feel
feel free to play with it but it is
experimental and the
last thing is don't forget the rest of
the software supply chain
these scripts it was a fun project for
me but it's just one tiny corner of the
overall
software supply chain challenge