hey uh thank you
omar uh i'm super excited to host
the luminaries of the ai village and the
red team village where i think what may
be the first
joint panel uh in this space so to set a
little bit of contact like
ai red teams and whatever teams are kind
of mushrooming everywhere from
facebook to microphone to nvidia and
even in the government
but there's a lot of like misconception
about water days and
we really don't we really want to take
this time to kind of like peace out how
different it is from the regular red
teaming
and most importantly how can the
security community and the machine
learning community come together
to kind of like deal with this change so
i want to quickly introduce the panel um
we have with us
the celebrated security guru he's been
thinking a lot about
uh attacks on ai systems um
you know under recently so welcome bruce
uh we also have omar santos who we who
leads the
uh we really need to be thankful to him
for all the wonderful cpfs that are
happening
as well as as chris capelle from nvidia
um
ai red team head honcho so thank you uh
omar and chris from representing the red
team village
um from the ai village we have uh dr
anita nikolaj
who's the director of research and
technology innovation
at the university of illinois
urbana-champaign and dr anita is really
gonna
ground us today on what is actually
possible
and what is actually happening in the
cutting edge research from academia
and finally uh we have with us uh
security data scientist
uh rich harang from duosac who's also
now put out
very exciting uh twitter's uh bias but
bug bunny which we will be talking about
today so you know with this i really
want to get uh
quickly started i want all of you to
hear from the experts
bruce i want to start with you um
you know you you had a very thoughtful
real quick
at the ei villain um recently
you know he spoke about how lifespans
will uh eventually find loopholes
at blazing speed i really love your um
for example with tax evasion so
how do you think about can you just
elaborate uh about that for
just a quick minute for people from the
red team village and i really want you
to kind of like touch on
how you think about humans attacking ai
systems
so i've been watching ais become hackers
that happened at def con i think 2016
when uh ais had their own capture the
flag contest
and more interestingly there's a lot of
research in ai's finding vulnerabilities
in code
it's kind of the thing you'd expect
iii's be good at it's uh
it's pattern matching there's a lot of
data it's a lot of it's a lot of
repetitive
work and they're not very good at it yet
they're going to get better at it
so when we think about hacking and
vulnerability finding
it's no longer going to be a human-only
creative endeavor right the two parts to
hacking
this there's the creative part figuring
out what the hack is
and there's the execution i mean yes we
can automate the execution
that's pretty easy but automating the
finding the clever
vulnerability finding the exploit making
it work
that's going to become increasingly
automated so
that you look at at hacking ai systems
now
and you know ram you all know that it's
primarily
human it's done by humans it's a human
creative process
that's going to change probably the next
few years slowly and then quickly like
all of these things change
oh i i was looking at a great time to
kind of bring omar in
uh omar you've been like you know uh you
have a wealth of experience in kind of
like retaining traditional systems
so right now hacking involves like human
um you know like how bruce pointed out
how did you see the space kind of like
uh
evolving yeah i think that actually ruse
mentions something extremely relevant
right but before
before i go in deeper into the red
teaming of ai systems
let me actually define the two aspects
of that word
or the phrase a i read timing that you
mentioned right so
one is about attacks against the an
environment so we're gonna talk about
that a little bit later
and the other one is what bruce was
actually mentioning is actually using ai
or machine learning to
to attack you know different platforms
or to perform data
manipulation because at the end of the
day a lot of the attacks are going to be
seen is a data manipulation and
poisoning of data
of training data so that you will you
know uh
cause some damage into a into the ni
system
but you know going going back to what
bruce mentioned
if you remember four to five years ago
darpa had a competition that's what
bruce mentioned
here in defcon right it was called the
darpa cyber challenge
and they had different teams that they
created machine learning environments to
do
both things so to attack and emulate
different type of attack methods
find vulnerabilities and then the other
side of the coin of course protect
protect against those vulnerabilities
right so they were just were trying to
actually patch at the same time
and defend based on that behavior the
adversary behavior right
now ai you know of course machine
learning will definitely be a
big big role of the attacks in the
future from manipulating
people right the masses in in social
engineering
a type of tactics and so on to learning
about
weaknesses and the underlying systems
and learning how humans may also
defend against those attacks and respond
to those attacks so remember
that in a traditional incident response
you know
you you have the ability to you know
detect whatever that the threat actor is
actually doing what we call
the tactics techniques and procedures
what if
you know the attacker is actually able
to learn what are the mitigations and
the responses
from the from the you know security team
and then you know of course evolve into
that right and then
i can think of you know things like
anti-forensics capabilities you know
being inserted
into these environments and so on now
shifting back to the other concept
attacking the ai and ml systems the
first thing that
the traditional you know is the
traditional vulnerabilities against the
underlying system itself you're going to
continue to see at the end of the day
you're actually using
you know a proven technology nowadays
for machine learning however
one thing that you're seeing a lot
nowadays there are companies using black
box
machine learning solutions that you have
in the cloud and so on right
and even though they actually try to
sell you some machine learning
you know thing they're actually using
somebody else's technology that is
probably cloud-driven
they're sending a whole bunch of data to
that system
what if you actually can manipulate that
system and as a
whole now not only attack one but many
you know different resolutions right at
the same time
um data manipulation if you can actually
poison
data you can actually manipulate the
results so i work at cisco as you
mentioned
we use machine learning solutions of
course you know
and reach here you know it's from duo as
well and then
one of the things that we're actually
doing is that what if you can manipulate
and poison network telemetry
that could lead into different attack
vectors or activations
false negatives or unnecessary actions
taken
by an administrator or the security team
right so all those aspects actually come
into play
from an adversarial perspective so
i mean i think that's a i i kind of like
i'm very excited to hear about like
rich's point of view on this uh as well
because you've been like building like
machine learning systems
like forever for security now so
what is in this ai red teaming how are
you going to kill him
uh in this case so i think
omar was was right on point when he
pointed out that
there's an entire system involved in
doing these um and doing these attacks
right you're not just attacking
only the ml you're attacking the system
that it's embedded in
and you still have these same
traditional
vulnerabilities that you can go after
and i think
what's maybe a little bit different
about the ai space when you get into
stuff that's specific to ai
it's sort of just it's it's the same
really it's there's new capabilities
that exist right so you have
like you can attack and you can do like
training data extraction you can attack
you can do model model stealing or
something like that
um but you have to do it when this model
is
in sort of an embedded system and so we
have academic research
which says oh yeah these things are
possible right we can
do a black box attack and we can extract
some of the training data or we can
you know find if there was you know ppi
that was used to
to train with the model or something
like that
but most of those techniques don't seem
to have really made the leap
into practice and i think part of that
is because we have this disconnect
between treating the model
as sort of part of a you know treating
the model on its own which is kind of
how it typically is handled in an
academic setting
versus as part of a complete system so
when you're talking to people that are
building and deploying models
in the industry right now we are
thinking about those things right
we we fuzz our feature our feature
extraction
to make sure that there's no crashes
there we sanitize our inputs we double
check
uh double check our telemetry and we
keep an eye on telemetry to see
if where things are going so all of sort
of like the good software hygiene
that comes along with deploying any
application ml or not it's
it's very similar you have very strong
parallels between those
um where we sort of fall over
is we know that there are these other
kinds of attacks that can be launched
against
ai driven systems but right now it's all
sort of in the academic space
and so i think this is where we're kind
of hoping
that that red teams can lead the way
right as they find
which of these theoretical attacks are
actually sort of
practically achievable and practically
deployable
that gives people who are invested in
defending these models
specifics right as opposed to just
trying to prove a negative right tell me
that you know
to me that this model cannot is not
subject to a data extraction attack
versus we've done this one now defend
against it
yeah i really like your point about like
how a lot of these attacks are in this
academic space i want to kind of like
um get to anita just real fast but we
have somebody with us
who's actually reading leading an ai red
team
uh chris you kind of like you know uh
are leading this like nvidia's ai red
team you know we just heard from like
rich that a lot of these attacks are
still like in the academic phase
and kind of like bruce also kind of said
hey we've got like humans going after
like
machine learning systems there is no
like ai behind it at this point
so what exactly do you do chris at
nvidia what exactly is like
you know the nvidia ai red teams charter
why did you create this team and what
exactly are you doing
in terms of attacking ml systems sure
um i'm super excited to be here by the
way
um this is a new field for a lot of us
and
uh we started it because it sounded cool
first and foremost and and second of all
because like
uh specifically at nvidia we
we have a lot of ai things we have a lot
of machine learning things
um and like omar mentioned you get a lot
of these companies that have like black
boxes they take models they
they ingest models from other places we
are actually creating
some models on our own so it behooves us
to figure out how to do this stuff um
because not only
are we consumers we're also suppliers in
some regards
um so a lot of people came to
me and my team and they were like when
are you gonna start doing ai red team or
how do we hack an ai or how do we hack
ml
and you know a year ago we really can
answer that question
and you know red teamers we like
challenges and we
just tried to do what we could we
started researching it and a year later
you know we've
we've gotten into partnerships like
these and
um it's still very table top-ish
right now but we are getting to the
point where we're
gonna start doing some operations
probably soon um
chris can i follow up um you know i
think you mentioned the core of today's
talk which is like
hack ai systems what does it mean in
practice like
you know even if you're doing tabletop
exercise can you just like tell the red
team members
what an exercise would look like
yeah sure um so some of the things we
have scoped out so far
there's there to me there's three main
attack areas
uh that we can go into is like when
you're actually attacking an ai system
there's
as the model is being trained as like
after the models trained and after the
model is deployed
um so each of those three phases you
have different attack scenarios and
depending on what our customer was or
how we wanted to do things we would have
to figure out
what type of operation we wanted to do
do we want to poison the model do we
want to replicate the model
do we want to you know after the model's
been deployed like are we going to try
to test systems to see if we can confuse
it
um so there's there's there's a lot to
it it's like a mile wide and an inch
deep
so at least that's what i'm thinking so
far like i said we're pretty new to this
too
oh thank you i'm gonna i'm gonna pull
that through a little bit more
but i first wanted to get dr uh nicholas
uh into their conversation so
yeah you're hearing like you know rich's
comment about how a lot of these are
like academic
uh can you paint us a picture on what
exactly
are those types of attacks that
academicians are kind of like
thinking about on machine learning
systems and do they have actual real
impact
or is this more theoretical kind of
exercise
so uh since i'm not in my day job i can
be a little down on
on ai and ai security and academia so ai
security is a new concept for
academicians
um ai gets a lot of federal funding by
the way nsf just for another round of
these ai institutes none of them are
focused on ai security
but the focus being on agriculture
chemicals water physics science if you
can imagine nothing
in there talks about the security of any
of this ai
um i think much like you know a few
years ago disinformation
and that whole concept was like a big
no-no for researching
uh but now we figured out it's important
i think the same thing is going to
happen with ai security
but to get back to your question many of
these attacks on ai are very esoteric
and they are focused on different
portions in the pipeline
my frustration is that ai is a pipeline
right there's data collection and
cleaning and ethics and
system and software and most of the
academic attacks focus on one esoteric
niche of that not the entire pipeline
and i think that's that's a big problem
and and can you also help us explain
like when when people talk about ai
security
uh is it like attacking ai systems you
mentioned misinformation like gpd3 to
kind of like generate like
disinformation
then there's like what bruce can have
alluded to in this awesome keynote about
the coming of ai attackers so what
exactly
is ai security and how are people
perceiving that anita
so i mean ai security we know that you
know as
more these ai systems proliferate we
know that there's unanticipated
behaviors
one of the places academia can focus is
on how do we quantify how do i iden how
do we identify these unanticipated
how do we how do we deal with the
unknowns and right now that's becoming
really impossible
uh one thing i'll throw out there that i
think um i've seen a little bit in
academia and i wish i would see more but
many people are not keen on this
is to think about design thinking and
futures thinking
and that really takes you know the whole
pipeline to say okay there's
this this type of system what's the best
use of it what's the worst use of it
what's the optimal use of it how could
an attacker do it so really almost like
um
almost a tabletop exercise but thinking
in the future and this is facilitated
not by
technical people like many of us but by
design thinkers to think how society can
use this and then thinking about the
pipeline
then you could have the you know the
more academic people think all along
that way
where can i then attack it just one
suggestion
uh absolutely uh what do you mean by
design thinking anita
i'm sorry can you just like explain that
concept a little bit
yeah so designers i mean there's whole
schools of design
and one of the things they do is called
futures thinking this is something that
came out in the 70s so
you might imagine like this and i took a
futures design class for fun and
somebody did this which is what they
said
what if funding for public art was no
longer what would happen
well maybe a company like amazon would
make public art but maybe
and this is part of the design thinking
you sit and think about these futures
if amazon funded public art maybe it'd
have to be a prime member to see the
public art well how would that work
and so the kind of storyboard i think
all through and
uh ashken sultani who was an ftc cto had
this great coin this great phrase about
abusability thinking
so how can we use how can we abuse the
ai and i think if we think in the
broader context
that's when you can really think about
red teaming i mean much of red teaming
is the
physical aspect of social engineering
the technical trashing a model is easy
it's everything around it how do you get
to it how are you going to poison this
model
i think design thinking can help us
that's awesome because like
i have a question for bruce which i
think just tailors on to the brass tax
aspect
are we even prepared if attacks on
ancestors are coming are we prepared for
this
you know i think we're never prepared
abusability thinking is the security
mindset right how does it fail how can
it be made to fail
and if you think back to the the things
omar was saying
all of that those adversarial ways of
thinking about
ai machine learning and we're never
prepared because
people don't think about security i mean
you just point that the funding is
is for topics for things the system does
not the things the system prevents
and we saw this we saw this two days ago
apple announced this system they're
going to scan your iphone looking for uh
for uh for abuse images yeah and
you read theirs they have a lot of
security they do not
talk about the adversarial ml aspects at
all
they do not assume an adversarial model
they do not assume data poisoning they
do not
they do not assume any of those things
and this is apple
this isn't some bunch of idiots right so
here again and i see it again and again
people design systems for functionality
for how they work
they don't design them for how they fail
and i like the the phrase abusability
thinking i think of it as a security
mindset
it is something we at defcon have been
doing since
forever and it is the way we think of
systems and no of course
we're not ready because nobody's calling
us when they're designing the systems
they're just designing them
and and bruce for people who are just
like um you know
who are not exposed to adversarial ml
can you give us an example of what
um adversarial ml thinking in the apple
case would look like is there an example
that comes through
so there are easy examples so image
classifiers classify images and they
assume the images are like
regular images there's an entire class
of research
of making changes to images that the
human eye don'ts and detect
that the image classifier uh
deliberately fails
there's uh you can put stickers on stop
signs
turn them into machine uh speed limit
signs that the ml systems and cars will
ignore
there are ways you can change though the
famous one is changing a
turtle into a rifle just just look up
image classifier hacks and you'll see
lots of them and that's an easy example
that's not even a hard one
that is i mean i feel like especially
attacks on
images are so captured people's
imagination and i kind of want to bring
rich in here um which
uh before we get into like you know the
challenge i want to kind of like can you
talk through the chaos that happened
with twitter's image cropping algorithm
like um so um
i should clarify that i don't work for
twitter uh
okay um i was i was an interested
bystander in this
and uh yeah essentially what happened
was
um people began to notice so twitter has
a cropping algorithm
by which if you have a very large photo
that you post into a tweet it will
actually try to reduce down
to a narrow portion of that that could
be shown
in the complete stream of tweets um so
that it doesn't you know one image
doesn't blow up
you know and dominate an entire
somebody's entire feed
what people noticed was that it was
beginning
it was cropping in strange ways so if
you put
a person of african descent next to a
person next to a white person it might
preferentially crop the white person
if you put a woman up it very often
tended to focus
on their chest um and
a lot of this was driven by the fact
that what they did was they used a gaze
tracking algorithm
to train a saliency model to say what
parts of the image would be most
interesting to look at
unfortunately the population i believe
skewed white and male and so that drove
the saliency algorithm which then drove
the cropping
so again it's back to what anita was
talking about it's this entire pipeline
of decisions that you have to consider
um so to their credit twitter did
immediately address that they they
released a blog post where they
analyzed the results and they found that
really there was there was some small
bias there but what was happening
was people were finding sort of the
remarkable events and highlighting those
because those have the most impact
um and they've also been very very open
and transparent with this ethics bug
bounty that they've
uh launched in collaboration with the ai
village and again
that's that's something that all credit
goes to ramon and utah
and their teams um for pushing that
through twitter uh
really we we hosted it and we sort of
helped them kick the tires and think
through some of the
issues with the bounty a little bit um
but yeah it's it's another
illustration of how what seems like a
series of pretty good ideas
um can actually lead to a machine
learning classifier
that can have an impact that actually
upsets or even harms
some people by cropping them out of
photos where they really should have
been the center of focus
or um highlighting bits of their anatomy
when
really what should be highlighted is
what they did or
things like that so yeah i guess does
that answer the question yeah
i was i i really enjoyed the twitter
bounty because i think it's a concrete
example of how an organization is trying
to bring
uh anita's like anita streaming of
abusability thinking
into this like space um i would love for
you to kind of like
tell like people who want to get into
this ai red teaming
do you need to know math who actually
can test ai systems
should you actually have already machine
learning knowledge to work in this space
so i think it depends on how deep
into the space you want to get like with
any other pen testing
there is sort of different levels you
can do it at
if you want to just do sort of high
level
throw stuff at the wall use known
attacks redeploy them we're starting to
see
tools and frameworks come out so you've
got things like
toucan strike counterfeit clever hans is
another example
so frameworks that will actually let you
execute
um attacks against ml systems yourself
and really you don't need much technical
depth to be able to do that basically
you need to be able to figure out how to
run a model
because like metasploit for machine
learning sorry that is that how it works
counterfeit is yeah broadly i think you
can you can think of it as metasploit
for machine learning the other
frameworks are
a little bit too can strike um tries to
sort of go in the same direction clever
hans
you need to get your hands a little bit
dirtier none of them require
you know a phd in mathematics or
statistics to
to really get into um as you start to
push the boundaries a little more and
you want to think about developing your
own attacks
or you want to do stranger things with
the models
then you begin i think that's when you
begin to need a little bit more of the
specialized knowledge about how machine
learning works what sort of
inputs and outputs you expect from them
but if you take a look you know moving
it back to the
twitter ethics bug bounty a lot of that
was entirely data driven um
all people needed to do was collect data
that showed some sort of disparate
impact or some sort of harm
uh run it through the model to
demonstrate that harm or that
that impact and you could produce um
you know essentially a about right it's
undesired behavior
even undesired behavior in the sense
that the entire pipeline is working
kind of as it's programmed to it just
gave you what you
asked for not what you really wanted so
again
any you know there's entry points at all
levels i really do encourage people to
get into this because this is a really
fascinating space
um but to to to really sort of push the
boundaries of it i don't think there's
much of a way around
uh getting your hands into at least a
little bit of calculus
yeah i like your perspective of like how
should like
red team members know machine learning i
think um
omar this is more like a question for
you you've seen like red team members
kind of grow what advice do you have for
machine learning people who want to
learn red teaming skills
um like is that required for in your
perspective
for building like a ai pen test team
yeah that's an it's an interesting
question because let me define the red
team
part of it um so most people whenever
they mention about
what they call red teaming is you know
infiltrating a building
trying to impersonate somebody and then
launching some exploits to attack
a system so it's a it's a bigger scope
pen testing right
um now what bruce mentioned about
people not even paying attention to
security from the beginning
when designing things that's the number
one thing that i'm gonna start
highlighting right
how you fix this is actually not by
doing a pen test
or a red team engagement after the fact
you fix it
you know what the the boss worries you
know moving security to the left right
at that moment that you're trying to
design these systems
try to think about the adversarial
methodologies right
that will apply right whether it's to
data as we were talking about before
and actually rich brought an amazing
point related to
ethics and i think i need to touch on
that what you're also going to see
is that a lot of people will concentrate
okay i know security because i know how
to do this fuzzing techniques or i know
how to do this type of traditional
you know penetration testing you know
typing
activities but in order to solve this
problem
you also have to think about if i'm an
attacker and i'm able to manipulate
this type of behavior that then has a
collateral damage to ethics
like we're talking about pixelation and
manipulating of images
what if i actually can do that and then
affect the background and the foreground
of an image and then
have face recognition not work for
certain people right
yes things like that just thinking
outside of
the traditional confinements of a pen
test
and then truly understanding one how the
technology is going to be used
uh second how can it be abused and the
third one what are gonna be the evasion
techniques from an attacker perspective
that then you know is trying to cover
the tracks how can i detect it
what happens if this is actually
compromised because it's not so much
about protecting you're never gonna be
able to protect
everything 100 so it's also how to react
what is going to be the way one thing
that i'm actually publicly
discussing with many other entities is
the disclosure of vulnerabilities right
how are you also going to be disclosing
vulnerabilities in an ai system
it has been manipulated especially
something that can be
potentially you know affecting many
vendors many
implementations many you know software
out there so
so that's something to also keep in mind
is not only about the
the cool pen testing or red teaming
methodologies it's also
the whole ecosystem the design the
adversarial techniques and the response
i i really like that point because you
know
i want to get to the vulnerability
disclosure in just a moment
um anita like some of the tools that
rich mentioned like to construct and
clever hands
actually has its roots in academia like
um
and there seems to be considerable work
happening in academia in this space
especially when it comes to tooling
uh why is that
i think that uh it's easy to come up
with these cute tools
you mean you get a grant do you have a
study and
you come up with these cute tools and
then somebody comes up with a better
tool
and it's like this cat and mouse game it
escalates and then the next conference
they're saying i got a better tool
um i don't think that's the best way to
do it i truly and this is like
just like with regular security i think
truly like it's not just a buzzword we
need to think about multi-disciplinary
teams
i mean you know i'm not a physicist but
i work with physicists
on the whole end-to-end pipeline when
they're trying to do ai for physics now
with the large hadron collider i don't
know anything about physics but i do
know
what could happen if someone walks in
their unprotected data center with a usb
drive
so i think like you know having more
humanities technical
and people who are not uh just machine
learning math experts
um is really important but ac but to get
back to your question you you get tenure
based on your cute paper and your cute
tool
um not based on not based on not harming
people
yeah well um bruce should people like
think about releasing these cute tools
like you know when people aren't even
thinking about securing machine learning
systems
or should they just hold back a little
bit um you know
and also systems are even perfected like
you know
do you think this future can actually
cause more harm you know so
like we've been doing this at defcon for
a bunch of decades and and we know by
now
that releasing the tools doing the
research
making it public improves systems so the
ml people might not like it but the car
people didn't like it and microsoft
didn't like it in the 90s and no one
likes it
but this is how we improve security if
we if we don't
do the work we don't release the tools
then the lousy stuff just stays in
production
and this this is the lesson that we can
teach everybody else
instead of we in the def con community
because we know this
we've been known it for decades um
kind of like people that tried a little
bit first like
you could have like you know mentioned
even like a little you know
uh for a couple of minutes ago like how
these systems
are kind of left unguarded so for
somebody in the machine learning space
who's just like you know wrapping their
head around security can you give an
example of how
releasing a security tool actually led
to
improving the net security posture like
mlml researchers might think
oh the release is full i'm empowering
attackers why is that a bad
why is that a bad thing so so
it's a common belief that that you're
giving attackers ideas you're giving
attackers tools attackers don't need
ideas
attackers have tools right who doesn't
have the tools are the defenders
are the non-security people doing the
designing those are the most ignorant
you know in this entire uh system
so that is and also it spurs companies
to action
microsoft took uh security seriously
because the community kept pushing them
automobiles are like what 10 15 years
later doing the same thing
and here's ml systems again behind so
we in security see this cycle again and
again and again
and it is only by doing the research
in public with disclosure
that you actually get improvement and
and
you'll see that here in ml systems this
is going to be the way it's going to
work because because
and and we know it it's almost like
bruce especially for you know for you
and the rest of the veterans in this
like
chat this is not new you perhaps you all
have seen this like multiple times you
see that
omar shaking your side of bruce shaking
his head and rich smiling
listen what chris said right she might
not know the domain
but she knows security turns out that
security knowledge is important and it
transfers and it doesn't matter if the
computer is attached
to a car or a refrigerator or a phone
it's a computer and we know how to
attack
and secure computers it's uh software
all the way down
uh chris i want to kind of like uh
you know bring bring this question to
you uh
what kind of tools like we've been
speaking a little bit about tooling
because we want to talk about brass
tacks of preparing for
red teaming what kind of tools
do you use as part of your ai red
teaming
effort at nvidia like um
can you just talk a little bit about
that sure
um that's a great question and there was
a bunch of really awesome topics that
were presented like literally right
before you asked me this question so it
it's an awesome segue um so there's been
talks about like
what does it mean to be a red team for
ai omr
brought up and bruce was talking about
how releasing tools and things like that
actually emboldens the security
community and makes things better
so there's there's three main tools um
that we have started to use one is
counterfeit that microsoft has released
uh that is and i wholeheartedly agree
that that would
that basically is metasploit for
attacking ai systems
and that would be attacking an ai system
after the model has been deployed
so that's you know one of the phases of
attacking it
there's a second tool that i think what
is great that we
as nvidia released uh to bring the
security community into
the fold you know if people don't need
math like you need to talk
talked about like uh
you don't need math to attack certain
aspects
um and that tool is called mint nv and
that is a docker container that you pull
down
um it's a boot to root you attack it
the initial access is you have to
circumvent a deployed ai model
you can use counterfeit with it uh but
that way you can
security people can go they can pull
this down they can use the two tools
and they can see what it looks like to
actually attack like one of the phases
of an ai system
um and the other the third tool is not
really a tool unfortunately it's a
it's in this publication it's nist
ir8269
episode real machine learning um
get a big old pot of coffee and just
read it and
you'll only really have to read it once
to to kind of digest it
but as you're going through the taxon
taxonomy and terminology of like
adversarial machine learning
you'll start to get an idea of like
where you can attack these things and
what tools would be applicable
for each of these different phases um
and the last thing that i would like to
mention about tooling is to think about
a concept
of what tools you would need to red team
an ai system like omar
saying what does that mean i've started
to come to the turn
that when you attack an ai this is the
first case where you're actually
socially engineering a technology
so build your tools around how do i
trick
how do i socially engineer this
technology
hey um i know you mentioned um
especially
rich and chris some tools so
can you tweet it out from your handle at
our
harangue and at ice bear friend after
the end of this live panel so people can
have pointers
for these tools that will be very
helpful of course
yeah thank you um you know i kind of
like
uh now i want to switch gears a little
bit and talk about
the future of like this field
and bruce i'm going to come back to you
uh
and i want you to kind of like paint
this picture for us
based on your experience in this field
when do you think we'll have the next
stuxnet for ai systems
like when is what are we going to have
like ept uses like hacks machine
learning system
and our estate vendors are going to like
sell us on the solution
when is that going to happen so
interesting so you you pick stuxnet
stuxtet is it was a
very very targeted hack that was not a
a general you know general hack that
that affected
hundreds or thousands of systems that
was against one particular
iranian nuclear plant and we'll have
that when a government decides that that
is an
efficacious way of advancing uh their
foreign policy
and it could be tomorrow and it could be
years from now
and we're going to see criminal hacks
when they become
profitable i mean this follows the
trajectory
of these systems being deployed as they
are deployed
in more places you will see them used so
again back to the apple and image
classifying and looking for
child abuse material on your phone
so you know we could expect to see hacks
that will attempt to frame somebody
we will see attempt we're going to see
hacks that will
bypass the system and and we'll we'll
see hacks that just you know cause
general mayhem
i don't know i mean i think we're going
to see the first papers on this in
in a year or two and this would go
down to the criminals and governments
you know when it uh when it does and all
this stuff flows downhill
thomas nsa program is tomorrow's phd
thesis and the next day's hacker tool
it's hard to know when
but we know it's coming and it's driven
by
how they're used
[Music]
and and what do you mean by how you're
used like we already see like machine
learning systems powering like
healthcare finance and all these like
important like fields and we all agree
they're unguarded
why aren't you seeing attacks on machine
learning systems more than
i'm not convinced we're not i mean the
question is whether they make the news
right so so you know right a proxy for
whether things occur or whether we know
about them and
that's an imperfect proxy i think
already
uh doctors know how to code
patient information in order to get the
ais at the insurance companies to
produce the outcomes they want
to approve the procedure i mean so i
think there is
that kind of adversarial ml
going on right now uh i think image
classifiers i mean the stuff that kills
people
you tend not to see that because a lot
of people don't want to kill people
that just you know is annoying right the
uh the way
uh microsoft's pay was turned into a
misogynistic nazi in 48 hours by 4chan
right that
was an adversarial ml attack
so we do see them i don't think they're
making the news because they're still
under the radar and the systems aren't
as widely deployed and understood
i feel that's a that's an interesting
point i think like chris
i'm bringing it back again because i
think like for
clearly like all these like flagship
companies
that you know that you know microsoft
nvidia
ibm twitter google they're all putting
like machine learning systems
front and center of like you know
their their competitive like advantage
so all of these teams tend to have red
teams right
like nvidia i'm sure had like a vanilla
red team before this
so based on what you're seeing like how
can they
invest in the space like what are some
of the organizational
like challenges and opportunities at
hand
uh no it's a great point um and i've
heard a lot of really good
feedback from the panelists so far and i
think anita nailed it she said just
getting in the room
um that is going to be like the best
entry point for any red team if you have
a red team you have
if you're going to be using ai systems
or models or anything like that just get
them in the room
get them at the table it's not as
technical up front
but they know how to attack things they
know
types of attacks they have adversarial
mindsets
just having them in the room and being
able to ask questions to scientists or
policy makers will change the trajectory
of whatever ai or ml system is being
created and if it's being
if the model is being created you've now
you've now
shifted how that model was going to be
secured people are going to start asking
questions
um so to for entry points for sit for
places that are using
ai technology or anything like that or
making their own models
i would say do table tops first and then
the
the next phase after that after you can
start affecting policy would be
try to start doing some of those more
later stage ai attacks because that's
more
aligned with what i would consider
traditional pin testing you know you're
going to be bleeding apis you'll be
pulling data stuff like that
and as more familiarity gets
accomplished with the red team you can
start moving into
attacking the model poisoning and as
it's being created
you know replicating the model um how to
attack those things
offline attacks and you can just get
more uh deeper with it
oh sorry chris i didn't mean to like
interrupt you but when you meant policy
um what were you meaning governmental
policy or like
like nvidia's like policy like can you
just like clarify that
sure yeah i i meant policy like that
that stakeholders
would implement like if we're going to
use an ml system
we're going to have to do this you know
i think rich talked about the bias
that stuff is like i mean that's
incredible once you start reading about
those things and
there has to be policy around you can't
just take a
ai system and just use it like there's
there's going to be inherent things
you need to account for
and being in the room and saying hey i
think
we need to account for this type of
attack
just being there does wonders like it
you went from zero percent to 100
so like when i when i met policy i meant
like stakeholders internally how are you
going to use the models
how they're going to be implemented and
things like that
that is awesome chris like i want to
just pull that through just a little bit
more
um or somebody who's just like listening
to this like panel
and they're super pumped they want to go
think about like you know think about
like say even talking to stakeholders
what are like
two or three questions that they can ask
an ml developer as a security person
just get the ball rolling anything on
top of your line
yeah where'd you get the model from like
work like is that our date or is it
somebody else's
so it'd be like where'd you get the
model do we trust them is the data that
goes that like the model was used
against is it ours or somebody else's
uh you start asking those questions and
then you can start figuring out like
where it come from
uh i need to talk about pipelines
attacks well like sometimes the pipeline
is
like the model doesn't come from you it
comes from somebody else
so if that would if the model was
compromised way upstream
by the time it gets to you you may not
even know it's compromised there may be
a backdoor sitting in it i don't know
this but like
there there may be a backdoor sitting in
it that has made it through like five
different channels that nobody knows
about
um so those would be my two biggest
questions it's like where the model come
from
where did the data come from i i want to
jump in on that just to emphasize data
data and data and where are you hosting
you know
something like a simple you know s3
bucket that's
probably a likely more likely vector
where is the data kept
how long are you keeping all these
boring security compliance things
which we go through and people don't
like to go through it where's it backed
up
who has it when are you gonna
de-accession who has access to it
those fundamentals that we know in
security
first thing i'd ask that's awesome like
i see
as i listen to what you're saying anita
and you know as i'm processing like
chris's information
like i'm and this is omar i want i
really want to get your thought here the
questions that chris and anita are
asking
for for you know in some levels you must
have been asking from like basic
software
for like you know like you know for a
long time so using the
questions that chris and anita kind of
like you know
just post how do you think the role of a
traditional rec team member
is going to change in five years you
know with ml systems
um can you just talk a little bit about
that
yeah if you if you go back in time at
least
i guess i'm getting old but whenever we
started
with you know router switches you know
embedded devices a lot of people didn't
know about that it was just you know
this taboo thing they concentrated in
end host machines and probably windows
we saw a big gap
in in the talent right so on how to
to look and manipulate those type of
systems in some cases because it was
cost prohibited by some people right
putting a two million dollar you know
core rather the internet you know that
was it was pretty pretty challenging
but um but we evolved right we looked
into not only
the red team part but also the forensics
there's a huge lack
feel to this day of really good people
that knows
how to do forensics in these type of
systems
now fast forward now with ml and ai
that's something that we have to think
through right what are the
the skills that are necessary not only
from the red team perspective i'm
touching that in a second
but also in other areas of security to
protect these systems right
again going back what if i'm already
compromised what are the things that i'm
going to be doing
from this ml system uh to see if
again you know for the attacker is
actually compromising this you know
a year from you know ago and then
same thing that you're gonna see you're
not gonna see the perfect
candidate or the perfect person that
will know all ai and old red teams right
um what chris mentioned about being on
the table that's number one
right he's having those conversations
because it's an evolving thing right
second
you don't is specializations within you
know this red teaming
pen testing offensive security i'm going
to generalize it of censor security just
like
i have probably more uh knowledge and
web applications you know and somebody
else in embedded devices and somebody
else in you know in other type of
technologies you will
traditionally you will see that right
and same thing goes with
quantum computing right we haven't even
got the kind of worms
right now another thing that you're
going to be seeing
is ai or nml is going to augment
the task of a red team error slash
offensive security person
right so in that case it's not so much
of attacking those things it's how can
you use those
systems to then do all types of attacks
right
what are the types of obfuscation other
types of evasion techniques
other types of manipulation right that
those notices today are probably
not a scale right so it's the augmenting
of the red teamer
or the offensive security type person
using these type of systems for some
other manipulation
i mean omar that was such a
comprehensive answer
and especially if you're interested uh
you know um
if you're listening to this panel if
you're interested in a lot last part
of how like uh red team can be like
automated you should really listen to
bruce's keynote
uh that's gonna be that's hosted in the
ai village right now it's
it really does open your mind to ai
hackers so thank you for bringing that
point omar
feels like we're tying down a lot of
like um tying down a lot of different
points
but rich i also want to kind of like ask
you this question you're you're an ml
you've been an ml engineer since
even the title didn't exist so how do
you see
like you know the role of analogy are
changing in the next five years
then think about actively securing these
machine learning systems
um i think we have sort of the good path
and the bad path
uh so tell us both the paths
okay so so the bad path is we
do nothing and what we end up with
is these systems which are deployed
everywhere we don't have a standard way
of understanding what the
vulnerabilities are in them or how to
address
them or what the best practices are we
haven't invested sufficiently in secure
tooling for them
and as a result it's the wild west like
you know the internet was i don't know
20 years ago and it's just it's a
complete disaster
um i think the good path is
we form these sort of multi-skilled
multi-disciplinary teams who can think
about these things holistically
right and there's you know the just back
to the data thing right there's a whole
issue there with like forensics and data
security and data privacy if you have
pii that's going for personally
identifiable information
that's going into those models that in
some jurisdictions is controlled
that's actually telemetry that you would
need to do forensics on the model so how
can you handle that securely right
you've got all of these
questions that we've we've barely even
started
to tackle that we need to think about to
be able to
essentially secure these models and
defend them properly so i think the good
path is essentially the
inverse of that we spent time
thinking about these things um you know
with the health perhaps of red teamers
we've identified
these are the classes of attacks that
are feasible these are how they usually
happen
in production systems and not just
in sort of academic settings
and this is how we can then go about
defending against them and we're
definitely seeing moves i think
towards the good path um we've got
nist is seeking input on how to do like
trustworthy and reliable
ai computing uh we have attack
frameworks i think um the
the atlas framework uh from microsoft
and mitre i believe
um so we're starting to sort of
systematize and categorize these
uh what we don't have at the moment is
again this transition from the academic
space where we have
all of these theoretical attacks and all
of these theoretical vulnerabilities
that are very specific to machine
learning models
um that haven't made the jump into oh
yeah that's actually happening all the
time now i mean
we have as far as i know we still today
since 2019 have exactly one cve
that has been filed in relationship to a
machine learning model and that was um
will and nick with the cbe against
fruitcoin and again that's a good
example because
that required both a weird configuration
in how the mail bounce was handled and
the leaking of the scores for that
attack to be effective so again it's the
entire system
that you have to consider and so those
kinds of things right in five years
maybe we'll be like oh yeah obviously we
should write
obviously we shouldn't make raw scores
obviously
we should you know um do differential
privacy on the inputs
stuff like this hopefully it'll just be
like
routine part of ml engineering the same
way that software engineers these days
think about very routine security tasks
that for a long time
nobody bothered with because we didn't
have sort of the framework to think
about them in a systematic manner
yeah i i really like that point about
not having frameworks and
especially like how contrasting it with
like how
chris mentioned um the mist framework
you know for like adversarial ml so it
feels like there's some piece of the
puzzle but there's still like
integration is still missing is that a
fair statement uh
uh um rich
yeah i think so i think i think there's
two pieces
that are missing so the first piece is
we sort of need to pull these different
threads together
sort of within the security community
insofar as such a thing exists
have an agreement that yeah this is how
we should think about vulnerabilities to
machine learning research
uh into machine in machine learning
models i think the second thing we need
to do though
is actually focus on how do we
transition these
theoretical attacks we have an
unbelievable number of papers about all
kinds of different attacks
against machine learning models in the
academic space so
in some sense it's kind of scary because
we know that there are these potential
vulnerabilities
against machine learning models um we
could be leaking data
we could be you know
having models stolen all the time and
and it's sort of the question that bruce
brought up
is it that it's happening and we don't
know about it or
is it just those they haven't made the
jump yeah and so i think what we really
need
to see is more collaboration
sort of industry academic to make these
transitions happen
so they can be like okay yeah that's
that's realistic we really do need to
worry about that we need to think about
how to protect against that this other
thing maybe not so much it's a cute
trick but it doesn't actually fly
in reality yeah i feel like i
one of anita's point is really sticking
to me is like how people
they get funding and they'd write cute
papers and cute tools
so that's gonna like you know be a big
takeaway for me anita
um i want to like um i think there's a
good closing question from at mammon
on twitter uh this is for i think it's a
great question for everybody in the
panel just like
uh a quick uh if you can just take 30
seconds and point to one
a piece of information that people you
think should know about air
teaming um i know it's pretty like
you know pretty generic but there's one
key takeaway that you want people to
take away from
you know ai red teaming what would it be
i need app do you want to start us
do you want to start start us off
i guess the key takeaway for me would be
intended use and that
ai red teaming is not in my opinion it's
not a technical exercise
um that's a great like you know way to
throw the ball at chris
because you are doing technical
exercises
well and that is a great ball um because
my takeaway would be
you don't have to be a data scientist to
attack it
and we should like pass the ball to the
data scientist now
rich well chris stole what i was going
to say which was
that the field is the field is wide open
um i think
maybe i would just expand about on that
a little bit and i think like i've maybe
been
harving on a bit too much we don't know
what we don't know yet
um the field really is wide open
i think tools are making it more
accessible uh there's
so much free content out there to just
get enough of an idea about how machine
learning kind of works to get a sense
for it what the inputs and outputs look
like
i really would encourage people to you
know grab
the docker image that chris was talking
about or take a look at what was posted
for the twitter ethics bug bounty
or take a look at some of these other ml
problems and see where you can dive in
and see what you can find because
there's there's a lot that's still out
there to discover so please get involved
and rich can i ask you to like tweet
this out from my twitter handle
at harang go people
omar how do what does your key take away
i like the controversy between the
non-technical versus technical so
so i'm gonna be between the two i think
that is a absolutely as a combination of
what i what i think that is paramount
instead of giving you just one resource
is a call to action right is because a
lot of the things that we
talked about before even bruce mentioned
some of these conversations are not
taking place
and whenever you're creating technology
right and so on and it's as rich
mention it's an evolving thing right so
it's up to us
in this panel and in this community
right to to come up with resources that
the you know the newcomers actually are
will
will take advantage of and i think
i'm really curious you know to actually
collect all the different feedback
all the tools that chris mentioned all
the research that anita mentioned and
everything and probably putting together
probably a github repository you know
tweet it out i mean whatever we want to
do but at least
start incorporating that you know i
guess it's a call to action versus
giving you the perfect solution yeah
thank you uh omar and bruce bring us
home
like uh tell us tell us from uh what we
need to know about this field
oh bruce uh this is this is i love it
it's 20 21
yes
i know i just feel it and it feels like
march 2020 all over again
what anita said that uh right think of
the big picture this is
do your threat modeling okay this is not
just technical this is
this is the entire system it's a
socio-technical system and the better
you can threat model the better you are
on both sides of this
fantastic um thank you bruce i you know
first of all big thank you to all of you
on the panel today i know how busy
it was according to accountants i really
appreciate all of you coming here
if you are listening to the panel i
strongly encourage you to go check out
ai village's discord channel they have
an entire channel on attacking ai
systems
and you know you can you can poke anita
and rich there and if you hang out at
the red team village now we've got omar
and chris also kind of guiding you there
so please make use of these resources
and a rich and chris will be tweeting
out some of these resources
uh so you should also go look into that
so with that thank you very much i
really appreciate all of your time today
thank you ron