hello
and welcome to my talk remote
adversarial phantom attacks
against tesla mobility
now i will start by introducing myself
my name is ben
i'm a computer scientist and a former
google employee
i'm a phd student at bengal university
of the negev
and a researcher at cyber at bgu my
research focuses our privacy
which includes side channel attacks
cover channel attacks and tempest
attacks
and i also focus on security of iot
devices
mainly on drones and advanced driver
assistance systems
now you can read more about my research
at my website
um this talk
is based on uh three papers that we
published during the last
two years um one of which was accepted
to acm
ccs 2020 the other two were accepted to
autosec autosec21 and you can find them
online now this is the agenda for today
we will start by discussing adversarial
ai attacks
we will then discuss about object
detectors and we will move on to discuss
about split-second phantom attacks
finally we'll see demonstrations of the
attack and we'll end with some
interesting conclusions
about the attack
okay so let's start to discuss about
adversarial ai attacks
and in our context adversarial ai
attacks
are physical objects that are perceived
differently
by ai than by humans and
it was already demonstrated by previous
studies
that object detector
misclassified a detected object
if attackers add physical artifact
to the object now for example the
physical object can be stickers it can
be gravity
it can be even printed blood colors and
all of which
caused object detector to misclassify
the road signs
for example the stop sign as for example
a speed limit
or to misclassify this specific speed
limit as another type of
speed limit and this was all done again
by adding some physical architect
either stickers gravity or even blood
columns
now the area of adversarial ai
is actually exists for i would say
about six seven years something like
this and many works have been published
in this area
and i would say that on a monthly basis
we continuously hear about new type of
adversarial ai attacks
that's demonstrated by uh
guys from academics and an immediate
question that arises
is why haven't adversarial attacks
been observed in the wild given that we
are continuously here about them in the
last few years
now um the several ai attacks
have many disadvantages and and they are
the reasons why
um these attacks were demonstrated in
the wild
for example one major disadvantage is
the fact that
these types of attacks require the
attacker to approach their toxin
now from an attacker's perspective this
is actually something that
he would like he or she would like to
avoid because they would like to stay
anonymous
and not approach that vaccine
another disadvantage is the fact that
some of the attacks can only be applied
by skilled attackers some of the attacks
are required to train an adversarial
network the adversarial neural network
in order to
create or to produce the adversarial
instance
and bear in mind that the set of skills
that require
to create or to train the adversarial
network to produce
under several instances is somehow very
different from the set of skills that
the average or the standard
hacker basically holds
also some of the attacks demand full
knowledge
of the attacked model in our language
they called the white box
white box attacks uh they require
the attackers to extract the algorithm
that is used by the ai in order to
attack it
and as a result they have somehow very
i would say tough preliminary stage in
order to start
at the beginning to start
the attack
also some of the attacks leave forensic
evidence at the attack scene now bear in
mind that
maybe the gravity the way the gravity
was painted
or the stickers or the way that the
sticker was printed uh may
even uh help investigators
find uh that tackle so they have some
disadvantage in terms of the forensic
evidence that is left
in the scene and also some of the
attacks require complicated preparation
bear in mind that uh training a neural
network
is what in my definition is a
complicated preparation because it takes
a lot of time
it may takes a lot of resources in order
to do so
and this is something that may be a
limitation for some of the attackers
okay so we understand what disadvantages
uh adversarial ai attacks have
let's try to understand how object
detectors
which are the target of the adversarial
ai attacks
work so first of all object detectors
are algorithms which are used to
identify the location
of instances of objects of a certain
class
now for example this can be pedestrians
these can be road signs
this can be even and
object detectors receive frames as
input and output a set of bounding boxes
with the classifications of associated
objects
now here you can see for example in the
left side
uh a pedestrian detector you can see a
bounding box
across the pedestrian in the middle you
can see
a car detector you can see how each and
every
car is actually uh have
bounding box around it and also
in the right side you can see roadside
detector which you can see how each and
every old sign have
a bounding box across it and this is
what
object detectors intended to do
now object detection starts with
a video stream basically
frames and each and every frame is being
received as an input to the object
detector
now the object detector
um now the object detector actually find
the boundaries
of objects uh basically boundaries of
objects are detected
and suppressed and when i mean surface
it means that
uh duplications uh being identical uh
of the same of the objects of the same
object that are being identified
multiple times
are suppressed to a single detection and
at the end of this stage we will have
this
nice picture basically you can see how
four pedestrians
are being detected by the object
detector however
there is also an additional component
which
i call a persistency function a
persistency function
is actually an interesting component
that is being added to real-time
object detectors and a persistency
function
mainly used to decrease false positive
detections
by verifying the existence of a detected
object
in a few consecutive frames which
corresponds to
an appearance pressure now a resistance
function works as follows
it receives an object and accepts the
objects
if the appearance duration of the object
is greater
than a preliminary predefined
appearance threshold otherwise it
rejects
the objects and consider it as a false
positive
now a persistency function have one
constraint
because you can understand that by using
a higher appearance threshold
it will result in a lower false positive
rate
and to the object detectors it means
that an object detector with
a higher attendance threshold will yield
a lower false positive rate which is
good for the overall system
however in terms of uh
real-time driving we have an additional
constraint
which requires that objects for example
the objects can be pedestrian
need to be detected as quickly as
possible
so the car can immediately react to the
detected object
so in on the one on in
on one hand you have a constraint which
required which you want to
um basically set a higher appearance
threshold
and on the other hand you have a
constraint which
requires you or you want where you want
to
set a lower appearance threshold and as
a result you have a trade-off point
where the appearance threshold is set by
the manufacturer
uh by taking both constraints into
account
now the appearance threshold must
decrease the false positive rate
basically it needs to be the sweet spot
that decreases
the false positive rate but also
should be short enough to detect objects
and in our case it should be
milliseconds for
driving cars in order to detect the
object and in order for the car to react
to them as quickly as possible
now as a result the appearance threshold
actually
creates very interesting uh i would say
reality where objects only objects that
appear
for a period of time greater than the
appearance threshold
are considered by the persistency
function
in objects otherwise they are completely
undetected to the
to the persistency function and to the
object detector as well
so here is a question which will uh
introduce us
the split second the motivation for the
speed second phantom attack
can we apply a split second phantom
attack
that appears in the detected area
but it is still imperceptible to the
human eye
and this is what actually takes us to
the next section which is the split
second phantom attacks
now let's try first of all to understand
what split second phantom attacks are
and let's discuss about the threat
now speed second phantom attacks are
basically attacks
where the attacker presents a phantom
object
to a car for a split second and
this object actually triggers an
undesired reaction from the cow
and this can be done by attackers
by for example hacking a digital
connected billboard
and embedding a phantom object into
an existing advertisement um
this is defcon and basically i think
that some of you are familiar with the
attacks that were
already demonstrated in defcon um
it was already demonstrated in defcon
how
uh easy this type of billboards
uh can be uh hacked um
you can find them there's an interesting
lecture uh talk
a few years ago that uh was demonstrated
uh
by uh that was demonstrated by a hacker
that uh
investigated what exactly need to be
done in order to uh for example
attack an internet connected the digital
billboard however there is another way
to apply the attack
this is by projecting a phantom object
from a drone
okay in both cases the phantom is uh
appeals only for a split second which
should be somewhere in the undetected
area
for the human but detected for the car
now let's try to understand the
significance of split-second phantom
attacks with respect
to uh previous adversarial attacks
now as opposed to the adversarial
attacks that require the attacker to
approach
the attack scene split-second phantom
attacks can be applied remotely either
by hacking a digital billboard or using
it wrong
also as opposed to the previous
adversarial attacks that rely on skilled
attackers which
maybe even data science with data
science capabilities
split second phantom do not require any
spatial expertise
as i will show you in a few slides
also as i mentioned before some of the
other serial attacks
uh are white box uh attacks which means
that they require the algorithm
in order to apply the attack the the
original algorithm that is being
attacked
uh split second phantom attacks on the
other hand do not rely on
white box approach they completely wear
black box
as opposed to the previous of the serial
attacks split second phantom attacks
do not leave any forensic evidence at
their taxi
at all they do not use any stickers
or gravity or things like this that can
help investigate us to find or to locate
the
attacker also as opposed to the previous
adversarial attacks that require
complicated preparation
split second phantom attacks do not
require any complex preparation at
all again i will uh discuss about it in
a few slides
and also as opposed to the previous
adversarial attack that exploits
the space domain split second phantom
attacks exploit the time domain
and this is somehow very interesting
property that they exploit
now um some of you probably wonder why
does a commercial advanced driving
assistance system
detect a phantom as a real object and
here is an experiment that we did
we took a picture of a road sign and
projected uh
the picture on the tree
and here is an interesting fact mobile
630
considers the projected road sign a real
speed limit
and this is despite the fact this is
completely uh
um as you can see project a projection
that
basically was projected on a tree
now some of you probably are not that
surprised by this result
because mobilize 640 consists only of a
video camera
so doesn't have any other type of sensor
which can be used to maybe identify an
object however here is another
interesting experiment that we did
we took the picture of mr elon musk and
projected it on a road and we placed
tesla in front of this uh as you can see
uh projection of a pedestrian
and here is another interesting fact
tesla model x
considers the projected pedestrian a
real pedestrian
despite the fact that the skull is
equipped
with radar and ultrasonic sensors now
bear in mind radar and ultrasonic
stencils
are sensors that intended to detect
death
and this is a projection completely
deathless
which is being identified by a system
that have the needed or the required
sensors in order to know this type of
projection
now some of you maybe again i'm not that
surprised because
um i would say that
pedestrians have i would say lower uh
cost radius uh section or radio
signature
than for example cars and as a result
a system which basically consists of
video camera a set of
ultrasonic sensors any radar may follow
the video camera instead of relying on
the
radar or ultrasonic sensor for the case
of pedestrian
detection however when it comes to
for example cars uh the case is somehow
very different because cars
have i would say very strong radio
signature that can
be used to identify them and here is
another experiment that we did
we actually took a picture of a car
as you can see here projected it again
on a road
and as you can see tesla model x
considered the projected car a real car
again despite the effect
is completely deathless and
the system which is tesla model x uh
consists of the needed sensors
to detect this type of object
uh which have uh much stronger
uh radio uh value cross
signature than for example because
signature of the pedestrian
so again despite the fact this is
completely deathless
and the tesla consists of the sensor
that can
for example uh be used in order to
cross-correlate the data that is
obtained by the video camera
with the data is obtained from the sense
from uh
from the radar from ultrasonic sensor
despite respect
it is still being uh considered this
uh projected car a real car and this is
somehow a very interesting fact
i think that you will agree with me that
there is a perceptual challenge
in this case there is a difference
between
what an advanced driving assistance
system thinks it is
and what is actually there and this is
actually demonstrated
in this three pictures that you can see
in this line
now um i i uh
do not consider phantoms as bugs
uh they are not the result of poor
security implementation that can be
attached with the
writing the most secured code now for
example an sql injection is something
that can be prevented or mitigated by
uh using uh i would say by programming
the code with security in mind how
this is not the case for phantoms
phantoms exploit the fundamental
inability
of object detectors to distinguish
between a real
and fake object and as a result
we consider phantoms a scientific gap
so why are phantoms a scientific gap
so uh object detectors
are basically the output of a long
training process
uh previous studies have already
discussed the implications
of using the training of a training
process to create
artificial intelligence they discussed
about
the high level of computing capabilities
that
are needed in order to train or to
create
an artificial intelligence they also
discuss about the energy consumption
the high energy consumption that the
training process
costs they already they also discussed
about the high cost
uh that uh required to uh train
in to train a model and artificial
intelligence to create it
finally they also discussed about the
long training time that it takes to
train
a model however it seems
that the training process despite all of
these disadvantages
are able to create robust ai models
that even function better than humans in
real life
now you can see this is a by the way
taken from about two years ago
where tesla tesla claimed that its
autopilot
is almost nine times
safer than the average driving so it
seems that the training process
despite the disadvantages create robust
ai models
however we try to ask a different type
of question
what object detectors don't live and as
you can see from the following
pictures is isn't taken into account by
object detectors
you can see how unrealistic objects
despite the fact that they are
completely unrealistic are being
identified as real objects you can also
see how
color in some cases isn't taken into
account
this entire road sign is made from gray
used
and uh is being identified by one of the
state-of-the-art
object detectors which is again
found online also as you can see
in our case texture uh basically either
whether it's a transparent or skilled
object
isn't taken into account and as a result
transparent and skewed objects
are being identified as a real object
despite the fact that that completely
phantoms
and we would like to mention that object
detectors are essentially feature
matures
they lack the ability of humans to
ignore fake
objects
so let's try to understand how this
effect can actually be exploited by
attackers
and the first thing that need to be
determined by
attackers is the appearance pressure now
in order to basically determine what is
the appearance threshold that is needed
to
project or to uh basically
um present fenton
we perform an experiment a black box
type of experiment where we place
um tesla with hardware 3
and mobilize 630 in front of
uh of the wall and we use a projector
to project on the wall
and for tesla we projected the stop sign
that you can see in the left side and
for mobilei we projected this
nice speed limit and we actually
projected
this type of uh phantoms for various
durations
and we tested how the system actually
identified them the success rate in
terms of identification or
detection of this uh speed limit of
these several signs
now we found that mobilize 630 detects
a phantom that appears for i would say
above
at least 125 milliseconds
for 100 of the time it means that if you
project a phantom for 100
for 125 milliseconds it will be
identified by mobilize 630.
now the case for tesla is a bit
different tesla for example
detects a phantom for 100 of the time
if it appears for at least 460
milliseconds
and you can see that results in the
graph
on the right side
now with that in mind you can actually
uh
use a drone by mounting the projector to
a drone
and fly drone as you can see in this
video demonstration
and to project for example in this case
uh
a road sign on a structure
that is located nearby a road
and caused this specific uh uh
car which equipped with mobilize 630
to consider this road sign is railroad
sign
despite the fact it is completely
a phantom this is not what this is not
the speed limit that uh
allowed in this specific code it was by
the way
uh was taken in our university which in
this case
you can uh drive only uh up to 30
kilometers per hour you can see how
this is identified by the specific cow
as 90 kilometers per hour
okay now here is another interesting
experiment that we did
we took a picture of a pedestrian and
projected it
uh on a road and we engaged
uh tesla in cruise control and the
cruise control was set to 18 miles per
hour
and you can see how tesla approaches the
projected pedestrian
detects it and immediately reduces
the speed of the car from 18 miles per
hour to 40 mice per hour
as a result of the detected pedestrian
and again
attackers can as a result trigger an
undesired reaction
from a driving car
now however in terms of uh
this specific threat model whether you
need to use a drone in order to project
a phantom
on the road this is what i call a
semi-remote
type of attack uh because attackers
still need to be within the flying range
of the drone
now bear in mind that most commercial
drones um
the limit is somewhere between seven and
eight kilometers
so it means that ideally attackers need
to be within this range
in order to apply the attack against uh
a location that
should be uh located in this specific
range so you can understand this type of
attack
cannot be applied from a rival country
for example because
again it requires somehow the attacker
to be
i would say in the books the
approximation of
up to seven or eight kilometers from uh
taxi
so in order for example for attackers to
apply the attack
from a rival country they can use uh the
second
type of the second flat model which is
uh exploiting
the digital uh internet connected
billboard
and let's discuss about this type of
model now
so um as i mentioned before
this specific uh flat model is more
interesting
and this is because uh digital
billboards are located near
uh roads also they
are somehow easier to attack than you
might
uh think of and
also we will show you that attackers can
actually exploit them in order to apply
this attack over the internet from an
even level company and
in order to do so they can even do it
with uh
instead of projecting the specific uh
uh object the specific phantom
for uh on their uh advertisement they
can actually manipulate
an existing advertisement by embedding
the existing advertisement
phantom inside of it and in order to do
so they can find what i call a dead area
which won't be identified
even by a person that maybe sits inside
the driving car for example in the case
of a semi-autonomous
now here's an algorithm for embedding a
phantom in advertisement
uh basically an advertisement
uh consists of a set of frames and
in each and every frame we compute a
local score
for each and every block in a frame and
the local scope is completed as follows
key points
are extracted from the frame you can see
the key points
on the right side they are marked with
the
with blue and the skull for a block beam
uh in this frame is computed based on
how much a dead area the block is
with respect to the key points extracted
it means that uh for blocks
that are located uh that are more
distant from the
uh this set of
key points they will receive a higher
score because they are considered more
a dead area inside this specific
now however i discussed before about an
appearance threshold that
for example in the case of mobilize it
need to be 125 milliseconds
now this is actually may correspond to a
few consecutive frames
maybe even four frames so we need to
take the
time domain into account and in order to
do so
we compute the global scope for each
block uh
by computing its
score with respect
to the scores in the uh subsequent
consecutive frames
now it means that if you have for
example uh you know that the
mobilia threshold was set to 125
milliseconds which corresponds to
four uh subsequent frames in this
specific advertisement
what you need to do is to compute the
global scope of each and every block
uh with respect to its uh full
consecutive frames
and by doing so you can locate
the best uh block that can be used to
embed a phantom
inside of it now here is just an example
for
an advertisement that we took and we
embedded the uh a stop sign in order to
attack tesla
on it it's a mcdonald's advertisement
that we took from the internet and
just to demonstrate it we um embedded
the
the stop sign for 500 milliseconds in
order to attack a
tesla because tesla have a pressure of
460 milliseconds
okay and here is the demonstration um
in this demonstration you can see how
tesla's autopilot
automatically triggers the car to stop
when
the car is approaching to the uh
presented the advertisement that i
showed you in the
slide before we actually took this
advertisement and presented it
uh on a digital billboard and when the
car
approaches this specific uh digital
billboard it
immediately identifies the stop sign
which was embedded for 500 milliseconds
and as a result you can see it right now
on the dashboard in the left side
it triggers an immediate stop of the car
so a phantom that was added to an
existing
advertisement caused the tesla's
autopilot
to trigger an automatic stop in the
middle of the road
okay so we have a few conclusions that i
want us to discuss
and the first conclusion is that sensor
attacks
are a very dangerous type of attack they
are not easy to counter
and they exploit the gaps between a
human and machine perception
now the commonly used approach to
counter sensor attacks
in the case in where you have
additional sensors is to apply a
multi-sensor
fusion policy or approach however in the
context
of semi or fully autonomous cars this is
a bad solution
and i want us to understand why this is
a bad solution
now first of all in some cases
multisensor fusion is not effective at
all now for example some objects
can only be detected by a single sensor
for example road signs
and lanes can only be detected by a
video camera
as a result
the existence of this specific phantom
cannot be dismissed by applying
multi-sensor fusion
in other cases some areas are covered
only by single sensor
for example you can see how uh this
specific area
uh of the tesla uh
which is uh only covered by a single
sensor
um won't be able to detect
this type of phantom which may appear in
this specific area
because it is only covered by one sensor
so
it cannot be dismissed by applying
multi-sensor fusion
now in some cases multisensor fusion
can be used to detect the existence of
an object
now this is for example the two cases
where multi-sensor fusion
can be used to detect a pedestrian and
can be used to detect
this projected car however
if you were a car manufacturer would you
program the car to ignore a detected
object
if the object was identified only by a
single sensor given that you know
that sensor detection accuracy may vary
at adverse or ambient uh it adverse
ambient or weather conditions
for example a heavy rain now in some
cases
as a result of this adverse ambient
weather conditions
uh some object might be undetected to
one sensor
will still be detected to other now with
that in mind
how would you program the car to resolve
a disagreement
between the radar and the video camera
regarding a detected object
now here is another interesting scenario
sensor detection accuracy varies for
different objects is
a known fact now for example it is known
that video cameras are more accurate
at detecting pedestrians than radars
now in some cases the detection of one
sensor
is more reliable than other sensors
with that in mind how would you program
the car
to resolve a disagreement between the
radar and the video camera
regarding the detected object so
these are basically questions that i ask
you however
the industry takes into account another
type of fact
which is the damage that may be posed by
uh to uh that maybe opposed to
pedestrian
drivers and nearby passengers if a real
object
is ignored due to a mistake
uh that was done by you know configuring
the the car to ignore
a specific uh um object
if it was identified only by a single
sensor
this type of mistake is uh i would say
very risky because
it's also i have a great impact uh it
may cause for an accident it may
cause a loss of life and as a result
it is better to react to a detection as
a real object
if the detection confidence confidence
crosses a predefined threshold
and this is probably the reason why the
tesla in this case uh
detects the pedestrian and the car
israel uh objects
despite the fact that they are
completely deathless and despite the
fact that
they do not have uh any validation
from the data that was obtained by the
radar or from the data that was obtained
by the ultrasonic sensor
now in practice
phantoms are not easy to counter with
multi-sensor fusion this is
one of the most interesting uh
conclusions that i wanted to take from
this
uh talk in addition
um phantoms as you can see in this
pictures
have already appeared in the while you
can see that how physical
uh basically objects cause
uh teslas to uh
consider them as real objects for
example this was taken from this week
where a moon was detected as
a traffic light and well an
advertisement
was detected as pedestrian and
in the right side you can see how a
bugling sign was detected as a stop sign
and again phantoms are not a new
concept they've already appeared in the
wild however
phantom attacks are more dangerous than
this type of phantoms
because they can be applied remotely and
they are not
is easy to counter
so i would like to thank you for
attending this talk um
thank you very much and this is it
you