hello defcon
my name is andrea downing and i'm really
excited and honored to be back
here at the biohacking village for my
third year
um today i'm going to be talking to you
about predictive algorithms and how they
affect your health
and your future the name of my talk is
no aggregation without representation
so let's get started my talk has five
parts
why me why should you listen to me uh
what i'm going to talk about predictive
power of health data harm and threat
models hacking algorithms
and how we think and act differently to
protect patients
i want to start by dedicating my talk to
a friend
and fellow disability rights activist
aaron gilmer
who took her own life not long ago
while she can't be here today to
raise her voice i thought i would start
with her words
she said this in 2014 i come from the
privilege
of being an educated white woman and a
background
in a culture where it's okay to question
and fight for what i want
i know the correct jargon and after so
much time in hospitals
i've have enough knowledge to ask for
and fight for my care
some patients die trying to get food
medicine housing and medical care
if you don't die along the way you
honestly wish you could because it's
also exhausting frustrating and
degrading
i may not speak for all patient
communities when i talk about
predicting your health and your future
and i might
might not represent the entire patient
population or the diversity of it
but my own origin story and path
to becoming a hacker and starting
on this road that got me to defcon
began when i was very young and
predicting my
own future around cancer
i have a lot to say about one gene and
the history of that gene
what it tells us about predicting your
health
i never wanted to be on this path i
first
learned about the power of data to
predict
my own future when i was 25 and i share
this in every talk
i have a bug in my code i have a brca1
mutation
what that means is my genetic counselor
sat me down when i was 25 years old and
told me that i had
up to an 87 chance of developing breast
cancer
and up to a 60 chance of developing
ovarian cancer in my lifetime
that's me getting surgery one of seven
surgeries
up at the top left and my first foray
into data sharing
and disability rights starts at the
bottom right here at the steps of the
supreme court
when i served as one of the spokespeople
a media spokesperson for
a plaintiff in a case that went to the
supreme court on whether human genes
could be patented
the case and and the genes in question
were brca 1 and 2 where i
had a genetic mutation and the company
that had done
my genetic testing had a patent on those
genes
before that case over 40 percent of the
human genome had been patented
and after that case the access to
genetic testing
and genomic text technologies exploded
and we really are still only at the tip
of the iceberg when it comes to
understanding the power of genomics
my path took an interesting turn
in 2018 after cambridge analytica
where i became an accidental hacker
asking myself
a simple question about the privacy
implications of having
a support group of genetic mutation
carriers on facebook
that's a whole other talk my first talk
at def con in 2019 if you want to go
check it out it's there
and all i want to say here is from that
experience
i found other patients and this
incredible community here
which i'm very thankful for the other
patients around me who have
you know sought a path forward are
working to advance our rights but also
fairer representation in the systems and
technology
and data that affects our lives our
health and our future
we are a non-profit called the light
collective and i'll talk to you a little
bit more about our work at the end
but first let's go into uh what my talk
is about
predictive algorithms are everywhere not
only in healthcare
but increasingly it they exist in ways
that
impact our health that we may not
understand or realize because
the choices aren't apparent to us
we don't truly understand as we walk
through our day and on the internet some
of the ways in which we can be targeted
or how our identities and our futures
can be affected by predictive algorithms
increasingly the choices that we have
and the way we're treated both inside
and outside
the walls of a clinic are dependent on
the data that we generate
and the choices that predictive
algorithms make about us
clinicians develop predictive algorithms
by studying health outcomes
of patient populations over time that
would be the
like five second explanation of
predictive algorithms
and you know that's very basic it's
obviously a lot more complex than that
when we think about
informed medical decision making
clinical grade diagnostics
and making sure that those things are
good and trustworthy for care
when we're seeking knowledge to make
informed decisions
how are those choices really our own my
view on this
from the cheap seats after 15 years of
lived experience
navigating my own future and my destiny
based on genetic
information is that your choices should
be
your own and what you don't know may
kill you
i'm going to talk about black boxes
black box predictions
about our health are killing us in ways
we don't know
and the examples here emerge in four key
themes
patient safety inaccuracy and bias
privacy and security and lack of digital
rights
let me start with emrs epic has
about 20 proprietary algorithms designed
to predict things like
how long a patient might stay in the
hospital
how you might get care and one specific
example of a predictive
algorithm is your chances of developing
a serious condition called sepsis
well this example i'm calling out and i
want to just say that
epic isn't the only emr with predictive
algorithms this is an industry-wide
practice
and this just happens to be an example
that just hit the news so i'm
giving that here an external validation
cohort study looked at epic's
proprietary
algorithm for sepsis and they found that
67
of patients with sepsis were not
identified based on this predictive
algorithm looking at the wider
population
so that's very interesting when we have
a black box making decisions about
you know are you going to stay in the
hospital longer or not or what's your
risk of readmission or developing sepsis
you might not be given the same care as
somebody else based on the data that is
generated about you
what are the sources of that data how is
it accurate
and how is it worthy of decision making
that is informed by
you and your doctor another example on
the right here
is something many may be familiar with
the opioid epidemic
what a lot of people don't realize is
the opioid epidemic over the last
two decades has cost here in the united
states
600 000 lives how did it start
aggressive marketing while underplaying
the risk
of overdose and addiction to drugs in
the middle
of the opioid crisis a silicon valley
startup had a really fantastic
business model idea they were called
practice fusion
and the company developed an electronic
medical record system for doctors and
instead of
charging for the software like their
competitors
the company generated the bulk of its
revenue by advertising to doctors
specifically practice fusion solicited a
one million dollar
uh payment from opioid companies to
create alert systems that would
encourage doctors to prescribe
extended release opioids with two
patients
this has since been busted and is in a
huge
criminal case the doj is involved it's
really messy and ugly
and the maybe the result of that here
is maybe it's a bad idea to um
have drug companies as a business model
um
share with dr c and choose or predict
about a patient
and that's all i have to say about that
how are we solving
the opioid crisis well here's a shining
example
you may be surprised you may not be
predictive algorithms
this is a fantastic piece by formerly
political
reporter mahonda mohanna ravindranav
now the interesting thing here is it's
not based on just clinical or emr
or study data in an irb approved study
without your consent what's happening is
data aggregators
are buying up uh data about the food you
eat
the time you spend watching tv um the
kinds of medications you've been just
prescribed before and and they are
mixing that up into a soup of prediction
and then selling this to
providers in order to predict your risk
this is unregulated
and it is used without patient knowledge
or consent
this is tammy dobbs she lives in the
state of arkansas
and relies on out-of-home care for years
the state of arkansas gave
tammy 56 hours of care per week to do
basic tasks like
getting out of bed going to the bathroom
making meals and cleaning
then in 2016 the state implemented an
algorithm that decided that tammy could
only get
36 hours of care she had to ration
that care and decide and schedule when
she was going to
go to the bathroom and eat and in this
specific case
the the designed algorithm was so wildly
irrational that not even the developer
understood how and why the algorithm
made the recommendations that it did
and what does that leave patients when
something like this happens
it leaves us with loss of care loss of
benefits
uh inability to get food back to
the the the words of aaron in the
beginning of my talk
um and just ways that
these systems can be used to
charge you the most amount of money in
order to give you the least amount of
care
now let me take it back to my little
domain that i know a lot about which is
brca
that happens to be one of the most
studied genes on the human genome now
if you remember i told you those 87
and 60 like super high scary numbers uh
back in the beginning when i was told my
genetic risk
and you have to keep in mind when we
talk about informed medical decisions
that was like a death sentence to me at
25 years old
and a lot of my path and my identity was
has been
shaped by that sense and what i've
wanted
are better options not only for myself
and my future but my family and my
community
and since the the patents were
overturned in 2013
i mentioned that we've expanded access
on genomic testing and all kinds of
technologies and we're learning about
new predictive algorithms that are very
interesting but still need to be
validated
in order for them to be worthy of
clinical grade decision making
so these are in early stages when you
think about
those over 60 like scientists already
now
know there are 66 000
variants on brca1 and 2 alone and the
way that we classify those variants are
some are
harmful or gonna kill you or you might
develop cancer i don't want to scare
anybody
some are totally benign and then in the
middle there are variants
of uncertain significance and those
variants of uncertain significance
some of them are so rare that we will
never have enough data in the population
to make
accurate predictions but they can still
be harmful the way that we are
predicting these things
are looking at for example in silico
prior prediction
um to be considered with other evidence
in
partnership with your clinician or
thinly functional scores and these are
all like
where on the protein did the mutation
happen or
if you look at model organisms and that
organism has
the mutation did they all develop cancer
like those are other ways that you can
look at
prediction of a health outcome without
actually looking at the population
this is dr lynette hammond jorito and
valencia robinson
lynette on the left the lindsay on the
right not pictured is tia tomlin
founder of my style matters in atlanta
and i want to share this as a good
example of
patients representing themselves
in partnership with a researcher to
tackle the problem
of genomic disparities in brca
and data sharing what lynette
has done in this initial study is look
at the many layers of the cancer
disparities problem
we know from prior studies that african
american women are 39 to 44
more likely to die from breast cancer
than white women
well there are many layers to that
problem including lack of access to care
systemic racism in the healthcare system
but i want to talk about how genetic
testing doesn't work
for people of color because those
genes were patented in 2013 what
we knew at the time what what clinicians
and and the guidelines at the time were
that
brca mainly affects ashkenazi jewish
women it's breast
and ovarian cancer those are the um like
very very extreme cases of cancer risk
when it comes to
the population therefore only
people with those certain markers were
able to get tested so back in 2005 there
were just so many hubs that you had to
jump through to get
insurance coverage for a four thousand
dollar test and
to get into a genetic counselor that was
a six week wait time
now over time what happened was people
with less access to care
um were not part of that original data
set or the development of those
algorithms
and now what we have is a problem where
genomic
testing genetic testing for brca
literally does not work
for black people and that is because
the the systems were that you know the
way that we have
developed genetic testing is on
reference genomes for
white people so we need data
and we need algorithms that represent
diverse populations and my thesis here
looking at this example
is that needs to be led and supported
institutional support
for researchers and patient community
advocates
like lynette like valencia and lake tia
leading the way because that's the only
way we are going to earn and build trust
this is their time to lead and it's very
exciting
i promised i talked about hectic hacking
uh predictive algorithms so here you go
it's really not that hard you think of
it like oh my gosh
you know you gotta do the thing and and
get into the system
and breach it um i don't even know if i
said the right thing here but
here's a beautiful example of simon
weckert
a a german artist who took a
a wagon full of cell phones
down the street in order to affect
google's
algorithm on whether or not there was
traffic you know
and i think this is a fantastic example
well not in healthcare
it applies these barriers to entry and
how to hack a predictive algorithm are
pretty low
like you just give it you just feed a
bad data
there you go so up until now i've really
focused on
algorithmic bias where testing works and
where it doesn't
and when we lose care and we lose trust
and we fall through the cracks of the
health care system
where do patients and the public go they
go to social media
when patients fall through the cracks of
the health care system
they turn to social media for help
support and resources
and i'm telling you this because i did
it in 2013
when i was going through my surgeries i
found a beautiful brave support group
before the days of disinformation and it
was a lifeline for me
finding people with a shared identity
and a health condition
is one of the most important parts it
has been for me
and many studies on this show that it is
a really
important part of the healing process
and
the question becomes how do we guide
that and do it in a way that isn't
exploiting people
or harming and abusing their privacy
and what happens outside the walls of a
health care system
that researchers and clinicians often
don't think about when they don't have a
tick tock account
well i will start with this
fantastic study that was done in 2019
evaluating
the predictability of medical conditions
in social media posts
and this was in place it was uh
roughly a n of 1 000
and the post analyzed through natural
language processing was roughly a
million posts 949 000.
and so what they did was they took all
the posts they
looked at um different types of medical
conditions and correlated with um
a way to predict risk of diabetes
pregnancy anxiety psychosis everything
down to
really personal things like whether
you're not whether or not you're likely
to abuse alcohol
whether not you have a digestive
disorder
and you have to keep in mind this is an
irb
approved study with consent from the
people
involved and the power
of these predictive models are very
early
but facebook has a lot more data than
that we know
that when
predictive analytics are used in ad
targeting on social media
pharma companies are able to target you
target you based on your health
conditions
one might think great more effective
marketing you can get to the right
patient
at the right time and we're really
getting effective about micro targeting
these populations
but you have to keep in mind that if you
go back to 2018 we warned facebook about
the huge problems with their
group architecture and the way that
those were feeding recommendations into
uh group recommendations and interests
those things were basically swept under
the rug and then a pandemic happen
i want you to think back to that
practice fusion example
where they had a business model to
target ads to physicians
now this is what i'm showing you outside
the walls of a clinic where we just go
around the physicians and target
directly to the patient and the key here
is to know
that there's no governance no digital
governance that is protecting
populations vulnerable populations from
being
micro-targeted these three ads are just
three examples of what
my community sees when we go on social
media
and there are thousands more of it you
know these are just three examples of
thousands
you can target ads bourbon ads to
alcoholics
you can target snake oil and fake
treatments to us
you can give a
send out a scam of dna based life
insurance
to people with genetic mutations so that
you can deny them care
and and that is actually a real thing
that
happened between july and november of
2020
the health care and pharmaceutical
industry in the united states spent
198 million dollars on facebook
advertising
instagram followed with 151 million
and ad investments have just exploded
when it comes to healthcare
and direct to consumer pharma ads on
facebook and
other places on social media but we need
to stop just like
shifting risks down the road and saying
to people
oh well they should have known better
than putting their health information
out there on the internet communities
rely
on old and outdated tactics when we're
responding to misinformation
you know i have witnessed it time and
again from the world health organization
to the
cdc to hospitals
putting up faqs on their website
download this
yet another app so you can learn about
medical misinformation
and get educated and yay we're going to
be experts teaching you
and that's not working obviously we're
in a place where
that's pretty much played out and we
need to try some different tactics
when i talk in just towards the end of
this presentation i'm going to give you
a few examples of what i think may work
what we don't see is also a lot more
dangerous
i'm going to give you this very
important example of
clinical suicide risk versus social
suicide risk
and i'll point you to this fantastic
piece from mason marks
on artificial intelligence-based suicide
prediction
in the yale journal of health policy law
and ethics
we break down if you think about a
clinical setting and traditional risk
models for suicide
something as intimate as predicting your
risk and making an informed decision
about
how to navigate care that's got to be a
really hard thing right
um well unfortunately in
the traditional clinical risk models the
the the risk scores and algorithms that
have been developed for
suicide risk are notoriously inaccurate
uh this study breaks down these
sensitivity
and specificity and level of evidence
um for a a couple of a few dozen
different predictive risk scores that
have been developed with irb
approval and you know the gist of it
is the a lot more needs to be done
before these are worthy
of informed clinical decision making
in a practice with your doctor
and at the very least something like
this should be
in your own hands as a risk or that you
could
make decisions about on your own and be
informed about rather than something
that is
a decision made about you
well it's starting to be a thing that
we are able to predict risk of suicide
based on what you post and it doesn't
have to be that you intend self-harm
this is a study on machine learning
approach for
predicting uh suicidal ideation on
social media data
and it's it's there are a few studies
like this right now
in their early stages where this was
just a participant
group of 283 because
the study was on twitter data
they felt it was important that to
outline there was no need
for consent because the tweets were
public
and that's an important thing we know
when we go on twitter any user knows
there's no expectation of private
privacy and what you broadcast is what
you broadcast
but what you may not know is that could
predict even if you're talking about
something totally unrelated the words
you use
those could be used to train machine
learning algorithms on your risk of
suicide and then
who knows if they're accurate or not
those proprietary black boxes can be
sold off
to make decisions about your employment
your
ability to get a mortgage or a loan
a student loan who knows it's a black
box and
that's a problem and it begs the
question
should your risk of suicide be a trade
secret at all
or at your risk of any health
information should it be a trade secret
i get that companies have to build
business models
on uh the software that they develop but
there are different ways to do this
and as we think about paths forward and
ethical ai
this is a really important one and
it scares me um because
when we are
aggregating the pain of people or
making decisions about somebody without
their knowledge or consent
it hopefully i've said it over and over
again
is a complete breach of our human rights
and our privacy and it shouldn't happen
you should feel free to be vulnerable as
you want to be
on social media without thinking about
what the implications are of what you
post
and i'm gonna bring up by the way that
goes for any
jerk on the internet who wants to share
things that are
um misinformed people need the freedom
to learn
and to disagree and be wrong
and trust is completely broken maybe i
have shared enough
at this point to convince you that
trust is broken in institutions in
technologies
and and patients the public know it
i personally have an interesting
definition of trust that comes from
stanford professor lindridge greer
and she studies teams and how teams
behave
one of the things she says is trust is
not a feeling
it is a measure of how accurately you
can predict
benefit or harm and
often i think we look at patients as an
abstraction
we think of us as a market or as
consumers or users with no rights
and therefore our voices don't matter
i think of of patients very differently
as needing to
be representative in the policies the
technology and the field of cyber
security
i think e-patients patients like me who
are
activists are hackers too they just
don't know it yet and if we build
trust we have to do so in ways that are
accurate and measurable
what do we do and how do we do that
as early adopters of technology
the brca community and i have been
hacked
scammed sold patented by the last
generation of
innovators in healthcare and i think
it's time for a change
i hope i've scared you enough because
now i'm going to get to some more
lighthearted stuff and a path forward
what is my call to arms well i'm going
to channel
rosalind franklin who was
the original x-ray crystallographer who
found the structure of dna and then add
that
co-opted by watson and crick
and she said
science and everyday life cannot and
should not be separated
she also died of ovarian cancer the
truth is
though when it comes to dismantling the
problems that we
share these are all problems that the
second you go into the wells of a clinic
these problems are going to affect you
too
and i think that we have to start
bringing together the experts and
institutions by supporting communities
of patients and their leadership with
governance digital governance
in order to create a structure like this
to address these problems
i shared at the beginning we formed a
non-profit to serve patient communities
and represent our collective digital
rights today we think of
informed consent and okay if i look at a
50-page document and i sign at the end
well
i am like about to go into the emergency
room box checked
i've given my consent well it just it
that doesn't
do justice to anybody and it is not
sustainable
and it is further enabling these systems
of harm
so our mission is to help peer support
groups people with shared identities and
health conditions
to build capacity around digital
government governance
and foster healthy human connections
when it comes to
representing our identity and our rights
in health data and technologies that
affect our lives
heidi larson vaccine anthropologist said
it beautifully
you know when we think about the problem
of medical misinformation
we should be looking at rumors as an
ecosystem
not unlike a microbiome when instead
today we're just continuing to go to
here go to this faq and educate yourself
when people don't trust it it doesn't
add anything to their experience
so here's what i am doing take this
for example is a picture of the brca
community
the breast cancer community brain tumor
tumor community how we interact this is
a
network graph and i'm learning um
how predictive algorithms work so that i
might
be a part of the solution and for the
past year i've been working with this
amazing
amazing open source a project called
project domino
in fact leo mayarovic and cody webb
are one of them maybe
in our q a so keep an eye out for that
but what we've been doing is working to
understand how our networks are
visualized
and helping to inform the design of
predictive algorithms when there are
often
positives and false negatives how do we
become part of the solution and
represent ourselves
in the data or the algorithms that we're
developing
i think this lends to a more important
question in my mind which is
are patient activists when we think
patients can be hackers too patients
like me can be hackers too we just don't
know it yet
can we be a part of a digital community
of health workers that are helping to
counter the effects of medical
misinformation and become
stewards of fairer representation
in technology um
so little nerd out in eye candy this is
my little
jupiter notebook repository of
brca tweets and here's a beautiful
visualization of
some of the things that we've been
playing around with this is actually um
looking at disinformation networks and
bot networks and
understanding how to identify them and
when you're a patient community going
through trauma
and diagnosis you might not know the
first thing about whether a bot network
is targeting you or not or whether
that's real
and you might only see a small sliver of
the problem where this is a way to think
about
of it as a living breathing ecosystem
and how
information and knowledge is shared in a
visual way
what is my call to arms i am
asking those listening to understand
that patient communities who
are users of technology also have
no rights when we are generating
health data as the supply in the supply
chain
and those data and algorithms can either
cure or kill us
we need digital rights so this is a
draft of six
principles for rights that in 2019 we
drafted with
the the help of patient leaders
disability rights activists e-patients
and
policy and technology experts i
encourage you to go take a look at this
on our website
this is very much in the early stages
it's just a first draft
i have another call to arms and that is
to the people
not above me who think of themselves as
experts representing
institutions or elite hackers who
are just talking about hacking all the
things this is for
the people beside me who are struggling
to learn about the technologies that
affect us and that we often struggle to
trust
we need to start building core
competencies
in technology as we organize communities
on social media
and help them to engage not only with
experts in the healthcare system but
advocate for our digital rights
we are going to be piloting a civic
trust this is a legal model
and civic trust law is nothing new it's
been around for a thousand years
but when we think about having a
fiduciary responsibility to shareholders
instead of
users or patients this is part of the
fundamental misalignment of incentives
in business models like practice fusion
that i shared
or in ad targeting on social media that
can cause harm
so if we think about patients and
leaders and experts as
data stewards that can be part of a new
model
and define the purpose of how data are
used
and communities as beneficiaries of
those data
i think we have an opportunity not only
to create a lot of value in the design
of better systems
that improve health outcomes but also
take a more
networked approach to representation
of communities and diverse identities
that can be a part of a path forward
finally i am i think i already said
finally but
finally again i am asking
for you to partner with us and directly
work with
patients as colleagues and equals rather
than research subjects
and users i am asking you when you think
about diversity on your team
in cyber security or elsewhere in the
design of your health care systems
do you have patients representing
your team if you don't you are part of
the problem
because we can help and we need to be
treated not just as advocates being
asked for feedback
we need to be treated as experts in our
own experience
and build the capacity and skills not
only in the past career path of cyber
security
but in other places because when we do
that there are
so many examples of where we have
changed things not only in the diabetes
community with we are not waiting and
opening
aps but also the think about the hiv and
aids movement and how that changed the
course of
their disease
so finally a third time finally
i'm asking you to join us in rebuilding
trust
uh you can follow twit on twitter be
like light
check out the light collective i'm going
to be sharing how to
get involved with us if you are on the
discord channel
and i'll be tweeting a few things out on
how to get involved
we are currently seeking applications
for our first cohort of partners
and we would love for you to be a part
of that finally i want to give a whole
bunch of thank yous
uh to the board of the light collective
who are e-patients who have been working
on this for two and a half years
advisors or legal advisors or clinical
experts and cyber security experts and
health data experts
i also want to thank the incredible
communities on social media like
bcsm with alicia staley i want to thank
robert with johnson foundation for
for supporting our work and um
brca sisterhood american living organ
donor fund
digital public my style matters and
so much more so thank you
and i'll see you in the q a