- Hello, everyone.
Thank you for attending our session.
My name is Glenice.
And together with my team, we will be presenting,
Hacking Humans with AI as a Service.
We are part of a research team of four from Singapore.
We work at a government technology agency's
Cyber Security Group, or CSG for short.
Let me introduce to you our team.
Firstly, we have Eugene, who has been working at CSG
for about one and a half years.
He specializes in Application Security
and Vulnerability Research.
He is also an avid white hat hacker.
And many will know him by his handle, Spaceracoon.
Next up, this is me.
I'm a part of the Red Team
and Social Engineering Team in CSG.
Apart from human hacking.
I'm also interested in web in cloud security.
Along with Kee Hock be presenting with us today.
We lead the AI phishing exercises.
Kee Hock has been at CSG for about two and a half years,
where he contributes to our Red Team
and Cyber Engineering Capabilities.
On top of that, he participates actively
in Capture the Flag competitions.
Last but not least, Timothy is a member
of the Mobile Penetration Testing Team at CSG.
And also a (indistinct) in Red Team exercises.
He led the Technical Defense Research
against AI generated texts.
All right, let's get on to the exciting stuff.
Here's what we will cover today.
Starting with an introduction of human hacking.
We also experience some challenges faced
when conducting social engineering exercises.
Next, we'll have a quick look at the recent advances in AI
and how the challenges faced in phishing
could be addressed with AI as a service.
Then we will examine the viability of human hacking
using AI as a service with Tuna Fish,
a service pipeline we developed
for simulated phishing exercises.
Finally, we're going to the defenses
against AI phishing, such as, automated AI detection
before summing up important takeaways from our presentation.
Let's begin an introduction to social engineering,
the old school method.
Social engineering is a psychological manipulation of people
into performing actions they usually may not.
Three common influencing tactics include, authority,
whether a attacker assimilates someone
with the right to exercise their power.
They may also attempt to manipulate
the decision making process by creating a sense of urgency.
Providing context specific information
tends to lower people's guard
as the attacker attempts to exploit the pattern
targets are comfortable in.
All this aims to exploit our blind spots
and break security checks.
Here are three common ways social engineers
can deliver their attack.
Firstly, email phishing,
which is the main focus of our presentation today.
I believe many of us here are familiar with it.
If you haven't received a phishing email before,
chances are you have, just unknowingly.
Phishing can also be done via a phone call,
and that is known as vishing.
Lastly, impersonation often allows social engineers
to enter a basis they shouldn't have access to.
Social engineering is practiced for many different reasons,
malicious actors wants your money and information,
so they create a pretext in an attempt to beat you.
But it can also be used by authorized actors,
such as Red Team engineers, and those involved
in raising awareness and conducting security training.
Speaking of security training and exercises,
here are some statistics.
According to a 2020 report by Terranova Security,
almost one fifth of employees in the targeted companies
click on simulated phishing email links,
even after going through a training program previously.
This rate increased to 43% in the study
on spear-phishing emails against more general users.
How are phishing emails sent to the victims?
Let's now take a look at email phishing campaign
from a victim engineer's perspective.
Here is a typical manual phishing workflow
in a simulated phishing exercise.
The victim operator would typically start
with intelligence gathering of the target.
This is also commonly referred to as OSINT.
This includes looking at your social media presence.
OSINT can be done manually or with the help of tools.
Then with the understanding of the targets,
the operators work on crafting a suitable phishing email.
The amount of time required for context generation
varies depending on each target.
The Red Team operator will also craft their email
based on different weapons of influence,
such as, authority, scarcity, and so on.
We can see that the entire workflow is manual
and dependent on the operators.
Even though spear-phishing often yields good results,
it may require significant time and effort
to analyze the results from OSINT,
contextualize it and bring some of possible pretexts.
These efforts increase with each new target
and these challenges can be broadly categorized
as behavior analysis and text generation.
Interestingly, these are also strengths of AI.
Capabilities of AI have improved over the past few years
and we now have a greater understanding
and more realistic expectation
of what AI can do and cannot do.
For a start, here are some tools
that are meant to be used by recruiters and promoters
to automatically analyze public information
such as LinkedIn profiles.
Coming from an ethics perspective,
we could repurpose it to perform personality analysis.
Although the results could only provide a general direction,
it is a good foundation for the Red Team
engineers to work on.
Humantic AI was used in our phishing pipeline,
but many of these tools have a freedom
that allows anyone to register and use the API right away.
Next up, text generation.
During last year OpenAI released a text-in, text-out
interface for assessing the state
of the GPT-3 language model family.
The initial release of the API generated a lot of hype,
such as, an article in "The Guardian",
that's claimed to be written by the GBT-3 API.
While the amount of curation inputs are unknown,
it was good, GPT-3 possessed significant capabilities
in natural language generation.
In practical terms, the GPT-3 API in this
presents a major accessibility.
Based on the estimates by Lambda Labs,
GPT-3 will have taken a really long copy time
and millions of dollars (indistinct).
But users can now use it via the API
for just a few cents per thousand tokens.
Open API provides four language models
of different quality and price from Ada to Davinci.
The instruct-series beta for Curie and Davinci
are recently released, optimized to understand
and follow text instruction.
As a simple example, we could input
the following instructions.
Explain quantum physics to a six year old
and here's what the API returned to the instruction.
How realistic are the emails generated by OpenAI?
With this few sentence, we instruct the model
to create an email to encourage target
to open an attached document.
Right.
Nicely done there with a proper email format.
We've created a context and detailed instructions.
No model may generate coherent email content
for the considerations of the Red Team operators.
When generating context,
the temperature parameter in OpenAI comes in handy.
It controls the randomness of the text generator,
the lower the temperature, the more deterministic
area of the particular results.
Thus, with the same prompt,
a different type of output can be obtained.
This will likely ease the process of pretext generation.
Having introduced the viability of an AI phishing pipeline,
how are the results in the real world?
I'll let Kee Hock elaborate on this
and deliver the rest of our presentation.
Over to you, Kee Hock.
- Thanks, Glenice.
Hello, everyone.
I'm Kee Hock.
I will be sharing more on how we can use AI as a service
in Red Team operations, particularly
in the conduct of phishing campaigns.
In addition, I'll be sharing our team's work
on detecting synthetic texts and also
to highlight some good governance practices
in which both AI service suppliers and consumers can adopt.
Previously, Glanice did a good job
on highlighting some of the AI services
available in the market.
What you see in this diagram
is the revised phishing workflow
after we applied various AI services,
during the different stages of the phishing campaign.
I'll be focusing on the key differences in the workflow.
Firstly, Red Team operators will perform
reconnaissance on the target, such as,
gathering the target's LinkedIn profile,
Twitter profile, blog posts.
Next, we'll feed this open source information
into Humantics AI, which will perform
personality analysis on the target.
For phishing context generation,
we repurposed Humantics AI API demo
to perform personality analysis.
The service is meant to be used by recruiters
and sales people to automatically analyze public information
such as LinkedIn profiles, to produce a personality report
and generate competition advice.
The Humantics AI API will return adjacent output
as shown in this snippet here.
It contains the personality analysis results of the target.
Just a point to note, Humantics AI
is just one of many AI services
which you can adopt in this pipeline.
Depending on your preferences, you can choose to integrate
other personality analysis services.
We thought it would be cool to use these services
from a different speed, especially
with a Red Teaming perspective.
These services are extremely useful
as they help us quickly analyze the targets
and allows us to crop up more personalized instruction set
to feed OpenAI's GPT-3.
Next, we have passed the adjacent output
into simple set of plaintext instructions
and feed it to OpenAI's GPT-3 model.
Essentially, the plaintext instruction
described to the model about the target
and how to approach them.
Depending on a team's requirement,
the logic to pass adjacent output can be customized
to feed a certain context.
For example, you may want the AI to generate
a text message instead of an email.
In this example, we are simply asking the model
to craft an email, to convince the target
to take interest in our company.
We ask the model to be formal and objective.
Finally, we have the output for OpenAI's GPT-3.
At this point, depending on the team's comfort level,
you may choose to edit the output
to repay some of the elements of the content.
For example, you may choose to change the sender
to another team or department.
In our experiment, which we will share later on,
we made edits to the center and the subject title.
We refrained from editing the main email content.
As you can see, the generated email is coherent
and fairly convincing, applying authority
and consistency in its instructions.
However, there are a few quotes that require human edits.
For example, you may want to add in a phishing link
or you are containing the phishing link.
Nevertheless, this demonstrates the need for a human
in the loop for this AI phishing pipeline.
Now, the phishing content is generated
and ready to be delivered.
For the subsequent steps,
it'll be the same as any other phishing workflow
like the one Glenice shared earlier.
So, in our case, each phishing email sent to the target
has a unique phishing link, which helps to track
if the target click on that phishing link.
You may choose to use any phishing framework
to perform the delivery or the phishing emails.
In our case, we built a custom pipeline
with a custom phishing tracking system.
This will still accommodate some of the constraints
we face when conducting the phishing exercises.
Before I talk about experiment,
I would like to share that, although we build our pipeline
with a custom user interface and backend,
the possibility of your API's
meant that we could easily integrate this
into any existing tools,
such as the GoFish open source phish Framework.
This highlights how AI service offers a step up
in accessibility from open source language models,
rather than having to worry about a compute
and server-side generation.
Adding AI capabilities simply requires an API key
and a (indistinct) request.
We put the revised workflow to the test for three months.
We conducted two types of experiments
with over 200 targets across multiple
authorized phishing exercises.
The first side of our experiment,
which we call Type One Experiment,
is designed to investigate the effectiveness
of convincing the targets to click on the phishing link
in a phishing email.
Our Type One Experiment, each target receive two emails.
One will be generated by the AI,
while the other will be generated manually
by the Red Team operator.
The second type of experiment
which we call Type Two Experiment,
is designed to investigate effectiveness
of convincing the targets to open
malicious attachment in the email.
For Type Two Experiment, the target group
is further divided into two subgroups.
One subgroup will receive AI generated phishing content,
while the other will receive phishing content
generated manually by the Red Team operator.
Next, I'll elaborate more about a set up
for a Type One Experiment.
Essentially, we conducted in two stages.
Stage one, is a mass phishing exercise
to identify targets who are susceptible to phishing.
Stage two, is a spear-phishing exercise
to harvest credentials from the targets
identified from stage one.
For each stage, we collect a few key metrics.
First, we measure the phishing success rate,
which is the percentage of targets
clicking on the phishing link.
Next, I'll elaborate more about a set up
for a Type One Experiment.
Essentially, we conducted it in two stages.
Stage one is a mass phishing exercise
to identify targets who are susceptible to phishing.
Stage two is a spear-phishing exercise
to harvest credentials from the targets
identified from stage one.
For each stage, we collect a few key metrics.
First, we measure the phishing success rate,
which is the percentage of targets
clicking on the phishing link.
Next, from the victims who click on the links,
we observe the interaction and measure the percentage
of victims that interacted with the phishing site.
The table you see on the right is a sample size
for each phishing exercise conducted
using Type One Experiment set up.
Notice that there are some rows under the human column
that are empty, especially for stage two.
This is because there are no victims who fell prey
in stage one for the content generated by human operators.
So this is the results from our Type One Experiment
for stage one mass phishing campaign.
The results are encouraging.
To then interpret the chart,
we take a look at the bar chart.
Next, we will show the results from our Type One Experiment,
for stage one mass phishing campaign.
The results are encouraging.
To interpret this chart,
we take a look at the bar chart first.
For example, the column represents the exercise.
In exercise A, 20% of the targets
picked on the phishing link from the phishing content
generated by AI, while none fell prey
for the phishing content generated by humans.
Then we move down.
We look at a corresponding pie chart.
For exercise A, 80% of victims
who clicked on the phishing link
interacted with the phishing site.
In our case, they interacted with the forms
on our phishing site.
While AI outperformed the human counterparts
in exercise A and C, exercise B
tells us a slightly different story.
In the bar chart for exercise B,
we see that human generated content
had a 9.4% phishing success rate,
while the AI generated content only had an 8.55%
phishing success rate.
In terms of raw numbers,
the differences in percentages
translated just one more victim
who fell pray for the human generated content.
Now, we move on to stage two results.
Just want to reiterate that stage two victims
were identified from stage one.
Basically, those who click on a phishing link from stage one
will be our target for stage two.
The only difference here is that we are conducting
a spear-phishing exercise,
which means that phishing content is more personalized
using the workflow revised.
In this chart, for exercise A, we see considerable success
in convincing the victims to click on the phishing link.
And 33.3% of them attempted to submit their credentials.
Exercise B, results were more interesting
because in stage one, human generated content
outperformed the AI counterpart.
But in stage two, AI outperformed
the human generated content.
For exercise C, our phishing site
is (indistinct) by most browsers as a deceptive site.
We postulate that the corresponding phishing emails
are likely to be caught by email filters
or any other defensive controls.
Thus, the experiment kind of stopped for exercise C.
Finally, let's talk about Type Two Experiment,
whereby we investigated effectiveness
of phishing content generated by AI and human
in convincing the targets to open malicious documents
attached in the email.
For Type Two Experiment, we only conducted stage one
mass phishing campaign for both target groups.
Do note that the target groups
are receiving either of the phishing content
are mutually exclusive.
For Type Two Experiment, we are looking at the percentage
of targets who open the malicious document.
In this exercise, the AI generated content
convinced more targets to open the malicious document.
From our experiment, we've made thee key observations.
Firstly, the revised phishing workflow
using AI as a service definitely helped
to reduce the time required for phishing content
curation and context analysis.
This allows Red Team operators to focus on other tasks,
such as performing OSINT investigation on targets,
which can be difficult to automate.
Next, we observed that AI generated content
was more convincing compared to the human counterparts.
However, this is still not conclusive,
as there may be other variables at work.
Nonetheless, this is an indication
that AI does have a lot of potential
in innovating Red Team operations.
Lastly, we observed that there's a varying degree
of governance relating to the access
of AI services in the market.
For example, during our research,
some of the AI services can be accessed
by simply registering for a trial account,
while services like OpenAI was certainly way stricter
in terms of the usage, as they validated use cases
by the individual when they attempt
to register for an account.
Having demonstrated the viability
of an AI phishing pipeline,
we decided to explore potential defenses against it.
This is a important question,
as success to AI as a service grows over time.
In the next segment, I'll be sharing more
about the possible defenses against
AI generated phishing content.
Kudos to Timothy for the heavy lifting
on this aspect of the research.
Naturally, the first thing that came into our mind
is to detect synthetic text.
However, detecting synthetic texts remains a hard problem.
There are three main ways that we can go about solving this.
Simple classifiers where we define certain rules
based on tags or using MLBs and LT classifiers.
Fine-tuning based detection,
where we train a new model based on inputs
from different kinds of language models from GPT-3,
GPT-2, BRT, and so on.
Zero-shot detection, using this method,
the models learn a classifier on a set of labels,
and then evaluates on a different set of labels
that a classifier had never seen before.
We have decided on this approach
as it gives us more flexibility,
as we will be able to use it
to predict other sets of language models.
The white paper for GLTR has given us
insightful solution to tackle our problem.
Again, we want to highlight that detecting synthetic texts
still depends on the model that's used to generate a text
and a model use for text detection.
(indistinct) Language Model Test Room, or GLTR for short,
is a project based on a collaboration
between MITIBM (indistinct) and Harvard NRP.
Based on the white paper released by them,
the approach to detect synthetic texts
were the probability of a word
given the previous words in sequence.
The absolute rank of a word,
the entropy of the predicted distribution.
Our research used the same metrics along with GPT-3
to evaluate if an email is written
by a language model or a human.
We have also formed GLTR report
and used the GPT-3 model as shown in the screenshot.
While deciding the approach of detecting synthetic texts
by GPT-3, one major challenge our team faced
was that we do not have direct access to the model.
Without their access, we are limited in the parameters
that we are able to control.
We are also limited in the datasets
that has returned from generated texts,
such as, top 100, lock prop, or choice of words.
We have decided to extend GLTR in our research
as the data we can get from GPT-3
has transferable patterns from GPT-2,
which was used in GLTR.
With that you will look at the finding from our research.
In our tests, we compared GPT-3,
GPT-2, GPT-2 Tuned and human samples.
Our GPT-2 Tuned model is based on an email corpus
that our team has harvested.
For GPT-2 and GPT-2 Tune model,
we followd the settings that was described
in the white paper.
We found that human samples,
humans frequently use words that are out of the top 100
predictions from the GPT models.
We found that for human samples,
humans frequently use words that are out of the top 100
predictions from the GPT-3 model.
Looking at the KDE graph reinforces the conclusion
that humans frequently use words
that are out of the top 100 predictive words from GPT-3.
The graph for humans also looks more distributed
compared to GPT-3.
From our research, we can conclude
that evaluating a probability for a sequence of texts
is a good indicator of whether the text
is synthetic or written by humans.
However, there are limitations in this approach,
as it is heavily dependent on the model
that's used to predict the sequence of texts
and the model used to generate the text.
Next, we put our proof of concept
GPT-3 detection two to the test.
In this slide, you will see three email samples.
Only one is crafted manually by hand,
while the other two is generated by OpenAI.
Do not worry if you cannot figure out
which one is generated by AI.
Even for our team, we find it difficult
to differentiate which content is generated by AI
and which isn't.
Let's take a look at sample A.
This is the visual output from the tool for sample E the
ratio between the properties of a top predictor and the
following. What is that?
0.8 tree tree.
Then we look at sample B, the ratio is zero point fall.
So an observation from the GLT,
our team is that when you think are to the detection that
attacks attacks generated by humans tend to be more or
rather have more highlights.
This is because human generated content tend to have more
random words in terms of the choice of words,
and often forced out of the top a hundred predict of what by
GB tree needs to be. Look at sample B.
The ratio is that 0.84.
So an observation from the GITR team is that when you,
you have to detect synthetic text or tax generator by humans
tend to have more real highlights.
This is the costume and generated content tend to be more
random in terms of choice of words,
any of the force out of the top hundred period of words,
as highlighted earlier.
Finally, we look at sample C.
The ratio is at 0.55 for the relative lower ratio is an
indication that is human generated content,
which in the case,
this is true on top of identical defenses.
We look at the non-technical aspect of the book.
We found that model EA governance,
Fremont released by Singapore is a useful reference to
address key ethical and governance issues.
When deploying AI solutions,
we will not be covering the content of framework,
but rather we extract the development details of it for both
suppliers and consumers, they can adopt the following,
use the proposed implementation and self-assessment guide
from the framework,
establish policies for explanation and practice general
disclosure use perform ethical evaluation,
implement clear roles and responsibilities for ethical
deployment of AI.
For consumers,
they can adopt the human in the loop approach for AI,
the decision-making for suppliers and show traceability and
auditability of use also to enforce acceptable use policies.
Finally, our conclusion,
we somehow are sharing three key points for everyone to take
away.
Firstly,
the rapid growth of AI service has placed advance cost
effective AI text generation capabilities in the hands of
the global market.
They can be used by both authorized or malicious actors,
but ultimately the tools can be used to build defenses
against AI generated texts.
Current approaches are brittle and model dependent.
AI assisted human detection of AI generated text could be
more effective.
Lastly,
decision makers have the responsibility to implement sound
strategies,
governing the supply and consumption of advance AI service
tightening the usage of advanced AI as a service can
potentially reduce the likelihood of abuse
we've reached the end of our sharing,
and we hope that there are takeaways for you from our work
to catch us live at Def con and we are more than willing to
discuss our resist review.
Thank you.