- For the last couple of months,
I've been playing with different SAST tools.
I'm playing with SAST tools
because part of my job and part of my actually interest
is how to secure code and then the best way
and doing so, we always start by playing with SAST.
After SAST, the much more highest than SAST
but we'll focus about static tools and I turned to play
with them and abuse them in different ways.
I will show you how.
So, who am I?
I'm Rotem.
Nice to meet you.
I am the head of marketplace at Cider Security.
It's a startup I joined a few months ago.
But more important, I'm a bug bounty researcher.
I play with lots of different programs over the years
and I'm a Cyber Paladin for the last 20 years now.
So I started my InfoSec career 20 years ago,
and then I'm just playing around with different,
some application testing to infrastructure testing
to developing a different SAST tools
and now playing and abusing the SAST tools.
I will show you what I've done.
So, first of all, who is my target audience?
So I'm talking to the different type of people,
and I think this will interest also security engineers
who want to fix problems
and they have to tell other people
where they have gone wrong.
I'm talking to DevOps people
because they are in charge of large scale deployments,
and everyday we have more and more automation
and more automated deployments going out everyday.
SAST Builders, of course, it's a different,
like this talk is about SAST
so I'm talking with you guys.
And I bet the bad guys,
people who have decided to harm other people for a living.
I hope none in the audience here, but if you are,
please go to the good side and start helping people.
So a bit about what we'll be talking about.
It's SAST, how it works.
I'm going to give a brief intro
about how we can get the works
and why I started this research.
I will show up the hacking and what I've done,
and that will wrap up with different conclusions
and what is the impact of all of this.
So SAST 101.
What is SAST?
SAST is static application security testing.
This means that we are testing applications
in a static manner.
So I went to Wikipedia
and Wikipedia says very, very easy,
that static program analysis is the analysis
of computer software that is performed
without actually executing any programs.
This is very important because static analysis
can happen on different targets
with different types of class
and we don't want to execute any programs.
Let's say that we have different puzzles
that do execute programs and their sandbox environments,
but SAST was supposed not to execute nothing.
So why do we run the SAST?
The first thing is to stop bad security practices.
We want to make sure that nobody
and so it's a bad code into an organization.
We want to prevent infrastructure mistakes
and now we have lots of infrastructure as code
and we want to assess code.
So if we have like a big security test,
sometimes I will run a bunch of SAST tools
to check the code.
And they want to create standardization
and consistency across lots of codes.
So let's say, there was no eval in our code.
We can create a rule.
No eval and we run it and it would be standardized.
So why SAST?
And we have different pros vs cons.
But the pros, it's very fast.
It can run on code.
Different SAST have different like times,
but it's much faster than SAST
and much faster than other fuzzing and various solutions.
It's safe.
It doesn't execute any code and it's easy.
We can run on the code.
We usually don't need any other resources.
Sometimes we have inputs and stuff like that,
but usually, it's inside the same code based
and we don't need external resources for it.
About the cons, we have lots of false positives.
Like SAST can do as much as looking at the code,
but it doesn't know what's the logic
and it doesn't know how was it use.
It's very hard to track a flow control.
So if you have lots of complex flows,
SAST will be a nightmare over you.
Not why SAST, but how do they work?
How does the scanners work exactly?
So, first of all, the scanners take the code.
They pass the code and they look for different,
let's say, job script.
So they look for job script files
and then they start converting these JavaScript text files
into AST structures.
AST is a structure of how to template code
in a way we can process it later.
So after we create the say, standard,
we'll show you later how it looks like.
We start processing it.
When we process the code,
in basic, we have different rules and different findings
where we want to look for.
And then the more complex SAST,
we have flow control analysis,
and we start looking for sources and sinks.
But then we have all the results
and we'll create a result based on SAST.
Usually today, we have SARIF.
It's a really fun format
that lots of tools are starting to adapt.
And we are working with this
and we are creating the results,
so other systems can not consume them.
Let's take an example.
If we have a Log one plus two times three.
It's a very simple function
that does a arithmetic computation.
And how does it look like in AST?
So, first of all, if we look at the binary of the tree,
so we see the program, the CallExpression is Log.
It has a plus expression.
And then there's a split like one,
and because times is before class, this is a basic math,
so we have split over there
and then there's the two and a three.
Sometimes these AST are by ANTLR.
It's like the structuring tokenize of a language
or by other means, it doesn't matter how it created it
but then it creates this too.
From this tree, we can create JSON or XML
or however we want to represent it
but we are creating the data
that we can start looking up on it.
So let's say, we have here a CallExpression
in the identifier.
Again, we have a callee and arguments
and you can see the binary expression,
so now we can start walking the two
and start understanding what's happening over here.
If we take a basic rule, let's say,
if this is a CallExpression and we have a log
and the length is above zero, we have arguments,
then we can say there's a log function
with more than one argument.
It can get complex.
We can have more very flow control
and a very complex architectures.
So let's say we have this if and then,
and all this goes into a variable we call source.
We target as a source.
And then we create another argument,
another look up forward sinks.
If we using nil for JSON or some other graph database,
we can start connecting sinks and source
and seeing the whole trace, and then if data source,
we can report a finding.
So it can get very complex and very fast,
but still it's very functional, it's very static.
We never executed the code.
So as Wikipedia said, it clearly doesn't execute the code.
We don't have any execution over here and we are good to go
and we can start using the SAST to assess
even the most malicious programs and malicious code areas.
So my hypothesis and my question was,
what if I could write code that will intentionally abuse
a scanner once statically scanned?
So this means, can I create a method or way
with these areas to change the behavior of the scanner?
To abuse it or change it though.
So I looked up at different previously researches
and we had in the past,
there was a Checkov remote code execution.
It was fixed.
And then I Checkov 2 one day created it.
Someone could create a malicious terraform file.
And with this terraform file,
it could execute code inside Checkov
and the one executing the code had access
to the whole environment of work was executed.
And there was a very simple Checkov walkaround.
Do not run Checkov on terraform file
from untrusted sources or pull requests.
But I want to run Checkov on untested sources.
This is why I'm doing it.
So it's a bit of a mix.
So they fixed it and now, I feel I'm okay,
and I can run again Checkov.
But I was playing with, it was a repo,
but you can use it also as a SAST.
It's a Clojure repo that you can use also as a SAST tool
and create different types of rules,
and I will expand about it a bit later.
But I opened a bug about it that it actually evaluates code.
I will show you exactly what it does and how it does it,
but in the comment I got, I talked to the guy over there
and they said, "okay, but this is how this kind of works."
And we have nothing to do about it.
Maybe add the computation.
So I'm looking at terraform.
Terraform was actually, it's not SAST, I know,
but we do have different SAST scanners
like Snyk or Terrascan that do rely on the terraform plan.
So one of the recommendations inside CI/CD environments
is before running like Synk IaC is to create a TF plan,
and they show here in the documentation, run terraform plan,
and then terraform will show.
So we have terraform.
Terraform has a plan and then apply.
And in the apply, it does actually do stuff
with the environment and executes code,
but the plan shouldn't do anything malicious.
Hiroki Suezawa, someone I talked with
in one of the cloud forums and he pointed
that actually terraform plan can run code.
You can create a terraform provider exec
and you can run code with it.
So if you're running any SAST
that is relying in terraform plan
and the other terraform plan before the SAST,
then you should know that someone can create a provider
and you can create even an unofficial provider
that will download it from our HTTP server and execute.
So you can see also Alex Casso has created a very nice blog
about it, but this is other.
So by the hacking time
and now we are going coming into different stuff.
But first I want to have a small disclaimer.
I believe in open source
and I think that makes the world secure
and it makes our life easier with open source,
but we need to use it responsibly,
because open source is not a full blown commercial tool.
It didn't have the use of development and lots of clients.
Maybe it didn't have lots of clients,
but it doesn't have enough resources
to always to be in the best security maturity level.
And we need to understand this,
'cause when we use open source and I believe in using it,
but we need to make sure that we run it
and we treat it as something that it's not fully baked
or it's not like some are more mature, some are less mature,
and we have to know about these areas.
And one of the parts of my mission and what I want to do
is to make sure that it will be safer and safer
to use open source in companies and in real life.
But for starters, we have to make sure
that we are running in a safe environment
and that everything is configurable,
then we know how to configure it properly.
So that's about that.
But my experiment.
So my experiment is very simple.
I looked at different scanners like collected
through my experience.
And then I started looking at how they will execute
or what can I do with them?
And they created different kinds of evil files.
What is evil files?
I will show you in the experiments there now,
but I'm adding these evil files to the repo.
I'm letting the environment clone the repo
and then assuming the scanner will walk
in the same working directory as the repo clone.
Now, this is a assumption based on different levels
of knowledge in companies and they saw in lots of places,
the scanners are running inside the same working directory
as the repo and scanning themselves,
scanning the repo itself.
Then executing the scanner
and they want to see what's the outcome.
What can I do?
So experiment number one, Checkov.
Before we had the rce on Checkhov
and this is why I started looking at it.
And they started looking and edit the commendation
and they saw that Checkov is actually running against,
you can run it against the directory.
Against a working directory.
The same working directory or the home directory,
and once you are running against the directory,
it looks for a .checkov.yml file
to load it as a config file.
This is interesting, because it's actually how you need
to work with Chekhov or with other SAST tools.
The SAST tools as you develop different changes
and different types of levels of maturity inside the code
and developers are able to skip rules
and optimize each repo according to what they want to do.
But this means that also, if I will take a repo,
I would scan it.
And I added the checkov.yml file into the repo,
I can say, check none.
I can give it a check none file.
What will happen is that the Checkov
will pick the configuration up
if there's no forced configuration and it's big if
but most or lots of places I saw
don't have force configuration.
So it will actually not scan the code.
It will give you everything is okay.
This is in a philosophical way, a question.
If this is how we should run code,
because we are giving the developer lots of permission
and lots of ability to configure properly
what he wants to do, but at the same time,
we are giving him the permission to do nothing.
To say, just skip it.
Let's bypass security altogether and just leave me alone.
This is a good question
about what is the security team I should do?
Let's look at a demo.
Here we took TerraGoat, a TerraGoat is,
let's see if we can see it here.
(Rotem speaking faintly)
We talked TerraGoat.
TerraGoat is a repo that has lots of problems
for the Checkov, and you can see,
we have lots of fair checks.
Then the echo, this is a very important thing.
The tone code is one.
One means there was a problem
and if it was in a CI/CD environment, it would fail.
Now I'm echoing again the same thing,
but I'm adding the bypass checks into .checkov.yml
and running again Checkov.
Nothing.
Everything is good.
There are no errors, no nothing.
Their code is zero.
Everything is awesome.
So this means with a very simple adding of the rule,
I think of this file,
we bypass the whole scanner configuration.
So after we saw what we can do with Checkov,
I looked into different other tools
and they all, as I say though, by definition or not all,
but lots of them give you the same configuration,
the same ability.
They have a different configuration file.
You can create it and then bypass or create different rules.
And we can see here, PHP Stan, Tfsec, and KICS, and Bandit,
and Brakeman, Checkov and SEMGREP
and I'm sure there are more that I didn't check.
We have the different configuration.
If you put this different configuration,
tell them bypass our rules,
they will bypass all the rules.
You can see there's so many stars on the scanners.
I think it's about like 12, 15, we have 25.
We have lots of get up stars.
Over here, let's say get up stars we are looking at.
Before we went more,
I want to emphasize and talk about the scanners hijacking.
So what have we done?
We were able to add all the source code in a manner
that we were able to manipulate
and abuse how this kinda works,
this kind of behavior itself.
So if someone adds a file inside a repo and I'm running it,
I'm able to tell them now everything is okay,
just skip skittling.
Sometimes it's good that we can do it
and sometimes we don't know about it
and we don't have the proper visibility
if someone really did it.
Like I don't look at all the files out there
in my 1,000 peoples inside my organization
and start looking did someone manipulate and abuse
and bypass my scanners.
Like this of what I'm doing.
The scanners don't tell you,
ah someone bypassed me, please check this.
So that's a good place to think about
and understand what we want to do with this for the future.
Let's go, this is DEF CON.
So I'm going into experiment number two,
and this is much more interesting.
So I continued to look inside Checkov
and I saw there is external checks there.
Now does a directory for custom checks to be loaded?
I like custom checks, because custom checks means code.
So looking at the different what I can do.
So again, I created a repo that I can clone it, scan it.
It has .checkov.yml inside.
I load the configuration.
And inside, I tell it, go to external checks,
so inside the directory checks.
Now I can call the clone, I can call the repo.
So I created a directory with that checks with init py,
and then inside, I can create any Python file I want,
and it will load it.
Like one Checkov will scan,
it will first load my files and execute them.
So now not only I'm able to bypass the configuration.
I'm actually able to execute code inside the environment
that the scanner is running.
And let's see a demo about this what we have in demos.
So I created here like I have my pipeline
that is waiting for an event.
I have a rce file over here
and I created a .checkov.yml file with different tunnels.
Inside the tunnel, it just creates it,
cause the rce.sh with Checkov.
Rce.sh just send something to my pipeline
so I get on call, I don't remember.
And as you see, at the moment I run checkov,
I see a post.
I got something from Checkov.
I even see it had a picash complied to my files,
I side it's stuff.
So it's actually pretty cool.
Now with this command execution, I can execute.
I can access environ variables.
I can access different networks.
I'm actually running as the scanners themselves.
So that's important.
Experiment number three.
Now I talked about Kibit before, about Clojure.
Clojure is a very interesting language.
It's very, very dynamic and very fun.
This is better from the source code of Kibit.
Those are read file, and this is how it reads
the files themselves the source code.
It uses a function called read.
Now we have a warning inside the source code
that you should not use clojure core read or its string
which is weird, because every junior developer
will try to read string and read data
from untrusted sources.
So this means that the question is why.
Like you may be asking why is this a warning?
What read actually does
is reads the Clojure code and evaluates it.
And then Clojure, we have something called
like a self-evaluating form and I will get to it
in the later slide, but this is Kibit.
Kibit it as a static code analyzer.
It takes some code and then when you run it,
you run it to your applying user.
It's lein is like the npm of the Clojure.
You want it to apply and it tells you,
okay, this is a problem.
We'll consider using when instead of if.
But then if you use self-evaluating form,
that's #= something.
This will actually run if you run it
through read the function.
So I created, again, I load different libraries
because Clojure is very dynamic.
I can load everything I want,
and I did a print line to a shell
to run again the same rce Kibit
and then shut down agents just to exit nicely.
So I run it line, Kibit, running code
and then I see the exit code, exit 1: out success.
I'm able to run my code.
Experiment number four.
We see we have pre-processing.
So Rubocop for instance has a configuration file
that is very cool and very dynamic and so dynamic.
Once it sees, it'll be contemplating, it executes it.
I saw in the documentation.
Yeah, let's do get status.
So I just did run rce Rubocop, exit.
It worked.
So the moment I load Rubocop, it looks for .rubocop.yml,
executes my codes and exit.
Great success.
Experiment number five.
And this is a bit different than the others,
because I didn't find a PMD any way
to have a configuration file,
but it looks for so much environ variables
and one of the environ variables is the Java Opts.
So you can tell that what options
to give Java one running PMD.
So I told them, use jar.
Use my jar instead of your jar.
You can run an Eviljar, it works
and you can load the another jar.
But the question is how can I tell it, the environ valuable?
So in some CI environment, so in some areas I know
like even I know my sh, you can create a plugin
that if you see a .env file, load it automatically.
So if I also submitted that .env file, ENV,
sometimes it will load in the environ variables
before running the scanner.
So it's a good way to do stuff.
I played with it.
Some areas I will able to put a .env file
into lower the different environment variable
and they executed my code.
So if I am concluding and I have much more scanners
in the pipeline to check and I have different areas,
but this is a bit about the stuff that I talked about.
We have Checkov.
For the configuration of file.
Checkov, PHP Stan, Rubocop for the configuration file.
We have Kibit for code.
I can create code that will execute.
I have PMD and cdxgen and Dep-Scan.
I didn't talk about them,
but same thing with environ variables.
Every time someone shows me now study cos.
I'm checking what the configuration file?
Does it have some kind of a loading of rules,
and more, more and more.
Every day I'm finding more and more.
So what Wikipedia says is correct.
It doesn't actually execute the program
over the static program, but it does execute programs.
So I am able to execute programs for different,
we can call side channels from the configuration
or from other methods of the scanners themselves
to be dynamic, but except Kibit that does execute programs.
So normally, but all the others are really for the ecosystem
of different work that we are using.
So I'm saying like your code will probably be able
to execute other programs.
This is something we need to understand
and know and live with.
Big question is what is the impact?
If I'm looking at the impact and then trying to look
at what it can do to my environment,
then I need to understand first where I'm gonna get.
So I can run static analysis in development machines usually
except if someone will do a get clone from untrusted sources
and just load into his idea or some other areas,
and the security team automatically analyzes it.
He decides to analyze it.
This is a risk.
We have security researchers.
If I want to scan untrusted code, how do I do it?
Because SAST may execute code
and I don't want to execute the code on my computer.
But my environment and what I am researching mainly
in the last couple of months is the CI/CD.
So inside the CI/CD, how does it work?
First of all, developers commit and push
to development branches.
When you commit and push,
lots of systems do CI checks on every push.
They do it in their internal cloud or in Jenkins
or in other CI environments,
but they run different types of security testing
on the code so we make sure
that no code will be insert into production.
But maybe we can attack the systems themselves.
The next place is the pull request.
I never pull request.
This is the request I want to merge
this code into production.
We are doing lots of checks
and lots of checks usually also done
against the production environment
or against staging environments
that have really credentials into areas.
And lastly, we have merge into production.
So the merge into production,
we can merge from people that by mistake
or by intention, push directly into the master branch.
And for the pull request.
But the same, we have to have the CI checks,
and then we need CD deployments to deploy.
Now, this gets tricky because sometimes companies forget
to separate between the checks and the deployments
or give too much formations to the checks.
The implications over here.
First of all, I can extract sensitive data.
We had different attacks in the last couple of months,
that running code inside our CI/CD environment,
extracted sensitive data and like environ variables
and different AWS keys and even code
and send them back home.
If you have access to the internet, it's problematic.
You can bypass protections.
Let's say we don't even have command execution,
but we do have the configuration bypass.
We can create like a policy to bypass the code,
to say, skip security.
I don't need them.
I don't want them.
They were just making it harder for me
to deploy into production.
We can infiltrate the network.
We can go and use this as a stepping stone
inside the network itself.
Lots of good teams I've done.
I played from one place inside the internal network,
you can stop nmapping, you can stop going our SNMP.
You can do lots of stuff into other areas,
maybe it's connected.
The last, but most important,
maybe you can deploy as production.
You can skip the CI checks.
You can false deployment through the CI
and this is problematic.
Assume code will execute.
This is my assumptions.
We need to put around this code as much guards
and lasers and boxes and sandboxes
and different areas we can,
that eventually someone will find a way to execute code.
If it's from a very small open source SAST tool
to a very large commercial SAST tool or some other things
like let's say, code coverage tools.
I don't know.
Someone can hack a code coverage tool.
It can be awesome.
We assume the code will execute, and if the code executes,
we need to be prepared to make sure that it can't do nothing
and we know about it.
A bit about before I showed the workflow,
the CI/CD workflow, but in this attack flow,
I can add code execution to a scanner configuration file.
We can push new commit into the branch
and then create a PR request.
This is every user inside the organization can do it.
Sometimes we can do it from the outside,
even let's say, and get up from folk.
Actually get up, fix this problem or I don't know,
fix but made it much harder, because now today,
new commits don't execute the workflow automatically.
And this is lots of attacks, got stopped in this area,
but other areas are less mature and do execute code
if you push it to the branch or the PR.
But then one repo will be scanned by a scanner
and the PR, it will execute.
This execution has access to our network.
Sometimes also it's the same network
and same environ variables that the deployments
and the CD deployments have.
We can also just skip the whole check
and tell the scanner all is good.
Continue on into production
even that we introduced very bad code practices
back those or whatever you want.
A bit about high level possible solutions.
I don't want to go into much here, so just gonna scan this.
A bit the network protected, deny photos,
deny access to the outside,
isolate only what you need inside the host.
The same great content knows and pods and run everything
and list play permission,
and then verify everything is deleted.
Sometimes if you have a part of designing for 10 hours,
maybe it's a crypto mining,
maybe it's just stealing the data.
Monitor everything,
just monitor everything that you can do
and then look for malicious activity.
We need to understand the risks
on running unverified code inside our laptop
and inside our CI/CD environments.
We need to educate ourselves, educate the organization,
educate the DevOps in what's the best
and the how to do it properly.
Do what teams on the CI/CD.
We want to verify the tools we are running are good,
but we don't have enough resources to do it,
so we need to create a framework.
To check that how we can ensure that the execution
is running in the most secure way.
Just ensure that your scanner
aren't doing any malicious stuff,
try to stick to SAST, to static
and deny any execution possible.
If you can make sure configuration is really picked up
by your configuration.
How it code in the configuration.
Lots of tools have option to hold coded configuration
to something specific.
So it won't pick up the default configuration files,
also environment variables
and set them if you don't need them,
and the list goes on and on.
I think that security needs
are getting bigger and bigger every day.
I think I see it and we are getting into massive automation.
We're going into a DevOps deploying
different production features every minutes, seconds even.
We have to be proactive
and understand the next generation of attackers
and how they will abuse our infrastructure, our automations.
How they will attack us from side channels, from internally,
just abusing automation that we are doing
in order to make our lives easier, and secure them.
Other study code SAST tools.
It's not only security.
It's also linters.
It's also code coverage, testing frameworks.
We have much more automations coming on.
We need to deep dive
and understand the different SAST scanners.
We want to analyze the Wrappers.
Let's say we have GitHub actions and then talk about it
over here, but different GitHub actions or Wrapping or Orbs.
Wrapping all scanners and creating it more difficult
to add configuration files.
It's more difficult to check
if there is a configuration file to run security.
We need to analyze them.
We need to create a standard for security working
with code analysis tool of any kind.
I would want the standard, not only for the output
like sorry for that, as for input
How am I running security code tools?
Maybe do a SAST for executing security tools.
I don't know, it's a bit meta,
but we have much more to do.
First of all, I want to thank you guys
for coming to my presentation,
and I want to thank all the open source developers out there
for creating the awesome tools they are developing.
It's tons of work.
We have millions, thousands of companies working
with these tools and you're helping them in so much ways.
I created a bit of a POC about what I talked about,
CI/CD lamb.
You can check it out over here.
This POC is running great.
Once scanned with different scan tools,
it's time to execute them.
You can just pick it up, try to use it.
Don't attack nobody.
Just do it for research purposes
and start to understand how can you run it by then?
I started creating a community,
I called it Security Tools DEF CON.
This link code goes to a Slack
and they want to start connecting
between the open source developers out there.
I don't know, anyone else that wants to join
and started thinking about how can we standardize our tools,
how can we create better recommendation
and better ways to make sure
that once people are running our tools,
they're doing it in the most secure way.
Also, I want to raise awareness about this.
You can pick me up on Twitter.
You can pick me up in the Slack and I will be in DEF CON.
I will be going around.
Just send me a message.
Hey, let's meet.
I have an idea.
I want to talk.
I will be happy to talk with you
and have a beer, go to a party.
I don't know.
Just hang around.
Thanks again and we'll see you at DEF CON.