- Hi and welcome to Instrument and Find Out
Writing Parasitic tracers for High Level Languages.
I'm Jeff and I'm at NCC group
I like to do,
hack on stuff and do various things
for the purpose of this talk.
That means programs, languages, runtimes, memory, and bytes.
But first up, a notice.
By viewing this presentation, you agree to indemnify
and hold harmless the presenter in the event
you decide to take any of his advice
and find yourself unable to sleep at four in the morning
due to "language demons".
So just as an outline of the structure of this talk,
I'm gonna talk about kind of the background
of what lead me to this work,
what parasitic tracers are,
how to kind of design them
for tracing high level language runtimes,
looking at Ruby as some sort of case study,
and then some concluding thoughts.
So first about me,
I've done a fair amount of work with
the dynamic instrumentation and tracing
from a Java Bytecode, various stuff in Android,
Linux, both user land and in the kernel with eBPF.
Generally I do a lot of this stuff mostly for reversing
and learning stuff and also to kind of
script up existing things to do other things.
So for dynamic instrumentation,
just as a quick refresher,
this generally means
function hooking or instruction instrumentation.
The latter of which mostly means
that you kind of modify bytecode or assembly
to do
be different by coder assembly to do something different.
Whereas function hooking generally
you are doing something that hijacks
control flow directly to go somewhere else.
Dynamic tracing can refer to
dynamically enabling or disabling
existing logging functionality,
but for our purposes this mostly means
adding enhanced logging functionality
that wasn't there before.
I've also been recently doing tracing with Frida for Ruby
which is what this talk is about.
So some background on Ruby and myself.
Little while back, I had to do some
Ruby bytecode transformation stuff
and convert more modern bytecode to an older one,
an older format so translate newer op codes
into equivalent older ones.
So that a decompiler that only knew the older format
would work.
And that worked quite nicely for me at the time.
More recently a colleague and I were looking
at Ruby'sdRuby protocol.
We were writing a scanner for it in Ruby
of all things.
We gave a talk in this in North Sec earlier this year.
There were some weird issues that came up
and I spent a lot of time debugging this
and going through the Ruby internal source code
and see to find out you don't want to call IO#read
on a Socket object.
Instead you just want to call receive.
This lead me to start writing this parasitic low level
Ruby tracer.
So what are parasitic tracers?
Well what are tracers?
So a tracer is an enhanced logger that
basically dumps everything you might want
about program state running code et cetera.
And a parasite is a highly-specialized
unwanted organism that symbiotically lives
on or off of inside of another organism
that is completely adapted to.
So a parasitic tracer is a combination of these two.
It's basically a tracer that's specially adapted
to he target process, that it kind of hooks onto
and injects itself into.
And makes use of internal functionality
that wasn't really intended to be you know, accessible.
So the tracing of this part is just kind of a goal.
I want to write a tracer for Ruby to better understand it,
but the parasitic part is more of an implementation detail.
Chances are, you've done this if
you've ever used LD_PRELOAD to inject code into something.
So why would you write these things?
Well to get a better understanding of
where the higher level extractions
meet the lower level implementations
in say runtimes and things.
So for reversing, or debugging,
or just plain performance analysis.
You could also be writing one of these things
mostly to avoid having to maintain
a fork of the actual code base
if you kind of want to maintain the tracer out of tree.
Because you can just do it on the process itself
and not have to recompile it against the whole code base.
So some examples of these parasitic tracers would be
Frida's Java Bridge API
which is actually really two of them.
One for Android and one for the JVM itself.
They provide basically an API
for hooking into higher level Java operations,
but in ways that weren't really intended
to be allowed by the platform.
So in Android it's totally hooking the runtime
and for the JVM it's using some of the
JVM instrumentation APIs,
but it's definitely doing stuff that doesn't involve those
in weird ways.
And so whereas a normal vanilla Java agent
that uses those things wouldn't really qualify
as a parasitic tracer because it's
using kind of public APIs specifically for this purpose.
The way the Frida does it is a little bit more invasive.
But let's just say that if you're crawling around
the memory of a process or intercepting its syscalls,
chances are you just have a tracer.
But, you know like S trace.
But if you are hooking around in functions
inside the process itself
or really calling functions from inside the process
then you're doing some parasitic stuff.
So let's talk about designing these things
for high-level language runtimes.
So first some prereqs,
you're gonna need some means to actually
hook the code or instrument it.
Generally, ideally, one that allows you to
kind of remove those hooks or re-add them at runtime.
You could do this with a debugger and breakpoints
especially a scripted debugger.
You could do this with and instrumentation toolkit
like Frida which is what I generally do these days.
You also need a way to invoke existing functionality
that's in that code, in that process.
So generally speaking you do that
with a debugger or with Frida.
The debugger would be something like the expression syntax
for calling functions.
But the thing is , you need to know what you're gonna call
So the hierarchy of how you wanna be preferring things
is ideally public APIs that aren't going to change
all that often.
Then internal APIs with symbols.
Then internal APIs that don't have symbols
but that you can get handles on fairly easily.
Say if the pointers to them are passed into other functions
you can just catch them there.
Then after that, you're probably just gonna want to
opt for re-implementing stuff locally yourself.
And then finally all the way at the end, you know,
if you need to reuse existing code that's inside
the process that you can't get a find a good way to find.
You might need to search for bytecode sequences
and match on them.
But moving on, the first step to owning a target
is recon and that is the first step to
designing a parasitic tracer.
And you're going to need to be
doing some reverse engineering.
Really to understand the internals of what is you're
gonna be mucking around with.
And as you do so, you'll learn more.
But you may actually have sources
like I was looking into Ruby, C Ruby.
But you still need to know what's actually going on
in native level especially with the way
that your instrumentation itself works
because at that point C doesn't really matter anymore.
And optimizations can elide out functions
or can lead to weird situations where
garbage values are sent to functions
that don't process on them anyway
and we need to be careful when handling those inputs
stuff like that.
And then additionally just all these kind of runtimes
heavily rely on like super implementation
defined behavior.
And so you need to be really careful
about how you're interacting with their code
from your code.
After that you're gonna wanna identify
all of the things you're gonna wanna hook on
or call into to build up all of your,
whatever it is you're going to get out of
the runtime or the language.
And then next is actually doing all of the hooking
and calling of those things.
You're gonna hook all the functionality.
You're gonna extract all of the relevance
data that you can get.
You're gonna start invoking you know,
function calls that are in the thing
to get other pieces of data out of it, et cetera.
And then after that you're gonna kind of
bring it all together and orchestrate all that
in what I like to call puppeteering
to bring together all your hooks,
have them coordinate with one another.
Possibly be managed by some sort of injected thread
or whatnot.
But at this point, you're mostly building up from there
to have better interop between your own hooks
and better interop between the actual platform
you are messing around with.
So in this case for me it was Frida
which is Javascript .
So basically a Javascript to Ruby bridged more or less.
So ideally you start small and build big.
You compose together a larger set of hooks
from a smaller set of modular pieces.
You're in a good position to do this
because you're hooking on to a full program
that already exists and runs on its own.
So mostly you just need to make sure
that you don't break it with what you're doing
and injecting into it.
But other than that, the thing will continue
to run on it's own just fine.
So the next thing about this layering stuff
is that you can take advantage of
you know, layering on abstract calls
that are implemented with versions specific behaviors.
So for example, if you have a pointer
to obstruct between two different versions of the binary
the field you want that's inside of that strucks
is at different offsets
you need to have some functionality
to be able to handle that.
But the pointer to the start of it is still the same.
So you can do that from per-version implementations
or version-based switches
kind of like ifdefs or both.
But let's talk about Ruby.
So Ruby is a scripting language, that's right.
The most interesting thing about Ruby
is that it's super object oriented
and every time you try to access something
on an object that's actually a method call
and all the method calls are basically
handled via sending messages.
Ruby is super feature full,
but it doesn't have really and good low level
introspection tracing capabilities.
It does have this thing called TracePoint
which is an API for various events that go on
as Ruby executes.
But it can't really intercept method arguments
or native function parameters.
It can't really provide information on bytecode execution.
And it doesn't really provide all that useful
information from any time you're switching back and forth
between Ruby and native code.
This is mostly an artifact,
with the fact that Ruby is a language
and CRuby is an implementation and so
this bytecode stuff, this lower level stuff
are kind of implementation details.
And this API needs to theoretically work across
multiple different Ruby implementations.
But really the CRuby implementation
should have better tracing stuff
given that it basically functions similarly to Java..
And Java has a very well defined
and extensive API for instrumentation.
So I wrote this thing called ruby-trace
which is a Frida-based CLI tool for instrumenting Ruby
and kind of dumping everything that goes on
as it executes.
So it hooks all the opcodes.
The interesting thing about that is the implementation
of the opcode handlers are kind of all
a bunch of labeled go-to spots
and a giant state machine.
They're not really their own function
so they don't have your standard
calling convention preludes.
And then separately Ruby has a bunch of
C functions to call Ruby methods
And do a bunch of stuff about handling the methods.
And tying them to objects,
both native code to Ruby
and Ruby back to native code calls.
So I hook all that stuff
and I hook all the transition between Ruby and native code.
And then hook those native functions and et cetera.
And then separately it supports
kind of hooking into Ruby's internal
exception handling mechanisms.
What is pulls out of that is basically
all the arguments of all kinds
even the special internal ones for the opcodes.
And then it basically, Ruby inspects everything
which is a stringification.
Kind of like repr and python.
The one problem with that is many times
values aren't fully initialized
or with the Ruby VM itself isn't fully initialized.
You need to be very careful about how to try to call things
on things that aren't fully initialized.
So it handles a lot of that.
Trying to be very careful about when it's safe
to actually send the inspect method over
and doing alternative fallback approaches
when it can't.
It dumps out the bytecode whenever
it sees something like a method or
a block being defined.
It dumps all the return values for opcodes
in the native functions it hooks.
It gets all sorts of other metadata
and takes all sorts of things to make it human-readable.
It supports Ruby 2.6-3.0 and I assume
once 3.1 comes out it won't be too much effort
to get it working on 3.1.
I have a sort of generic implementation
with a couple of versions specific behaviors
and switches and then for separate lower level,
anytime I need to deal with Ruby structs from C
I just have a version and specific set of structs
to pull fields from essentially
using Frida's C module API.
So other cool things that it does
is it actually makes use of the TracePoint API
but not in the way you would expect.
It's just the TracePoint API has a very good way
of controlling whether or not it's enabled
based on various aspects.
And so whenever the TracePoint API is enabled
that turns it on.
And whenever it's turned off, it turns it off.
It gives you fine-grained control to
very minutely trace certain pieces of execution.
I have a bunch of test cases for various bytecode sequences
that seem to cover a greater span of more detail
of hedge cases than Ruby's own
internal opcode test suite.
Although not necessarily some of the other ones.
I also implement support for dead Ruby opcodes
that shouldn't even exist anymore for some reason.
But basically RubyTrace is kind of like its own
CRuby bytecode interpreter because of how it works.
So as a demo, let's just switch to this view.
So I have some Ruby code here
that defines TracePoint, tracer
and then in the middle of this big block
is actually a stringified block of this Ruby code
With this Foo method and then some calls into it.
And then it redefines symbol to redefine its
triple equal operator, then calls a lot of
those same things over again.
And then it compiles that code from the string
and the evaluates it under the tracing.
So in this case the tracer that's being used
doesn't really do anything.
So when we run this code it just kind of spits things out.
The more interesting thing is that the not found:wat
on the left side gets replaced with a symbol
on the right side because that triple equal
when it hits the comparison for the symbol.
It instantly matches stuff so the wat string
will on the last case will hit the symbol check
against foo, symbol foo
and that will just pass.
So now let's run this under
RubyTrace.
And basically RubyTrace dumps out a whole bunch of stuff.
One of the first things you can see is
that the instruction sequence part from that compilation.
You see it there and then you see
the call to the eval on it.
And then where inside of that,
the first thing that happens is the foo method
is defined.
And so it dumps out all of the bytecode
of that foo method.
We can see a bunch of values from it.
Next it adds that to the class it's in.
Then we see the first call into foo
from the hello string.
And then we run through that check operation.
So the first thing that happens we see
is a call into this opt case dispatch
which is a special case bytecode generated
for switches that don't have special types in them,
only simple types.
And basically it optimizes so that all of the cases
get added into a single Ruby hash
and then it checks if the value is a member of the hash.
But it first checks a bunch of things
about the object coming in to make sure it's a simple type
that the comparison would work in the first place.
So in this case hello is the simple string.
It's in there. It takes the hello path.
We move on.
The next thing is we have one.
It takes the one path and so forth.
But then we eventually see this big decimal 3.0 value
which you'll see represented variously
as 0.3E1.
And that thing will get passed into foo.
And the problem is, is that because it is
not a simple type. It will fall through.
And the way that this works is that the optimization
is just a quick check first.
And it is kind of a guard on top of what
the rest of the switch implementation will be,
which is the series of subsequent if else checks.
That's just how Ruby does it.
And so it falls through
and then starts doing all of the if else's.
It doesn't match like a string,
it doesn't match a whole bunch of stuff.
And then eventually we see a bunch of
operations where it's trying to compare against
the float value and the big int has to
do a bunch of math conversions to get the stuff out
for the comparison.
And so then it eventually does the comparison
and sees that, you know, it
0.3E1 is equal to 3.0 float.
And so then it check match passes.
That's the comparison for the branch.
And then I jumps to the code that is
part of a segment of the branch.
We continue doing all this for the rest of the values.
We see in this case the string wat
doesn't match anything so it ends up in the else path.
But it was a simple type so
it actually goes to the else path directly.
And then we see the code that redefines symbols,
triple equal method.
And from this point on,
things are gonna get a little bit weird.
So we start seeing that all of these opt case dispatches
end up falling through because triple equals
been redefined.
And so basically there's a short circuit
in the implementation where Ruby says,
"well if any of these core equals things has been redefined
on any of the core types, such as symbol,
it just can't bother to do any comparisons anymore
one way or the other."
And it's faster to just give up
and have them go through all of the checks
one at a time.
And so we run through this all one after another.
And then we get to the end with all the values.
And they get percolated up from
all the functionality.
So Future Work.
I have to implement support for Ractors,
Ruby's new multi VM in process concurrency model.
Right now I'm just relying on the one global
Ruby VM internal of the process.
And then just generally keeping up with the
Ruby versions.
The code will be available here at our github repo
shortly after this presentation airs.
But in conclusion, it's been really fun working on this.
Although it's been pretty tiring
because of all the tiring that goes on with Ruby
and various other things that can fail
when you're messing around in it's insides.
But I think that all of these techniques
pretty much apply to other
high-level languages and runtimes.
Some good examples are Python, Node, Golang, and Haskell.
And I really think people should be trying to
build some of these things.
So to paraphrase Arlo Guthrie,
"You know, if one person,
just one person does it,
they may think he's really sick.
And three people do it, three,
can you imagine three people writing parasitic tracers?
They may think it's an organization.
And can you, can you imagine 50 people,
I said 50 people, writing these tracers?
Friends, they may think it's a movement."
And that's what it is.
I'd like to thank Addison, my partner in crime
on the DRuby stuff
that lead us down this rabbit hole
for me doing this work.
And a wise man once said,
you can't hide secrets from the future using math.
I believe that is true, but I also believe
it is true that you simply can't hide from the future.
I would take questions, but this is a recording
so there are no questions to be had.
So instead I will answer a question about
why on my intro page I used an image from a
Pokemon crystal and not Pokemon Ruby.
Well to answer that question.
I do not like Ruby lang.
I do not like its yuppy gang.
I do not like its simple keys.
I do not like its optional parentheses.
I do not like its method send.
I do not like its begin and end.
I do not like its magic verbs.
I do not like its methodic key words.
I do not like its IO dot read.
I do not like its lackluster speed.
I do not like its deal open jit.
I do not like strong params permit.
I do not like its if unless.
I do not like its dependency mess.
I do not like its case in when.
I do not like its require middle men.
I do not like its polymorphism.
I do not like its object fanaticism.
Its object nil gives me pain.
That Ruby lang is profane.
Thank you.