hello everyone my name is jeremy brown
and this is
summer of fuzz august 2021
at the defcon 29 apsec village
mac os edition
quick run through the agenda uh we'll
take
a walkthrough of some of the debugging
tools used on mac for fuzzing
uh talk about sip and the app sandbox
crash reporting
uh how to deal with sleep and ssh stuff
looking at process execution like
enumerating file and url handlers
client and network services how to
discover those
and then talk about fuzzing the
different cli and gui apps
filtering network clients and servers
again and some of the bugs found
then finish off with a conclusion
so who am i uh mostly interested in bug
hunting
fuzzing offensive security these days
previously doing security stuff
at microsoft amazon and video to name a
few
like breaking stuff of course that feeds
into attacking
different products and services and you
know for the most part helping them get
in better shape
before release as well as a ton of other
security stuff
uh don't i like various different
domains
native code web services cloud
containers type stuff's all interesting
as well
so intro i didn't know much about mac
security before doing this research
i was pretty new to mac in general it's
always one of those things where
it's like ah it's on mac how do i you
know everything is so different
than the windows and linux world a lot
of similarities with linux but
just it was kind of a learning curve
just getting into and trying to wrap
your head around all this stuff
but i've been fuzzing for a while now
and you know there's so many advances
these days especially with afl plus plus
and and all the interesting stuff coming
out of that area
and innovations there but i really want
to look at it at a different angle
i wanted to kind of forget a lot of what
i already knew
and what was conventional wisdom and
what you know everything about
instrumentation these days and
got to make it super fast and scale and
all this stuff and
there's so many targets that just make
it really hard to make a fuzzer that
does that just kind of general purpose
that handles
all these different targets but is
really good at all those
you know very innovative stuff that's
happening in fuzzing these days so i
want to build something from scratch
that really serves a purpose um
and that makes it easy to fuzz a lot of
different things
although making some trade-offs and of
course this ties into mac because a lot
of the
targets i'll be showing how to fuzz
these days and how to set up fuzzing for
that
we'll get you started fuzzing uh
userland stuff on mac
so i'm mostly looking at core uh the a
lot of the default
applications on os x you know
like i said again client and gui apps
uh stuff that parses stuff on the
command line uh
files um but you know a lot of more
interesting stuff is actually fuzzing
the gui part
which afl can have a hard time doing and
of course the network
services and you know how to fuzz
network stuff
natively with a fuzzer and then how to
set up the environment for debugging
there's a lot of stuff to know to
actually get started
fuzzing and being able to debug and
triage crashes and things like that on
mac
again we're not we're not trying to do
something that scales for
a huge fuzzing farm a lot of this stuff
you can automate
to make the tooling do that but this is
you know a lot of these things you can
reproduce
at home with a macbook and you know
still have fun finding bugs
so related prior work ben nagy's always
got a ton of good stuff
on fuzzing osx and some tooling around
that
crashling wrangler there's some pretty
good stuff there too
uh and the inspiration for getting back
and fuzzing uh for me was this uh
tip out sh zine article which was
pretty interesting it really made me
think wow you know you can start over
with something super simple
and still be effective
so debugging xcode is your you know
development package
there's a headless script you can find
online that i've referenced
and you want to enable enable developer
mode
so pretty straightforward able developer
mode that's going to help you do a lot
of debugging stuff
especially in a non-interactive way
and then guard malloc so this is
something this is a malik helper
that can help you find some subtle
memory corruption bugs
you can use it you could just pass the
environmental argument
to the library and then run your
run your target and the arguments and it
will uh
basically help you find crashes a little
bit easier and more crashes too that you
may have missed by not
instrumenting and using this library and
you can read more about that
at the libg malik but that's available
on osx
then lldb is the debugger of choice on
mac gdb
had a hard time getting it working and
it's sort of oh you know the lldb is
it's nice interface nice environment a
lot of stuff's already set up for mac
so um yeah probably recommend using that
me and you can automate a lot of the
crash triage
and uh produce bucketing data um you can
parse
the um the debugger logs
for you know to sort things by pc you
know the
return address the program counter
instruction pointer all that stuff
look at the back traces a lot of stuff
you can just pass with dash o
which is just passing different uh one
line commands to lldb
and this is how you do you know
automatic crash triage uh
for in a very you know simple way
sip is an interesting one on mac so
system integrity protection
does a bunch of different stuff uh
restricts the root user
you know sign processes protect slash
system all that stuff
um for debugging and for fuzzing stuff
uh you're gonna wanna either enable sip
without
without a certain mode or just disable
entirely while you're using it for that
because that's going to get in the way
of a lot of again debugging app uh stuff
that that you're going to want to
utilize for fuzzing
so on a physical machine you can reboot
recovery mode and then disable it or
again enable it without debug
so that will give you some debug
privileges essentially
and if you're using vmware fusion uh or
any other
uh or you know virtual environment uh
for mac
for for fuzzing and debugging um there's
a bunch of steps there's a few
blog articles to go over it but
essentially you have to
do some special steps to get around
startup disk and
restart the firmware and and get this
stuff to work on the latest uh
os x and ox os x and os
11. so yeah big sur
it's a little bit different and all this
stuff but essentially you
you want to enable this uh to to be able
to
you know debug on os x
in the app sandbox uh it's another
interesting one that can get in the way
of fuzzing
it's not you can work around it but it
is something just to keep in mind
when you are trying to fuzz for example
guise
because a lot of files can only only be
opened in certain locations
so let's say you mutated files that are
in temp
and you know you place your fuzz files
and temp
or wherever on the file system if you
try to open that with some apps
you'll get uh it can be open because you
not permission so if you check the logs
you can see it's being denied the
sandbox is actually denying that file
read
so you can there's other directories you
can put them
so again if you can
you know if you can manipulate the file
system if you can write in certain
places you can figure out where
they want to see files open from
and of course you can still double click
and open the epub file from from you
know downloads directory and stuff
but for local fuzzing you're going to
want to place them
in somewhere that when the
target is launched it it looks like it's
in a good location and in the sandbox is
happy with it
so a report crash is an interesting one
um
generally uh if you're fuzzing and
you're catching your own crashes and
stuff you don't
you know you can just disable it it may
just
if nothing else for perf issues you may
be worried about
crash reports being sent to apple stuff
like that
there's other things you can work around
for submit diag info
but if you want to disable it for the
most part that can be useful
but also you can use it
you can use it to your advantage when
fuzzing because
if you enable it in certain cases
it will help you and you can actually
use those crash logs that it catches and
puts in the
diagnostic logs folder uh for triage
so if there's you know scenario say
we're fuzzing
uh a udp server where it's hard kind of
hard to catch crashes
um you know if you're fuzzing it from
from like a black box screen watch
gray box approach uh you can attach it
uh if you can attach to it run it into
bugger catch crashes
that's great uh sometimes for whatever
reason you can't
uh and in that case we could monitor the
diagnostic logs folder uh to look for
crashes to know when there is a crash
things like that so it can actually help
to enable report crash in certain cases
and sleep super important for humans but
on a mac
it gets really annoying when you're
frozen and all of a sudden your session
is disconnected because
it went to sleep and there's like 10
different sleep modes on mac
they're super eco friendly in that way
which is great when you're just using it
as a user but when you're fuzzing
i need the you know os to be super awake
all the time
and so there's a bunch of different ways
you can do it system sleep tm set you
can use keeping you awake
on the desktop side um and there's still
some more work
to do with ssh though
so with ssh you know it's nice to be
able to fuss guys over sh
instead of uh the physical session
you don't have to have physical access
to to fuzz things like that
it'll just open up from the command line
on your
desktop session but ssh can
depending on your network and the
environment stuff like that it can be a
pain
to get it to not uh you know disconnect
your sessions
uh in the middle of fuzzing so i wrote a
bunch of stuff into a fuzzer that
that tries to help with this by doing
like these custom
console pings to make sure it's always
alive and things like that
but uh yeah surprisingly uh that's it
can be a lot of work just to make it not
disconnect your session a lot so so one
way to do it
is modify the the your daemon config
to do keep lives and zero zero for the
client alive stuff
and then restart the daemon this may not
work for every
everyone and again the network
environment you know how many
um you know whatever's going on your
network how many disconnections you
normally get or
things like that can factor in uh but
this is one way i found to do it to
avoid that
so for monitoring process execution uh
see what's going on in the system
there's a really handy dtrace script
that's included on mac now
uh called new proc d and you can use
this
uh again if you disabled sip or or gave
yourself the bug privileges
um you can use this to see
all the stuff that's being executed and
the command command line args for that
which is super useful for looking at you
know what is the attack service here
and you can see different stuff
happening like turning off file sharing
turning on vnc stuff you know turning on
printer sharing
all those things when you're launching
apps you can see what args are being
launched
although with xpc proxy that sort of
does a lot of stuff under the hood
instead of on the console
which makes things a little bit harder
to debug or see what's actually going on
but uh d-trace is amazing and uh super
helpful for
process process execution stuff
numerating the handlers i came across
the swift default apps
pretty cool project um you can it's got
a
swda
program that will give you all the ur
uri handlers
so you can see what applications are
handling
address book afp cfs
facetime stuff files all that stuff and
you can see there's 101 different
handlers so
so yeah plenty of interesting stuff to
poke at there
and then for the file handlers you can
see what stuff is actually
handling the different file types that
are registered
on a mac so your flash videos pdfs
archives you can see archives handled by
archive utility
things like that if you do a list 420
different file formats that are
registered to an application um
yeah if you if you imagine uh half or
or whatever are valid and they're
actually going to map to something
that's actually parsing the file format
there's still a ton of attack surface
there so lots of good stuff
uh it doesn't necessarily tell you the
file extensions so for example
the apple script um you know it doesn't
tell you exactly what the file type
may be or the the file extension that
you would want to look for to get
test cases to fuzz with for example
but so you may have to do a little bit
of work for that when you start
targeting an application
but for the most part it's super helpful
looking at network processes again we
can use dtrace
to capture um the different uh receive
calls
or sends that are coming across the
network and we can see which processes
are actually
receiving network data and you know
probably operating on that network data
you can also just turn on everything you
know system preferences
sharing turn on everything look at
netstat you get a good idea
of what's listening uh you can use lsof
to look at process more process
information
and other tooling too uh but uh
yeah that's that's one way to look at
the network services that are listening
so for fuzzing
afl if you try to run traditional afl
on on mac um
it doesn't work so well uh doesn't think
a lot of things are a mock o even though
it knows
supposed to know how to recognize those
afl plus plus again
a ton of new development there looks
like they got that figured out so you
can use afl plus plus
when you're fuzzing on mac with on
simple console applications
things like that but what about gui apps
you know it doesn't work so well you
know
just just fundamentally the way it's
designed uh it just doesn't optimize
for doing things like that um you know
it's it's looking for the program to
exit for example
uh and you know a lot of gui apps you
need to exit them manually if you're
going to open
them what about network fuzzing windows
yeah when afl and afl net a lot of
respect goes to them
very interesting projects but they have
some limitations too
and there's quite a bit of setup that
goes along with them
so uh wrote a new fuzzer recently
that we'll be releasing in a few weeks
if all goes well
called life fuzz it just works
that's the biggest theme you know i
wanted a fuzzer that
i didn't need a different setup on mac i
didn't need a different setup on windows
it worked on linux it you know i wanted
to support
file network fuzzing i want to be able
to fix gui's
even interactive ones like postman or
filezilla stuff like that
you always want you know automatic crash
triage
diffing as much information as much
artifacts as possible that you can
create
to give yourself a you know a good
analysis
and a way to repro uh different crashes
um lifos is most useful for closing
source applications again
um clients and servers that's what sort
of was meant for of course you can do
open source libraries and stuff too
and it's really built for buck hunters
it does some
pretty interesting things unorthodox
fashion um
and it's it's able to sort of live off
the land
in a way and i'll talk about the
trade-offs
here in a moment for that
so uh it breaks a lot of rules for
modern fuzzing uh you know these days
everything is about how fast can we do
it
how can we instrument this like how can
we get down and get code coverage for
every single
um you know every single iteration uh
how can we fuzz it
you know a million iterations things
like that
um life us doesn't do that it says
okay i want to cover all these different
domains uh and i want to
be able to not have to worry about
crossfit platform problems for the most
part you know
other than if defs and stuff and i want
to make to make it super simple to set
up and i don't want it to
fail it only works on this type of
application or
oh you gotta you know spend several
hours instrumenting this app or getting
it set up
i want to make a trade-off to forget a
lot of that stuff and just say okay
it can flush black box apps it can focus
open source but it doesn't need
tons and tons of time to get set up you
can start fuzzing as soon as you get
some test cases for the most part
on three different major platforms it
doesn't have the
devops integration stuff with automatic
bug filing and all things like that
of course it can be written around that
and that can be a feature but most
importantly what should be judged by
for a bug hunter does it find bugs
uh does it find bugs does it do does it
do what it was meant to do like if
you're fuzzing and you're not
you're not finding any bug like you have
all this infrastructure set up and
everything's going great and but if
you're not finding bugs and you're not
uh you know getting like actionable
stuff out of
out of the tooling you know you can
measure it how you like but uh
i like to find bugs but then you ask
yeah i don't know the fuzzer is doing a
good job how have we compared it against
other fuzzers how we did things like
that
how where how does the stack against afl
other than the you know
stuff we talked about um yeah does it
find bugs
that's what i'm optimizing towards and
i've acknowledged trade-offs there too
so with fuzzing mac uh some there's some
intricacies
around gui apps for example um
notice when those fuzzing some apps they
like unique file names if you give them
another file name
maybe they cached it or maybe they
remember it somehow
and it doesn't doesn't feel so well like
that so they like unique file names
uh some some apps at least they also
prefer to have the right extension in a
lot of times so you don't want to just
create a bunch of files that are named
dot fuzz um
you tried to copy the
input file name the test case whatever
it was named on the extension as much as
possible to make it look like you know a
legit file so the parser's
more willing to uh not bail when it
looks at it and says oh
you know i don't know what dot fuzz is
but i know what dot pdf is for example
and again with reading files within
sandbox you may need to place
the test cases in a certain uh folder uh
which
of course is an easy command line thing
to do
to choose that uh but some things are
just hard to fuss directly
so sometimes passing a command line arg
it just won't accept it it wants it
through another another piece of
plumbing somewhere
through ipc for example but
many of the classic apps and a lot of
the stuff on mac
still work as command line apps for
example so
still quite a bit of attack service to
tackle there
so for target application do a cli
or gui we're going to look at ibooks
fontbook package util
you know collect the test files again if
it's not a known extension
and it may not open it and you you need
to specify if it only will open it from
certain locations like the container
locations
you need to specify that as well
if the target doesn't exit on its own
for example gui apps for the most part
measure reasonable timeout give it
enough time to parse the data
and then kill the kill the process and
then
start fuzzing so with ibooks
i'm just going to demo some ways to use
lifos to these things and you can infer
you know some of how it works from there
specify local mode
and then the command would be
the target application plus the keyword
fuzz which basically says
put put the fus file here and make that
command
execute you can also do sddn on most of
the platforms
by not specifying anything and it will
just
read the test case and then
throw that to the target stdn input
files give it some epub files
uh specify an output file sorry output
folder
um just a specific one or automatically
goes to crashes directory the
root crash directory um and then give it
a special tim directory
so normally it would go in uh temp slash
light fuzz
slash the run
the run number uh it's got each one gets
its own folder
each each fuzzing session gets its own
folder but you could specify um the temp
directory
uh to be that container directory so
that ibooks actually will open it
max running time yeah again you're
screaming oh this is going to take
forever
hey we're fuzzing the gui app so yeah
there's again some trade-offs to be made
for
app fusing if you can't directly hit the
harness
uh number of iterations and then easy
which we'll talk about
soon and then with fontbook
mostly the same thing except fontbook
doesn't need the special temp directory
um yeah easy is basically um the e
option uh is to recycle any of the
crashes found
so when you're fuzzing and it finds
crashes it'll actually
put that crash directory as the input
directory at the end of the session
and it'll try to go through and mutate
all of the
crashing input to see if you can get any
other
you know different crash points
and things like that so it's just a fun
little feature uh i
thought of doing and the z option is uh
to enable any malik
helpers such as um the uh
g-lib malik that we talked about earlier
the to help find some more subtle
crashes so that's just enabling it when
we run the application in the
environment
package util uh it's command line app so
super easy to fuzz
uh you could even fuzz this with afl in
non-instrumented mode if you wanted to
um uh yeah like i said life fuzzes
uh for this scenario you could use afl
but we can also use live fuzz for it
uh and then uh i guess one of the
advantages there is turning on
the uh malik debuggers uh for if you use
life us for for fuzzing package util
and during the filing process there's a
global std out
so you can check the process see what's
going on with the app so if we do this
while we're fuzzing
package util you can see that you know
it's outputting uh it's
it's you know it's xml file looks like
it mutated some input
can't open package for expansion um that
package
uh that fuzzing test case and uh yeah
you could just kind of monitor
um the the app's output and just see
some
pretty funny stuff depending on how how
much it outputs during errors for
example
and you can see the guard malloc is
enabled because that's at the top of the
output
so minimizing crashes is another fun
fun feature of light fuzz um so it's a
pretty
pretty simplistic approach um the way it
works is it just removes one bite at a
time
and runs the target with a new repro uh
if the target crashes uh just keep uh
continue on
removing the bright removing the bytes
and keep that new repro
because we removed to buy it and it
still crashed
so that's good that means we don't need
that byte for the test case
theoretically and if the byte doesn't or
the target doesn't crash when we remove
the byte
then restore the byte because apparently
the byte's important
because it made the program not crash
so simple bye bye bye minimization
and then you see here we had a crash
xc bat access sec fault
we throw that into the minimizer it
tries to repro it once it re-pros it
starts minimization and you can see it
took it down you know
you know a couple thousand bytes that it
didn't even need and it reduced the
crash
um to there and it's also a way to find
new crashes too so maybe when it's
minimizing it it removes bytes that make
it
crash at a different pc so it's also
interesting to know so it'll catch those
new crashes as well when it's minimizing
it and then superman mode is basically
just repeat the process until the target
is exhausted
we've exhausted all the bytes and it
will
we can't go any lower we've achieved the
maximum minimization on the bytes
so yeah pretty neat features to minimize
your test cases again super
super simple way of doing it but
effective
so for targeting clients um we're just
going to do a
demo of uh smb util uh
you essentially just capture uh you know
use wireshark whatever
to capture and export the protocol data
for example if we're doing clients you
want to get responses from
from servers during a valid session
so uh we use the um
local client mode for for life which
which allows us to still look for
crashes
uh in the app so we still have a lot of
visibility uh
there but we'll see later with some of
the remote server fuzzing we don't have
so much visibility
we run s b util uh and then we can we're
gonna listen so it's got a built-in
fuzzing server that's going to listen on
4455. we're going to tell smb view
to connect to our host and you know view
smb's stuff
at 4455 instead of 445
which is the default port give it some
responses
and then dash p is going to tell us to
fuzz a session
so with light files you can post binary
sessions
so this is saying that i've captured
a session smb session of responses
at this directory and to parse those
parsers responses and fuss with them
uh on the servers so uh you can pick
different servers
uh for example there's a there's some
remote management um
stuff with mac which is like a vnc type
server screen sharing is more like a vnc
server
some interesting mac nuances there and
again
capture the stuff flying across the data
or flying across the network
and start fuzzing
it's actually a great blog post too on
the uh on uh
red teaming blog post on some of this uh
the ard agent and and things like that
that
you can check out to get some more
information on how they work
so we can use lsof and check and see
the rd ar d agent is listening
and what port and then we can run
so if we were able to run this in
debugger like some
sometimes you you can run it stand alone
some of these servers you can run
standalone
as a in a debugger or you can attach to
them too
so you kind of got to play with things
here to see
you know how the target likes to be
interacted with uh
and then we can pick udp fuzzing and if
we're doing a remote server
that address and that's going to fuzz
udp packets
from our input that we captured and send
them over to
the target listing on that port
for rd a rt agent
so this is where you know using crash
report comes in to monitor for crashes
also some features like that you know
you can also do packet sniffing
and you know uh kill everything when
when no more packets are going on and
running debugger things like that so
there's a lot of different ways to catch
crashes here
and then we can see that rdard
agent is uh receiving our packets
and the you know it's it's should be
parsing your packets as well
using that detroit command to uh to see
what's going on while we're fuzzing
uh similar with screen sharing agent uh
we're gonna look and see what's
listening
uh some of these processes launch d only
runs them only runs like screen sharing
d
uh when you get a connection uh which
which can be interesting because
sometimes you need now you need to sleep
if you want to attach the process for
example you need to connect
and then sleep and then attach the
process uh
because you then you'll get the pid to
attach to stuff like that
and then yeah we can a screen sharing
session based on what we captured
remotely and or locally and use the
crash report to catch
capture crashes as well
which uh this is what we're doing here
uh
we're looking at report crash we're
saying anything that looks like
uh screen sharing d that you get you get
a crash file for
uh let me know because i'm interested in
that and that would be a i would mark
that as a crash
and start copying a repros and stuff
um interesting one when uh just trivial
fuzzing this one
looked like a cpu exhaustion bug when
fuzzing screen sharing d
um but yeah it just gives you an example
of
some of the stuff you may come across
when fuzzing
so as far as bugs go um yeah it's still
some research is
kind of still ongoing it's kind of early
honestly uh
but fuzz bugs out of a few different
components applescript
uh colorsync syslog
also parsing some interesting stuff uh
and some more upcoming cves so
pretty pretty successful so far and a
lot more to come
so maybe you're more interested in
fouling on mac now maybe before it was
kind of this so
i don't know it's different than windows
and linux but
now maybe you're more motivated and can
see that
yeah there's a learning curve but
there's you know decent amount of attack
service like every other os
uh windows linux included
you gotta work around some security
features but it's not too bad
by default though apple has spent a lot
of effort locking down a lot of
debugging features which
yeah if you're going to you know get
down for research
you need to turn off so in order to
figure
you know get more visibility into the
system
and uh yeah some of the tooling uh even
unconventional techniques and fuzzing
can still
yield good results and uh yeah pretty
fun stuff
so thank you very much for your time
and that's the end of my stuff