right hi everybody welcome to our talk
building an ics pie range in our kitchen
sharing our journey in lessons learned
so you don't have to
we have roughly 30 minutes to introduce
you to our
little project and give you some of the
details and
most importantly also of course the
challenges and difficulties we had along
the way
of building a firing range um
just wanted to take some time before to
introduce
myself and mort's here next to me
and also at the company i'm starting
with myself my name is nico
i've been working in the i.t security
for 15 years
um and now i'm the penetration testing
and red team lead for enviso in
germany hi my name is thomas
i've been with eviezo for over a year
now and i've been mostly involved in
this
r d project that we are going to present
today right
a few words about nviso as a company
and was originally founded in in belgium
in 2013
and then in 2018 we opened the offices
in germany in frankfurt in munich
and currently we're counting over 100
specialized security expert
from all areas of id security really um
and one point i wanted to highlight here
is the involvement in
research and development projects so
from the
annual revenue of the company we invest
roughly 10
into rd projects which is relevant for
this project because it's been partly
funded
as well by this of course and it helped
us a lot
to develop as well and try out some
things and make sure that they work as
we
as we wanted them to be
and this is how it all started was what
made us um
what motivated us to to build the ics
firing range
as it is nowadays it all started
towards the end of last year where we we
did some external trainings on ics
security or
dt security and we realized that there
is an abundance
of of skills and knowledge that we we
need to acquire and also to share
internally in the team
um for our pentester and red teamers
most of them of course they come from
from being coming from
the i.t world so they they don't have
any
didn't have any exposure to ot
environments and ics systems so far
and as you can imagine there are
differences between
for example doing pen testing against
iot systems
and ot systems so different requirements
in either
case so there was a really really um a
strong requirement
for our team to to get trained in that
respect you know to develop that
level of awareness in 4it
environments so we we thought about you
know what's the best way to do it
um to maybe develop a lab that we can
use internally where we do have a couple
of ot components
where people can try out different
techniques and just see how they
they react how they react in that
environment
so that was one of the main reasons and
of course having a lab um
once that is already built we can use it
for research and development where we
evaluate specific protocols for example
or components
um but also we can use it for testing
itself so if we if we're engaged in ics
security testing we can maybe isolate or
t components and test them
in our lab that would be then ready for
that purpose
the first concept idea was building
a water treatment plant that is
comprised of a three-stage water
filtration system where water is left
with three stages until it is then
then cleaned that of course is all
driven by pumping stations
for the various filtration system stages
and in the back we will have a fully
virtualized i.t network to also
simulate the enterprise part um
that you would find in a water treatment
plant
this is how it first looked like on the
left-hand side you see the schema of how
the filtration would work so the water
runs through
all the three stages and then ends up in
that quality assurance part
on the right hand side where we do have
sensors to measure the quality of the
water
and if we determine that the water isn't
clean yet then we would
go send it back to the first
refrigeration stage and
let it run through it again and we did
for the proof of concepts
we did that with food coloring i also
had food colored water send it for the
stages and then
in the end it would be clean um or not
and then it would reiterate
through the stages as well on the right
hand side you see the basic setup of the
ot components as well that are included
in that model so we do have a plc a
siemens plc that is specifically
but we also have a couple of raspberry
pi's running quarters to simulate the
plc's
that we might have in addition to drive
the pumps for example
for the qa control and so on
and i'm showing you that that first
prototype of an ics firing range i'm
just going to show you this
video very briefly
once we were in progress of building
that lab i think we were pretty much
halfway already you've seen interests in
action now
uh we were contacted by one of our
partners that's cwd
in the netherlands and they got to talk
to
alex batastadt which is the ministry of
infrastructure and water management
in the netherlands and they they
required
a a model or a lab environment for their
forensics team to
to train their techniques and also you
know develop some of the
the skills that they need in their field
of expertise one of the things they do
in the netherlands is operating
basket bridges for example so this was a
prime example for us
um to to use as a as a model because
it's something that um
i think we can get an idea of how it's
working it's maybe something that can be
simulated and it's also very very
visible
then as well their requirement is trick
requirements were
having a mobile solution because they
wanted to move it between sites
and then of course come with a workshop
that is
a scenario base so the basket bridge
that we
were meant to build that should be
should be used in a way that is
versatile enough to run different
scenarios for different
forensic teams for example and this is
where the journey began
of building this bridge and most is
going to run you through the concepts
and the details of
how we came to the model that we
currently have so right
um as nico said we now changed from
the the first contacts that we had in
the first scenario water treatment plant
to the basket bridge and it just turns
out that this is
of course part of critical
infrastructure and it's not that easy to
get good information about this online
so i would like to give a very short but
brief and very nice shout out to the
lovely folks over at the florida
department of transportation
which were nice enough to make a bridge
maintenance reference manual privately
and freely available
and this really contained a lot of
useful information so thanks to those
guys
and here you can see our first 2d
concept drawing that we came up with
right
so here you can see we basically have
two
sides of a bridge they both have a leaf
and a counterweight on the ground and on
top of the road
we also have some traffic indicators we
have a traffic light
and we have also roads barriers
that then should actually block some
cars removing right
so we had this idea and we then
transferred it
to a 3d concept so how were we going to
build this
you can see a very brief a very very
short and
simplified representation of our 3d
concept
so here we got an aluminum frame on
wheels basically
it's divided divided into two parts in
the front you have the o2
ot components mounted onto a steel
plate and in the back there are some
some black wooden panels
now but in the back there's space for a
virtualization server that we can use to
host uh different networks and different
systems on
and on the top there we have our 3d
model
of our scenario those bridges that we
transferred from the 2d
concept drawing well we had this
but we wanted to build it right so we
did this
using the lovely magic of 3d printing
and
due to covet we couldn't do this in the
office and
we wanted to do quick iterations though
so we had to do this
at home and uh in my kitchen actually
much to the dismay of my wife to be
honest
it was noisy it took up a lot of space
but it was
worth it to be honest it was really cool
so what did we do with that um well we
started working
on the mechanical challenges here we can
see a
a very simple axle holder right
here we got a bearing that is mounted
between
a clamp more or less and that's fixed
with some screws
here are actually two iterations of the
same part and this actually
represented some challenges to us um one
of those being okay if you want to use
screws um
how big should those holes be because
you don't
want the the screw to just fall out
again because it's not tight enough but
you don't want to
have the plastic brake because it's too
tight right
um also you need to find out okay how
can we hold other parts with 3d
with 3d printed parts so we had to
figure this
this out but once we did figure this out
and we're satisfied with the
results we can then just proceed with
other challenges so
how would we use two of those axles and
do
and then so how would we print two of
those axles and connect those to
something in between that
can actually make use of this axle so
what you can see here is
more or less the center part of a bridge
leaf and then we
also want to control this right so we
have to attach
some kind of stepper motor to it and
then you have to engineer some gears
that you attach to it
and let me tell you this took quite some
iterations but we
kind of got it really really nice um
so but when you when you 3d print you
will run into problems
and so did we of course one of our
problems was just the
infamous spaghetti print this is just a
result of very poor bed adhesion as we
learned
and we just solved this by using a
mirror as a print bed and turns
out that this greatly increases bed
adhesion
another problem was warping so uh here a
3d printed part just cooled down
mid-print and then it came loose and
went places it turns out if you
make sure that you maintain a steady and
constant
temperature in the environment this will
probably not happen
so what we did was just get a nice and
big
enclosure for this for this 3d printer
and we also did some tests with abs and
pla
and found out that if you especially if
you mix between and change between those
filaments that you will run into
problems with your nozzles
so just changing them may help um
mitigating the problem of
of irregular extrusion of filament and
this was really a problem
um but yeah eventually eventually we got
to a point
where we could easily and and
steadily print new parts and here you
can see our
cad model of our vision of what the
bridge should look like
and actually some printed parts of it so
those were
very nice wins once we had those and
once we could assemble those to a full
model
and this actually is our full model but
what do you do when you have this full
model of course you have to put it all
together right
and that's what we did so we got this
full model actually we built two and we
printed two which took hours and hours
and hours
but then of course we we had to assemble
it into a
into a finished and then complete and
then and then also usable and mobile
ics lab so what you can see here is our
aluminum frame structure with the
metal plate for the ocs components built
onto it
and then of course we installed the ot
components onto it with an awful lot of
awful wiring to be honest
then we just installed the 3d models on
top of it
using some spaces that we printed and
designed
and bolting those to the to the wooden
plank there
and you know just for fancy points we
added those panels
also that you cannot reach into it
during operation of course but it looks
fancy
in my opinion um and of course you need
some lighting right
because if it's not very very very
bright inside
then you also want to have some lighting
in there so you can actually take a look
inside
or else it's really gonna get really
dark in there
and then well lastly we did a lot of
debugging
a lot of software debugging and hardware
debugging and this ate up a lot of time
so really so if you want to build
something something similar something
like this
just plan ahead of time that you will
use a lot of time for debugging stuff
will break and this will take a lot of
time to fix
but let's go into detail about the
components that are in there right
so first of all maybe most importantly
we got the hmi for the bridge control
um this actually represents all the
movable and controllable parts
of the bridge model on top right so for
everything like the lights and for the
movable
parts like the bridge leaf and the
barriers you've got
individual controls that actually
display their current status
color-coded text
so that's nice to have and actually we
pre-programmed it so that you can
tap onto them and control the whole
setup using just the hmi
of course this needs to be powered right
we got power supplies for
5 volts 12 volts and 24 volts these
power then are plc's
here we got a big s7 1500 plc
that coordinates the other plc's um
we got the s7 1200 plc's that drive the
drive all of the motors
and the s7 300 plc the legendary plc
that
controls the lights and of course
an awful lot of the wiring was due to
the motor drivers for the
leaves and barriers and if you could see
the insides of those cable tunnels you
would be shocked to see how much cable
fits into such a small place
last but not least we also got some
raspberry pis for the cctvs that are
mounted on top of the
bridge model
now having the 3d model printed and all
the writing done already with the ot
components the next
step in that process would be the entire
entire ide infrastructure
in the back as you remember so the idea
of of this lab is also to have it
for for workshops specifically for
forensic workshops
so we do need that simulation of an
enterprise network for example in the
scada network
to realistically work on
attacks that might have happened against
a basket bridge
to then investigate what went what was
going on
in that case um so um on that in that
model
um there is space for virtualization
server
and what we did is deploy
this infrastructure on it for that it
infrastructure part so comprised of the
enterprise network where we do have a
domain controller
we also have office workstations and a
visualization server which
then gets its data from a historian that
is running in the scada network and in
the top of this
this slide here um in in the scada
network
as well we have the operator workstation
where we can
observe and supervise the the bridge and
check the status of
of the the plcs for example but also
control it
remotely via the hmi we get access to
the camera feeds of the cctvs
next to the operator workstation we do
have the engineering workstation where
we have
tier deployed
the tool to then deploy the ladder logic
to the plc's
but also maintain the dot components
as such from there on
we connected we are connected to the
lower levels of the the bridge
infrastructure so this is not
virtualized anymore that is where we
actually have connected the
plc's um with the centralized
plc in the area supervision control
level including the hdmi that you did
see on the screens before for the manual
but also automatic control of the bridge
components
we do have the cctvs which are
essentially just pi cams
connected to the raspberry pi's of that
model
and then on the lower levels we have the
three different pcs to control the
barriers but also the leaf
motors to get the to change the lights
from green to red and so on this entire
mit and ot infrastructure then serves as
the basis for a scenario that we carry
out
in preparation for a for example for a
forensics workshop
the idea was really that we we go
through all the levels of the purging
model
to simulate an attack as it would most
likely be in a real life scenario
so you can assume that for example a
workstation in the enterprise zone is
compromised
through an email where then the
attackers
deploy beacons that communicate to a c2
server
somewhere outside then at some point the
attackers
once say they got that initial photo
they they probably they escalate the
local privileges
do some credential harvesting too funny
then get access to the scada network
where we do find the
operator's workstation but also the
engineering workstation with the
tier portal on it assuming that maybe
it's not fully patched
that could be one of the scenarios then
we can exploit some vulnerabilities
against the engineering workstation
gain access to it gain also access to
the leather logic we can then
deploy any arbitrary code
really on the plc that might cause some
disruption to the bridge operation
but also maybe severe harm to actually
the the people using the bridge so
just imagine that if cars are driving
over the bridge you you slowly
lift the leaves so there might be
accidents happening then
in that case that's the um
that's the scenario idea of our ics
firing lab so really
that's an attack that already took place
and then in the next step the forensics
team
would get their hands on the lab and
would need to investigate what happened
there
and so much talking now we're going to
show you the
the lab now um in the demo so the real
app that we actually built
and just give you a few ideas on what's
going on there
this is the complete 3d model now with
all the ot
components connected you do see once
it's powering on you see the hdmi
turning on you see the plc's already
blinking
they're also booting up they're all
connected by ethernet
together so they um also with a with a
server
behind the uh the steep plate that you
see there now we should see
the hdmi ui also being displayed in a
second there it is
and this is now where the bridge
automatically starts into its cycle so
that is just
sort of for demo purposes right now we
just cycle through the entire process
every 30 seconds
starting with the leaves lifting
and then the the lights going on for the
the ships and boats to pass with the
bridge
uh you can see in the in the hmi on the
left-hand side
that this is now green so that means the
leaves are open
the traffic lights are green so the
ships can pass
and soon the the lights will go red
again the leaves
lower and once they have done that
and in a second we'll see the barriers
go up just any second now
the barrier is going up you can't see
the lights for the cast unfortunately
but you can see it in hdmi that they
switch green as well
um and now they're the cars can drive
this is the automatic mode
you can at any time go into the the
manual mode
and then operate any of those components
manually really so you can just
lift one of the leaves or the other one
and then it varies
in whatever sequence you like
so let's get to the lessons learned
because there were quite some few
lessons to be learned
to be honest well let's start with the
ics lab setup
um to be honest designing this stuff
yourself you have to account for
the assembly of how you're going to
reach parts of how you are going to
to connect them how to put them into
place
we didn't really take this into account
and then had some
some problems some issues with really
complex
assembly it just took a lot of time
maintenance was hell
don't do this think of this ahead of
time this really
pays this is really good during research
of course we had to find
out then which hardware um has which
dependencies and which uh
compatibilities
this really required a lot of digging
through data sheets
uh you have to acquire those you have to
read those you have to understand those
you know if you're not an expert and we
have by no means
experts in this field this will take
your time
take this into account too and of course
if you want to use industrial hardware
you will
have to pay for licenses right
especially the the ides they come with
with heavy licenses
um and the practical issue just was that
we managed to kill two stepper motors
by overheating we supplied too many amps
and
did not really get rid of all the heat
we didn't manage to dissipate
at all and they just died on us and
this this was really really bad because
then you have to order new ones and you
have to install those again and
you know as i said the the assembly was
not really straightforward
so this just took a lot of time was
unnecessary but hey this happens
uh regarding 3d printing well as i
pointed out earlier
there will be mechanical design that
will be challenging
if you're not a mechanical engineer and
again by no means we are we are
mechanical engineers right um what you
saw in our setup was
not very very very complex i mean it was
just an accident right and
and two gears that drove this axle you
have no idea how long it took to just
build this and to test this out until it
just worked so if you're not very
familiar with it
again i think of it ahead of time
also if you want to iterate quickly you
have to take into account that printing
is really time consuming
overall just one side of those bridges
took multiple hundreds of hours to print
like they were about 200 hours to print
just one side
so the whole setup was about 400 hours
of printing time
so this is insane for quick iterations
well print small parts and test them
right
um also if you're not familiar with cad
yet
well um i suggest you will because this
is really the preferred
choice uh you can parametrize everything
and you can make quick but very
consistent changes to your designs
you can just iterate better with this um
in my opinion it's better suited than
let's say creative software like blender
for 3d modeling because this will give
you other problems right
and to to to add some numbers to our
uh to our project well we started in
january 2020
and we invested more than 1 000 hours of
manual work
into this and more than 900 hours of
printing time into this
well i just said we it took about 400
hours of
printing all the components that make up
our bridge scenario right
well of course we had to do a lot of
prototyping i have a huge box of failed
prints or prototyping prints
uh still in my kitchen laying around
there
taking up space again but anyways um
we also processed about
15 pounds of filament so take this into
account
as i said the software licenses are kind
of pricey well they are
we used the tia portal first for the
siemens plcs
which cost us in total three and three
thousand five hundred us dollars
um all the hardware combined cost
roughly
well 14k us dollars and you know for all
the
the coffee that we that we just consumed
during this this
project it was roughly about 570 us
dollars i know it's highly specific but
just rough
estimate right um and then of course we
broke stuff
as i said we broke two stepper motors we
actually managed to
break one plc and today just i'm so
today i'm not quite sure how i managed
to do this but this happened
after i replaced it everything was okay
so i think that's nice
um one motor driver died in the process
due to
wrong wiring but hey that happens and of
course
at several occasions we just lost our
sanity on the line right so
if you're going to go into such projects
and work on those and you are
not that familiar with all the technical
aspects
well uh you will have problems and you
will pay with your sanity
i hope that you got a good idea of of
what we had to go through in the past
six or seven months um and i hope that
you found this interesting and maybe
there's something for you as well to to
take away from it
um in any way if you are interested in
in this process
project specifically or if you are
working on a similar project and just
you know
do some follow-up questions uh feel free
to reach out to other
modes myself anytime also i wanted to
point you
at the musical region to ics security so
we do have that
that ics page there please have a look
at it
and maybe there's something else for you
there and again reach out to us
if there's more you want to know about
and last but not least
we do have a couple of blog posts coming
out um for this
ics firing range um so we want to go
more into detail of
of how the development went show some of
the close-ups maybe some of the
schematics and how the wiring works
and more specifically how the attack
scenario that we envisioned
comes together in that lab and also with
some more detail describe the the actual
uh difficulties
we had just forever to see and share
with the community as well
um thanks very much for joining and have
a great
day