The Bro Network Security Monitor

Transcription

Robin Sommer
International Computer Science Institute, &
Lawrence Berkeley National Laboratory
[email protected]
http://www.icir.org/robin
What is Bro?
2
What is Bro?
Packet Capture
2
What is Bro?
Packet Capture
Traffic Inspection
2
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
2
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
NetFlow
Log Recording
syslog
2
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
NetFlow
Log Recording
syslog
Flexibility
Abstraction
Data Structures
2
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
NetFlow
Log Recording
syslog
Flexibility
Abstraction
Data Structures
2
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
NetFlow
Log Recording
syslog
Flexibility
Abstraction
Abstraction
Data Structures
Structures
Data
2
What is Bro?
Packet Capture
Traffic Inspection
Attack Detection
“Domain-specific Python”
NetFlow
Log Recording
syslog
Flexibility
Abstraction
Abstraction
Data Structures
Structures
Data
2
Philosophy
Fundamentally different from other IDS.
Need to reset your idea of an IDS before starting to use Bro.
Real-time network analysis framework.
Primarily an IDS, but many use it for general traffic analysis.
Can accommodate a range of detection approaches.
Policy-neutral at the core.
Highly stateful.
Tracks extensive application-layer network state.
Supports forensics.
Extensively logs what it sees.
3
Bro History
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Vern writes 1st
line of code
4
Bro History
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
v0.2
Vern writes 1st
1st CHANGES
line of code
entry
v0.6
RegExps
Login analysis
v0.7a90
Profiling
State Mgmt
v0.8aX/0.9aX
SSL/SMB
STABLE releases
BroLite
LBNL starts
using Bro
operationally
v0.7a175/0.8aX
Signatures
SMTP
IPv6 support
User manual
v0.4
HTTP analysis
Scan detector
IP fragments
Linux support
v0.7a48
Consistent
CHANGES
v1.1/v1.2
when Stmt
Resource
tuning
Broccoli
DPD
v1.0
BinPAC
IRC/RPC analyzers
64-bit support
Sane version
numbers
0.8a37
Communication
Persistence
Namespaces
Log Rotation
v1.5
BroControl
Bro 2.0
New Scripts
Bro SDCI
Bro 2.1
IPv6
Input Framework
v1.4
DHCP/BitTorrent
HTTP entities
NetFlow
Bro Lite Deprecated
v1.3
Ctor expressions
GeoIP
Conn Compressor
4
Bro History
Host Context
Time Machine
Enterprise Traffic
Academic
Publications
TRW
State Mgmt.
Independ. State
USENIX Paper
Anonymizer
Active Mapping
Context Signat.
Stepping Stone
Detector
Bro Cluster
Shunt
BinPAC
DPD
2nd Path
Parallel Prototype
Input Framework
Autotuning
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
v0.2
Vern writes 1st
1st CHANGES
line of code
entry
v0.6
RegExps
Login analysis
v0.7a90
Profiling
State Mgmt
v0.8aX/0.9aX
SSL/SMB
STABLE releases
BroLite
LBNL starts
using Bro
operationally
v0.7a175/0.8aX
Signatures
SMTP
IPv6 support
User manual
v0.4
HTTP analysis
Scan detector
IP fragments
Linux support
v0.7a48
Consistent
CHANGES
v1.1/v1.2
when Stmt
Resource
tuning
Broccoli
DPD
v1.0
BinPAC
IRC/RPC analyzers
64-bit support
Sane version
numbers
0.8a37
Communication
Persistence
Namespaces
Log Rotation
v1.5
BroControl
Bro 2.0
New Scripts
Bro SDCI
Bro 2.1
IPv6
Input Framework
v1.4
DHCP/BitTorrent
HTTP entities
NetFlow
Bro Lite Deprecated
v1.3
Ctor expressions
GeoIP
Conn Compressor
4
“Who’s Using It?”
Installations across the US
Universities
Research Labs
Supercomputer Centers
Industry
Examples
Lawrence Berkeley National Lab
Indiana University
National Center for Supercomputing Applications
National Center for Atmospheric Research
... and many more sites
Recent User Meetings
Fully integrated into Security Onion
Popular security-oriented Linux distribution
Bro Workshop 2011 at NCSA
Bro Exchange 2012 at NCAR
Each attended by about 50 operators from
from 30-35 organizations
5
Deployment
Internal
Network
Internet
6
Deployment
Tap
Internet
Internal
Network
Bro
6
Deployment
Tap
Internet
Internal
Network
Bro
Runs on commodity platforms.
! Standard PCs & NICs.
Supports FreeBSD/Linux/OS X.
6
Example Logs
7
Example Logs
> bro -i en0
[ ... wait ...]
> cat conn.log
7
Example Logs
> bro -i en0
[ ... wait ...]
> cat conn.log
#fields ts
id.orig_h
id.orig_p
id.resp_h
id.resp_p proto
service
1144876741.1198
1144876612.6063
1144876596.5597
1144876606.7789
1144876741.4693
1144876745.6102
1144876605.6847
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
53115
53090
53051
53082
53116
53117
53075
82.94.237.218
198.189.255.82
193.203.227.129
198.189.255.73
82.94.237.218
66.102.7.99
207.151.118.143
80
80
80
80
80
80
80
http
http
http
http
http
http
http
tcp
tcp
tcp
tcp
tcp
tcp
tcp
duration
16.14929
4.437460
0.372440
0.597711
16.02667
1.004346
0.029663
7
Example Logs
> bro -i en0
[ ... wait ...]
> cat conn.log
#fields ts
id.orig_h
id.orig_p
id.resp_h
id.resp_p proto
service
1144876741.1198
1144876612.6063
1144876596.5597
1144876606.7789
1144876741.4693
1144876745.6102
1144876605.6847
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
53115
53090
53051
53082
53116
53117
53075
82.94.237.218
198.189.255.82
193.203.227.129
198.189.255.73
82.94.237.218
66.102.7.99
207.151.118.143
80
80
80
80
80
80
80
http
http
http
http
http
http
http
tcp
tcp
tcp
tcp
tcp
tcp
tcp
duration
16.14929
4.437460
0.372440
0.597711
16.02667
1.004346
0.029663
> cat http.log
7
Example Logs
> bro -i en0
[ ... wait ...]
> cat conn.log
#fields ts
id.orig_h
id.orig_p
id.resp_h
id.resp_p proto
service
1144876741.1198
1144876612.6063
1144876596.5597
1144876606.7789
1144876741.4693
1144876745.6102
1144876605.6847
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
53115
53090
53051
53082
53116
53117
53075
82.94.237.218
198.189.255.82
193.203.227.129
198.189.255.73
82.94.237.218
66.102.7.99
207.151.118.143
80
80
80
80
80
80
80
http
http
http
http
http
http
http
tcp
tcp
tcp
tcp
tcp
tcp
tcp
duration
16.14929
4.437460
0.372440
0.597711
16.02667
1.004346
0.029663
> cat http.log
#fields ts
1144876741.6335
1144876742.1687
1144876741.2838
1144876742.3337
1144876742.3337
1144876742.3337
1144876742.3337
1144876742.3338
1144876745.6144
id.orig_h
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
192.150.186.169
id.orig_p [...] host
53116
docs.python.org
53116
docs.python.org
53115
docs.python.org
53116
docs.python.org
53116
docs.python.org
53116
docs.python.org
53116
docs.python.org
53116
docs.python.org
53117
www.google.com
uri
/lib/lib.css
/icons/previous.png
/lib/lib.html
/icons/up.png
/icons/next.png
/icons/contents.png
/icons/modules.png
/icons/index.png
/
status_code
200
304
200
304
304
304
304
304
200
user_agent [...]
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
7
Example Logs
> bro -i en0
[ ... wait ...]
> cat conn.log
#fields ts
id.orig_h
id.orig_p
id.resp_h
id.resp_p proto
1144876741.1198 192.150.186.169 53115
82.94.237.218
80
tcp
1144876612.6063 192.150.186.169 53090
198.189.255.82 80
tcp
1144876596.5597 192.150.186.169 53051
193.203.227.129 80
tcp
[...]
host
uri53082
1144876606.7789
192.150.186.169
198.189.255.73 status_code
80
tcp
1144876741.4693
192.150.186.169
53116
82.94.237.218 200
80
tcp
docs.python.org
/lib/lib.css
1144876745.6102 192.150.186.169 53117
66.102.7.99
80
tcp
docs.python.org
/icons/previous.png
304
1144876605.6847
192.150.186.169
53075
207.151.118.143
80
tcp
service
duration
http
16.14929
http
4.437460
http
0.372440
user_agent
[...]
http
0.597711
http
16.02667
Mozilla/5.0
http
1.004346
Mozilla/5.0
http
0.029663
docs.python.org
> catdocs.python.org
http.log
docs.python.org
#fields ts
id.orig_h
docs.python.org
1144876741.6335 192.150.186.169
1144876742.1687 192.150.186.169
docs.python.org
1144876741.2838 192.150.186.169
1144876742.3337
192.150.186.169
docs.python.org
1144876742.3337 192.150.186.169
www.google.com
1144876742.3337
192.150.186.169
/lib/lib.html
200
/icons/up.png
304
/icons/next.png
304
id.orig_p [...] host
uri
/icons/contents.png
304
53116
docs.python.org /lib/lib.css
53116
docs.python.org /icons/previous.png
/icons/modules.png
304
53115
docs.python.org /lib/lib.html
53116
docs.python.org /icons/up.png
/icons/index.png
304
53116
docs.python.org /icons/next.png
/
200
53116
docs.python.org /icons/contents.png
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
status_code user_agent [...]
200 Mozilla/5.0
Mozilla/5.0
304
Mozilla/5.0
Mozilla/5.0
200
Mozilla/5.0
304 Mozilla/5.0
Mozilla/5.0
304
Mozilla/5.0
Mozilla/5.0
304
Mozilla/5.0
1144876742.3337
1144876742.3338
1144876745.6144
53116
53116
53117
304
304
200
192.150.186.169
192.150.186.169
192.150.186.169
docs.python.org
docs.python.org
www.google.com
/icons/modules.png
/icons/index.png
/
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
7
Identifying HTTP Servers
8
Server Addresses
a198-189-255-200.deploy.akamaitechnolgies.com
72.21.91.19
nuq04s07-in-f27.1e100.net
a184-28-157-55.deploy.akamaitechnologies.com
upload-lb.pmtpa.wikimedia.org
8
Server Addresses
HTTP Host Headers
72.21.91.19
a184-28-157-55.deploy.akamaitechnologies.com
upload-lb.pmtpa.wikimedia.org
ad.doubleclick.net
ad.yieldmanager.com
b.scorecardresearch.com
clients1.google.com
googleads.g.doubleclick.net
graphics8.nytimes.com
l.yimg.com
liveupdate.symantecliveupdate.com
mt0.google.com
pixel.quantserve.com
platform.twitter.com
profile.ak.fbcdn.net
s0.2mdn.net
safebrowsing-cache.google.com
static.ak.fbcdn.net
swcdn.apple.com
upload.wikimedia.org
www.facebook.com
www.google-analytics.com
www.google.com
8
File Content
9
File Content
192.168.1.102
192.168.1.102
192.168.1.102
192.168.1.102
192.168.1.102
GET
GET
GET
GET
GET
192.168.1.102
GET
192.168.1.102
GET
192.168.1.102
GET
192.168.1.102
GET
192.168.1.102
GET
192.168.1.102
GET
/skins-1.5/common/images/magnify-clip.png
image/png
/skins-1.5/monobook/external.png
image/png
/softw/90/update/avg9infoavi.ctf
text/plain
/softw/90/update/avg9infowin.ctf
text/plain
/softw/90/update/u7avi1777u1705ff.bin
application/x-dosexec
0210a9516dd34abc481683f877bd8680
/softw/90/update/u7avi1778u1705z7.bin
9bd8e3a274d8ada852bc3d9736116bf6
/softw/90/update/u7iavi2511u2510ff.bin application/x-dosexec
5e63f63fd955207610a56dbd89d8688f
/softw/90/update/u7iavi2512u2511z7.bin application/x-dosexec
a8e1ef490967ef7eb6641bef9eed4003
/softw/90/update/x8xplsb2_118c8.bin
e6915411c5550e9fbf33ef15fed75e5a
/softw/90/update/x8xplsc_149d148c8.bin application/x-dosexec
db5b04f3c45da4c0686c678bfd0e241c
/sports/
text/html
-
9
Software Logging
10
Software Logging
192.168.1.104
65.54.95.64
65.54.95.64
65.55.184.16
65.55.184.16
192.168.1.102
212.227.97.133
212.227.97.133
87.106.1.47
87.106.1.47
87.106.1.89
87.106.1.89
87.106.12.47
87.106.12.47
87.106.12.77
87.106.12.77
87.106.66.233
87.106.66.233
87.106.9.29
87.106.9.29
HTTP::BROWSER
HTTP::SERVER
HTTP::APPSERVER
HTTP::SERVER
HTTP::APPSERVER
HTTP::BROWSER
HTTP::SERVER
HTTP::APPSERVER
HTTP::SERVER
HTTP::APPSERVER
HTTP::SERVER
HTTP::APPSERVER
HTTP::SERVER
HTTP::APPSERVER
HTTP::SERVER
HTTP::APPSERVER
HTTP::SERVER
HTTP::APPSERVER
HTTP::SERVER
HTTP::APPSERVER
Windows-Update-Agent
Microsoft-IIS
6
ASP.NET Microsoft-IIS
7
ASP.NET SCSDK
6
0
Apache 2
2
PHP
5
2
Apache 2
2
PHP
5
2
Apache 2
2
PHP
5
2
Apache 2
2
PHP
5
2
Apache 2
2
PHP
5
2
Apache 2
0
PHP
4
3
Apache 2
2
PHP
5
2
Windows-Update-Agent
0
Microsoft-IIS/6.0
ASP.NET
0
Microsoft-IIS/7.0
ASP.NET
SCSDK-6.0.0
Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny3
PHP/5.2.6-1+lenny3
PHP/5.2.6-1+lenny3
PHP/5.2.6-1+lenny3
PHP/5.2.6-1+lenny3
PHP/5.2.6-1+lenny3
Apache/2.0.54 (Debian GNU/Linux)
PHP/4.3.10-22
PHP/5.2.6-1+lenny3
10
SSL Certificate Logging
11
SSL Certificate Logging
65.55.184.16
66.235.128.158
65.55.184.155
65.55.16.121
65.54.186.79
96.6.248.124
96.6.245.186
66.235.139.152
65.54.234.75
96.6.244.212
216.223.0.208
98.137.50.24
63.245.209.39
65.55.184.27
CN=Microsoft Secure Server Authority,DC=redmond,DC=corp,DC=microsoft,DC=com
CN=Sun Microsystems Inc SSL CA,OU=Class 3 MPKI Secure Server CA,OU=VeriSign
CN=VeriSign Class 3 Extended Validation SSL CA,OU=Terms of use at
CN=Akamai Subordinate CA 3,O=Akamai Technologies Inc,C=US
OU=Equifax Secure Certificate Authority,O=Equifax,C=US
CN=VeriSign Class 3 Secure Server CA,OU=Terms of use at
CN=Network Solutions Certificate Authority,O=Network Solutions L.L.C.,C=US
11
Brownian
12
Architecture
Packets
Network
13
Architecture
Events
Protocol Decoding
Event Engine
Packets
Network
13
Architecture
Logs
Analysis Logic
Notification
Policy Script Interpreter
Events
Protocol Decoding
Event Engine
Packets
Network
13
Architecture
Logs
Analysis Logic
Notification
“User Interface”
Events
Protocol Decoding
Event Engine
Packets
Network
13
Event Model
Web
Client
Request for /index.html
1.2.3.4/4321
Status OK plus data
Web
Server
5.6.7.8/80
14
Event Model
Web
Client
1.2.3.4/4321
SYN SYN
ACK
...
Status OK plus data
Stream of TCP packets
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
14
Event Model
Web
Client
1.2.3.4/4321
SYN SYN
Event
ACK
...
Status OK plus data
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
connection_established(1.2.3.4/4321 5.6.7.8/80)
14
Event Model
Web
Client
1.2.3.4/4321
SYN SYN
Event
ACK
...
Status OK plus data
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
TCP stream reassembly for originator
Event
http_request(1.2.3.4/4321 5.6.7.8/80, “GET”, “/index.html”)
14
Event Model
Web
Client
1.2.3.4/4321
SYN SYN
Event
ACK
...
Status OK plus data
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
Event
TCP stream reassembly for responder
Event
http_reply(1.2.3.4/4321 5.6.7.8/80, 200, “OK”, data)
14
Event Model
Web
Client
1.2.3.4/4321
SYN SYN
Event
ACK
...
Status OK plus data
ACK
ACK
...
Web
Server
5.6.7.8/80
ACK FIN FIN
Event
TCP stream reassembly for responder
Event
http_reply(1.2.3.4/4321 5.6.7.8/80, 200, “OK”, data)
Event
connection_finished(1.2.3.4/4321, 5.6.7.8/80)
14
Script Example: Matching URLs
Task: Report all Web requests for files called “passwd”.
15
Script Example: Matching URLs
Task: Report all Web requests for files called “passwd”.
event http_request(c: connection,
method: string,
original_URI: string,
unescaped_URI: string,
version: string)
{
if ( method == "GET" && unescaped_URI
NOTICE(...); # Alarm.
}
#
#
#
#
#
Connection.
HTTP method.
Requested URL.
Decoded URL.
HTTP version.
== /.*passwd/ )
15
Script Example: Scan Detector
Task: Count failed connection attempts per source address.
16
Script Example: Scan Detector
Task: Count failed connection attempts per source address.
global attempts: table[addr] of count &default=0;
event connection_rejected(c: connection)
{
local source = c$id$orig_h;
#
local n = ++attempts[source];
#
if ( n == SOME_THRESHOLD )
#
NOTICE(...);
#
}
Get source address.
Increase counter.
Check for threshold.
Alarm.
16
Distributed Scripts
17
Distributed Scripts
Bro comes with >10,000 lines of script code.
Prewritten functionality that’s just loaded.
Scripts generate all the logs.
Amendable to extensive customization and extension.
17
Bro Ecosystem
Tap
Internet
Internal
Network
Bro
18
Bro Ecosystem
Tap
Internal
Network
Internet
Bro
Control
Output
BroControl
User Interface
18
Bro Ecosystem
Tap
Internal
Network
Internet
Contributed
Scripts
Functionality
Bro
Control
Output
BroControl
User Interface
18
Bro Ecosystem
Tap
Internal
Network
Internet
Contributed
Scripts
Functionality
Bro
Control
Events
State
Other Bros
Output
BroControl
User Interface
18
Bro Ecosystem
Tap
Internal
Network
Internet
Contributed
Scripts
Functionality
Bro
Control
Events
State
Other Bros
Output
Events
Bro Client Communication Library
BroControl
Broccoli
User Interface
18
Bro Ecosystem
Tap
Internal
Network
Internet
Contributed
Scripts
Functionality
Bro
Control
Events
State
Other Bros
Output
Events
BroControl
Broccoli Python
Broccoli
User Interface
Broccoli Ruby
(Broccoli Perl)
18
Bro Ecosystem
Time Machine
Tap
Internet
Contributed
Scripts
Functionality
Bro
Control
Internal
Network
Tap
Events
State
Other Bros
Output
Events
BroControl
Broccoli Python
Broccoli
User Interface
Broccoli Ruby
(Broccoli Perl)
18
Bro Ecosystem
Time Machine
Tap
Internet
Contributed
Scripts
Functionality
Bro
Control
Internal
Network
Tap
Events
State
Other Bros
Output
Events
bro-aux
BinPAC
capstats
BroControl
BTest
tracesummary
Broccoli Python
Broccoli
User Interface
Broccoli Ruby
(Broccoli Perl)
18
Bro Ecosystem
Time Machine
Bro Distribution
Tap
Internet
bro-2.1.tar.gz
Contributed
Scripts
Functionality
Bro
Control
Internal
Network
Tap
Events
State
Other Bros
Output
Events
bro-aux
BinPAC
capstats
BroControl
BTest
tracesummary
Broccoli Python
Broccoli
User Interface
Broccoli Ruby
(Broccoli Perl)
18
Bro Ecosystem
Time Machine
Bro Distribution
Tap
Internet
bro-2.1.tar.gz
Contributed
Scripts
Functionality
Bro
Control
Internal
Network
Tap
Events
State
Other Bros
Output
Events
bro-aux
BinPAC
capstats
BroControl
BTest
tracesummary
Broccoli Python
Broccoli
User Interface
Broccoli Ruby
(Broccoli Perl)
http:://www.bro-ids.org/download
git://git.bro-ids.org
18
Bro Cluster Ecosystem
Tap
Internal
Network
Internet
Contributed
Scripts
Functionality
Bro
Control
Events
State
External Bro
Output
Events
bro-aux
BinPAC
capstats
BroControl
BTest
tracesummary
Broccoli Python
Broccoli
User Interface
Broccoli Ruby
(Broccoli Perl)
19
Tap
Internal
Network
Internet
Contributed
Scripts
Functionality
Bro
Control
Events
State
External Bro
Output
Events
bro-aux
BinPAC
capstats
BroControl
BTest
tracesummary
Broccoli Python
Broccoli
User Interface
Broccoli Ruby
(Broccoli Perl)
19
Tap
Internal
Network
Internet
LoadBalancer
Contributed
Scripts
Functionality
Bro
Control
Events
State
External Bro
Output
Events
bro-aux
BinPAC
capstats
BroControl
BTest
tracesummary
Broccoli Python
Broccoli
User Interface
Broccoli Ruby
(Broccoli Perl)
19
Tap
Internal
Network
Internet
LoadBalancer
Packets
Contributed
Scripts
Functionality
Bro
Bro
Bro
Bro
Control
Events
Bro
State
External Bro
Output
Events
bro-aux
BinPAC
capstats
BroControl
BTest
tracesummary
Broccoli Python
Broccoli
User Interface
Broccoli Ruby
(Broccoli Perl)
19
Tap
Internal
Network
Internet
LoadBalancer
Packets
Contributed
Scripts
Functionality
Bro
Control
bro-aux
BinPAC
Bro
Bro
Bro
Control
Output
Events
Bro
State
External Bro
Output
Events
capstats
BroControl
BTest
tracesummary
Broccoli Python
Broccoli
User
UserInterface
Interface
Broccoli Ruby
(Broccoli Perl)
19
Tap
Internal
Network
Internet
LoadBalancer
Packets
Contributed
Scripts
Functionality
Bro
Control
bro-aux
BinPAC
capstats
“Frontend”
Bro
Bro
Bro
“Workers”
Control
Output
Events
Bro
State
External Bro
Output
Events
“Manager”
BroControl
BTest
tracesummary
Broccoli Python
Broccoli
User
UserInterface
Interface
Broccoli Ruby
(Broccoli Perl)
19
A Production Load-Balancer
cFlow: 10GE line-rate, stand-alone load-balancer
10 Gb/s in/out
Web & CLI
Filtering capabilities
Available from cPacket
20
A Production Load-Balancer
cFlow: 10GE line-rate, stand-alone load-balancer
10 Gb/s in/out
Web & CLI
Filtering capabilities
Available from cPacket
20
Indiana University
Indiana University OpenFlow Deployment
Bloomington
v.1.0
CIC
Chicago
Chicago
Testlab
via
10 Gig
via
2 Nodes
8 OpenFlow
Switches
10 Gig
via DWDM
System
2 Nodes
Test
Servers
5 Nodes
IU Production
Deployment
Indianapolis
2 Nodes
IU Wireless
SSID:
OpenFlow
ICTC
Testpoint
InterOp lab
Layer 3 router
on OpenFlow
switches
Telcom
Bldn
Workshop
Monitoring
Indianapolis
VM Server
4 OpenFlow
switches
Informatics
East
Informatics
West
Lindley
Hall
IU
Core
Network
12 x 10G
6 x 10G
IDS
Cluster
12 servers
OpenFlow load balancer
Source: Indiana University
21
Indiana University
Indiana University OpenFlow Deployment
Bloomington
v.1.0
CIC
Chicago
Chicago
Testlab
via
10 Gig
via
2 Nodes
8 OpenFlow
Switches
10 Gig
via DWDM
System
2 Nodes
Test
Servers
5 Nodes
IU Production
Deployment
Indianapolis
2 Nodes
IU Wireless
SSID:
OpenFlow
ICTC
Testpoint
InterOp lab
Layer 3 router
on OpenFlow
switches
Telcom
Bldn
Workshop
Monitoring
Indianapolis
VM Server
4 OpenFlow
switches
Informatics
East
Informatics
West
Lindley
Hall
IU
Core
Network
12 x 10G
6 x 10G
IDS
Cluster
12 servers
OpenFlow load balancer
Source: Indiana University
21
External Events: Broccoli
22
“Auditing SSHD”
22
Solu5on&Overview&
“Auditing SSHD”
STUNNEL'
PARENT'
SSHD'
SSLOGMUX'
BROPIPE'
CHILD'
SSHD'
Source: Scott Campbell / NERSC
The Bro Network5&Security Monitor
22
NERSC reserves the right to remove any data at any time and/or transfer data to
other individuals working on the same or similar project once a user account is
deleted or a person no longer has a business association with NERSC.
NERSC Computer Use Policies Form
Account Usage
Users are not allowed to share their accounts with others.
Monitoring and Privacy
Users have no explicit or implicit expectation of privacy. NERSC retains the right
to monitor the content of all activities on NERSC systems and networks and
access any computer files without prior knowledge or consent of users, senders
or recipients. NERSC may retain copies of any network traffic, computer files or
messages indefinitely without prior knowledge or consent.
NERSC personnel and users are required to address, safeguard against and
report misuse, abuse and criminal activities. Misuse of NERSC resources can
lead to temporary or permanent disabling of accounts, loss of DOE allocations,
and administrative or legal actions.
revision 1.1 date: 2007/October/11 20:06:56
23
ons for the
Classroom
The Security
Fence
Presented by NIEonline.com and the Association of American Editorial Cartoonists (AAEC)
vs.
y
lines
g civil
me of
as been
n the
ttacks
the
d peoe up
otecttion
ts in
ction
toon illustrates the problem of
Cartoon Courtesy Clay Bennett / The Christian Science Monitor
24
Version 2.0 (Jan 2012)
25
Version 2.0 (Jan 2012)
Default scripts rewritten from scratch.
Focus on ease of use and operational deployment.
New logging infrastructure.
New build and packaging system.
New auto-documentation system (Broxygen).
Lots of bugs fixed.
Obsolete code removed.
New development infrastructure.
New regression testing framework.
New web server.
New mailing lists.
New logo.
25
Just released ...
26
Just released ...
Bro 2.1
Comprehensive IPv6 support.
Tunnel decapsulation.
New logging formats (DataSeries / ElasticSearch)
Input Framework
26
Input Framework Example: Blacklists
IP
Reason
Timestamp
66.249.66.1
Connected to honeypot 1333252748
208.67.222.222 Too many DNS requests 1330235733
192.150.186.11 Sent spam
1333145108
27
User Interface
28
User Interface
type Index: record { ip: addr; };
type Value: record { reason:
string;
timestamp: time; };
global blacklist: table[addr] of Value;
Input::add_table(source="blacklist.tsv", idx=Index,
val=Value, destination=blacklist);
(Syntax simplified.)
28
User Interface
type Index: record { ip: addr; };
type Value: record { reason:
string;
timestamp: time; };
global blacklist: table[addr] of Value;
Input::add_table(source="blacklist.tsv", idx=Index,
val=Value, destination=blacklist);
(Syntax simplified.)
event connection_established(c: connection)
{
if ( c$id$orig_h in blacklist )
alarm(...)
}
28
Current Research
29
Performace: 100 Gb/s
Now these sites need a
ing
monitoring solution ... Work
da
lo
E
G
0
0
1
a
n
o
t
e
k
c
a
P
c
with
balancer!
DOE/ESNet
100G Advanced Networking Initiative
Source: ESNet
Source: ESNet
30
Production Backbone in Planing
31
100 Gb/s Load-balancer
100Gbps
cFlow 100G
10Gb/s
Bro Cluster
100Gbps
cFlow 100G
Control
10Gb/s
API
Bro Cluster
Concurrent Analysis
Logs
Analysis Logic
Notification
Events
Protocol Decoding
Event Engine
Packets
Network
33
Concurrent Analysis
Logs
Notification
Single Thread
Analysis Logic
Events
Protocol Decoding
Event Engine
Packets
Network
33
Architecture
Notification
Scripting Language
Script Threads
Detection Logic
Events
Event Engine
Event Engine
Threads
Packet Analysis
Packets
Dispatcher
Packet Dispatcher (NIC)
Network
34
Architecture
Notification
Scripting Language
Script Threads
Detection Logic
Events
Event Engine
Event Engine
Threads
Packet Analysis
“Cluster in a Box”
Packets
Dispatcher
Network
34
Architecture
How to parallelize
a scripting language?
Notification
Scripting Language
Script Threads
Detection Logic
Events
Event Engine
Event Engine
Threads
Packet Analysis
“Cluster in a Box”
Packets
Dispatcher
Network
34
Parallel Event Scheduling
Thread
1
Thread
2
Thread
3
Thread
4
…
Thread
n
Queue
Queue
Queue
Queue
Queue
Queue
Threaded Script Interpreter
35
Thread
1
Thread
2
Thread
3
Thread
4
…
Thread
n
Queue
Queue
Queue
Queue
Queue
Queue
Conn A
http_request
35
Conn A
http_request
Thread
2
Thread
3
Thread
4
…
Thread
n
Queue
Queue
Queue
Queue
Queue
Thread
1
Queue
Conn A
http_reply
35
Conn A
http_request
Conn A
http_reply
Thread
3
Thread
4
…
Thread
n
Queue
Queue
Queue
Queue
Thread
2
Queue
Thread
1
Queue
Conn B
http_request
35
Conn A
http_request
Conn A
http_reply
Conn B
http_request
Thread
4
…
Thread
n
Queue
Queue
Queue
Thread
3
Queue
Thread
2
Queue
Thread
1
Queue
Orig X
conn_rejected
35
Conn A
http_request
Conn A
http_reply
Conn B
http_request
Orig X
conn_rejected
…
Thread
n
Queue
Queue
Thread
4
Queue
Thread
3
Queue
Thread
2
Queue
Thread
1
Queue
Orig Y
conn_rejected
35
Conn A
http_request
Conn A
http_reply
Thread
4
Conn B
http_request
Orig X
conn_rejected
Thread
n
Orig Y
conn_rejected
Queue
Queue
…
Queue
Queue
Thread
3
Queue
Thread
2
Queue
Thread
1
Orig X
conn_rejected
35
Conn A
http_request
Conn A
http_reply
Thread
4
Conn B
http_request
Orig X
conn_rejected
Thread
n
Orig Y
conn_rejected
Queue
Queue
…
Queue
Queue
Thread
3
Queue
Thread
2
Queue
Thread
1
Orig X
conn_rejected
Conn B
http_reply
35
Conn A
http_request
Conn A
http_reply
Thread
4
Conn B
http_request
Orig X
conn_rejected
Thread
n
Orig Y
conn_rejected
Queue
Queue
…
Queue
Queue
Thread
3
Queue
Thread
2
Queue
Thread
1
Orig X
conn_rejected
Conn B
http_reply
Conn A
http_request
35
Improving Bro’s Performance
Bottlenecks: Single-thread structure & Script interpretation
36
Improving Bro’s Performance
Bottlenecks: Single-thread structure & Script interpretation
A High-Level Intermediary Language for Traffic Inspection
Host Application
HILTI Machine Environment
OS Toolchain
Application
Core
C
Interface
Stubs
Analysis
Specification
Analysis
Compiler
HILTI
Machine Code
HILTI
Compiler
Native Object
Code
System
Linker
Native
Executable
LLVM
Runtime
Library
hiltic
hilti-build
36
BinPAC: “Yacc for Network Protocols”
37
BinPAC: “Yacc for Network Protocols”
type SMB_header = record {
protocol
: bytestring &length = 4;
command
: uint8;
status
: SMB_error(err_status_type);
flags
: uint8;
flags2
: uint16;
pad
: padding[12];
tid
: uint16;
pid
: uint16;
uid
: uint16;
mid
: uint16;
} &let {
err_status_type = (flags2 >> 14) & 1;
unicode
= (flags2 >> 15) & 1;
} &byteorder = littleendian;
type SMB_error (err_status_type: int) = case err_status_type of {
0 -> dos_error: SMB_dos_error;
1 -> status: int32;
};
type SMB_dos_error = record {
error_class
: uint8;
reserved
: uint8;
error
: uint16;
};
37
Next-generation BinPAC
38
type Message = unit(body_default: bool) {
headers
: list<Header(self)>;
end_of_hdr: /\r?\n/;
body
: Body([...])
};
HTTP Message
38
headers
end_of_hdr: /\r?\n/;
body
: Body([...])
const HeaderName = /[^:\r\n]+/;
const HeaderValue = /[^\r\n]*/;
};
};
HTTP Message
type Header = unit(msg: Message) {
name
: HeaderName;
: /:[\t ]*/;
content: HeaderValue;
: NewLine;
HTTP Header
38
headers
end_of_hdr: NewLine;
body
: Body(self, self.delivery_mode)
if ( self.has_body );
on end_of_hdr {
if ( self?.content_length )
self.delivery_mode = DeliveryMode::Length;
name
: HeaderName &convert=to_lower;
: /:[\t ]*/;
: NewLine;
on content {
if ( self.name == "content-length" ) {
msg.content_length = to_uint(self.content);
msg.has_body = True;
}
if ( self.content_type.startswith("multipart/") )
[... Parse boundary ...]
}
[...]
var
var
var
var
var
var
if ( self.name == "transfer-encoding" ) {
msg.transfer_encoding = self.content;
}
content_length: uint64;
content_type: bytes;
delivery_mode: DeliveryMode;
has_body: bool;
multipart_boundary: bytes;
transfer_encoding: bytes;
if ( self.name == "content-type" )
msg.content_type = self.content;
};
};
HTTP Message
HTTP Header
39
BinPAC++
headers
end_of_hdr: NewLine;
body
: Body(self, self.delivery_mode)
if ( self.has_body );
Streamlined usage.
name
: HeaderName &convert=to_lower;
: /:[\t ]*/;
: NewLine;
Adding semantics to syntax.
on end_of_hdr {
if ( self?.content_length )
self.delivery_mode = DeliveryMode::Length;
Decoding layers of protocols.
if (
on content {
Robust
error
handling.
self.content_type.startswith("multipart/") )
if ( self.name
[... Parse boundary ...]
== "content-length" ) {
msg.content_length = to_uint(self.content);
}
Fully usable outside
of
Bro.
}
[...]
Compiles to HILTI.if
var
var
var
var
var
var
content_length: uint64;
content_type: bytes;
delivery_mode: DeliveryMode;
has_body: bool;
multipart_boundary: bytes;
transfer_encoding: bytes;
( self.name == "transfer-encoding" ) {
msg.transfer_encoding = self.content;
}
if ( self.name == "content-type" )
msg.content_type = self.content;
};
};
HTTP Message
HTTP Header
39
Outlook & Conclusion
40
More Things in the Bro Queue ...
41
More Things in the Bro Queue ...
Comprehensive File Analysis
Intelligence Framework
Metrics Framework
Database interface
Packet Filter Framework
New/improved protocol analyzers
SMB/GridFTP/Modbus/DNP3
Reaction Framework
Load-balancer Interface
41
The Curse of Success ...
42
Success can be kind of problematic in research ...
Bro is now used operationally by many sites.
Demands of operations community hard to meet for small team.
42
Aiming to establish sustainable development model.
Modernize the system to make usage and contributions easier.
Develop a community around the project.
42
Aiming to establish sustainable development model.
Modernize the system to make usage and contributions easier.
Develop a community around the project.
NSF supports work through a 3-year engineering grant.
Bro changed a lot over the couples years.
Collaboration with National Center for Supercomputing Applications.
42
Target: Blue Waters @ NCSA
43
Target: Blue Waters @ NCSA
10 PF/s peak performa
nce
>1 PF/s sustained on
applications
>300,000 cores
>1 Petabyte memory
>10 Petabyte disk stora
ge
>0.5 Exabyte archival
storage
Hosted in 88,000-squa
re-foot facility
43
Summary
44
Summary
Bro will keep bridging the research/operations gap.
We have plenty more ideas ...
44
Summary
Long-term goal is a sustainable development model.
We are planing to offer commercial services and support.
44
Summary
www.bro-ids.org
blog.bro-ids.org
git.bro-ids.org
tracker.bro-ids.org
@Bro_IDS on Twitter
44
Summary
www.bro-ids.org
blog.bro-ids.org
git.bro-ids.org
tracker.bro-ids.org
@Bro_IDS on Twitter
44

The Bro Network Security Monitor

Transcription

Similar documents

Ein BitTorrent Analyzer für das Bro NIDS

DEREK HAMILL - Zimtu Capital Corp.