5. Database/File Configuration & Rights¶
5.1. Directories and rights¶
Meer keeps track of its actions in the meer.log
and its position in the Sagan or Suricata EVE output file
through a “waldo” file. This means that Meer will need an area to record data to. The default location of these
files is in the /var/log/meer
directory. You will need to create and assign this directory the proper rights.
Making directories & assigning rights:
sudo mkdir /var/log/meer
sudo chown suricata /var/log/meer # Chown to 'sagan' if using with Sagan!
You will need to adjust your meer.yaml
rights in the runas
option to match your permissions.
5.2. Setting up a new database¶
If you do not have a database already configured to receive alerts, the instructions below will help
you get started. First, you will need to create a database. For the purpose of this document
the target database will be known as example_database
. Database schemas are stored in the meer/sql
directory.
Creating the example_database
with MySQL/MariaDB:
mysqladmin -u root -p create example_database
mysql -u root -p example_database < sql/create_mysql
When using PostgreSQL, use the meer/sql/create_postgresql
schema file.
5.3. Using an old database¶
If you have a legacy database that you wish to convert, do the following with MySQL/MariaDB:
mysql -u root -p example_database < sql/extend_mysql
This will create new tables (https, flows, dns, etc).
5.4. Setting database rights¶
It is important to setup the proper rights when using Meer. Meer needs only INSERT and SELECT on all tables.
It will need INSERT, SELECT and UPDATE on the example_database.sensor
table.
With MySQL/MariaDB:
GRANT INSERT,SELECT ON example_database.* to myusername@127.0.0.1 identified by 'mypassword`;
GRANT INSERT,SELECT,UPDATE,INSERT ON example_database.sensor to myusername@127.0.0.1 identified by 'mypassword';
6. Meer ‘core’ configuration:¶
Meers operations are mainly controlled by the meer.yaml
file. The configuration file is split into two sections. The meer-core
controls how Meer processes incoming data from EVE files. The output-plugins
controls how data extracted from the EVE files is transported to a database backend.
6.1. ‘meer-core’ example¶
meer-core:
core:
hostname: "mysensor" # Unique name for this sensor (no spaces)
interface: "eth0" # Can be anything. Sagan "syslog", suricata "eth0".
runas: "suricata" # User to "drop privileges" too.
#runas: "sagan"
classification: "/etc/suricata/classification.config"
#classification: "/usr/local/etc/sagan-rules/classification.config"
meer_log: "/var/log/meer/meer.log" # Meer log file
# Meer can decode various types of data from within an "alert". This
# section enabled/disabled various JSON decoders.
metadata: enabled
flow: enabled
http: enabled
tls: enabled
ssh: enabled
smtp: enabled
email: enabled
json: enabled # Original JSON from EVE
# This enables the "fingerprint" option. When used in conjunction with the
# "fingerprint.rules" (https://github.com/quadrantsec/fingerprint-rules),
# this will record things like operating system type, type of system it is
# (client/server), etc. This data get routed differently and does not
# generate "alerts".
fingerprint: enabled
fingerprint_log: "/tmp/fingerprint.eve"
# "client_stats" are specific to Sagan and allow Sagan/Meer to record
# informatin about systems sending Sagan data. This has no affect on
# Suricata.
client_stats: disabled
# If "dns" is enabled, Meer will do reverse DNS (PTR) lookups of an IP.
# The "dns_cache" is the amount of time Meer should "cache" a PTR record.
# The DNS cache prevents Meer from doing repeated lookups of an
# already looked-up PTR record. This reduces over-loading DNS servers.
dns: enabled
dns_cache: 900 # Time in seconds.
# "health" checks are a set of signatures that are triggered every so
# often to ensure a sensor is up and operational. When these events
# are triggered, they are not stored into the database as normal alert
# data. For example, with MySQL/MariaDB output enabled, they update the
# "sensor.health" table with the current epoch time. Think of these
# events like a "ping" for your sensor. This can be useful for detecting
# when Meer, Suricata, or Sagan have "died" unexpectedly.
health: enabled
health_signatures: 20000001,20000002,20000003,20000004
waldo_file: "/var/log/meer/meer.waldo" # Where to store the last
# position in the
# "follow-eve" file.
lock_file: "/var/log/meer/meer.lck" # To prevent dueling processes.
follow_eve: "/var/log/suricata/alert.json" # The Suricata/Sagan file to monitor
#follow_eve: "/var/log/sagan/alert.json
6.2. ‘meer-core’ options¶
Below describes the options in the meer-core section of the meer.yaml
.
6.2.1. hostname¶
- This is stored in the database in the
sensor
table under thehostname
column. - The
interface
is appended to thehostname
. This option is required.
6.2.2. interface¶
The interface
is stored in the sensor
table appended to the hostname
and
interface
columns. This describes in what interface the data was collected. This can
be any descriptive string. For example, “eth0”, “syslog”, etc. This option is required.
6.2.3. runas¶
This is the user name the Meer process should “drop privileges” to. You will likely
want to run Meer as the same user name that is collecting information. For example,
“sagan” or “suricata”. The runas
can protect your system from security flaws in
Meer. Do not run as “root”. This option is required.
6.2.4. classification¶
The classification
option tells Meer where to find classification types. This file
typically ships with Sagan, Suricata, and Snort rules. It defines a ‘classtype’ (for
example, “attempt-recon”) and assigns a numeric priority to the event. This option is
required.
6.2.5. meer_log¶
The meer_log
is the location of the file for Meer to record errors and statistics
to. The file will need to be writable by the same user specified in the runas
option.
6.2.6. metadata¶
The metadata
option tells Meer to decode “metadata” from Suricata or Sagan. If
the “metadata” is present in the alert, Meer will decode it and store its contents
in memory for later use.
6.2.7. flow¶
The flow
option tells Meer to decode “flow” data from Suricata or Sagan. If
the “flow” JSON is present in the alert, Meer will decode it and store its contents
in memory for later use.
6.2.8. http¶
The http
option tells Meer to decode “http” data from Suricata or Sagan. If
the “http” JSON is present in the alert, Meer will decode it and store its contents
in memory for later use.
6.2.9. tls¶
The tls
option tells Meer to decode “tls” data from Suricata or Sagan. If
the “tls” JSON is present in the alert, Meer will decode it and store its contents
in memory for later use.
6.2.10. ssh¶
The ssh
option tells Meer to decode “ssh” data from Suricata or Sagan. If
the “ssh” JSON is present in the alert, Meer will decode it and store its contents
in memory for later use.
6.2.11. smtp¶
The smtp
option tells Meer to decode “smtp” data from Suricata or Sagan. If
the “smtp” JSON is present in the alert, Meer will decode it and store its contents
in memory for later use.
6.2.12. email¶
The email
option tells Meer to decode “email” data from Suricata or Sagan. If
the “email” JSON is present in the alert, Meer will decode it and store its contents
in memory for later use. This is not to be confused with smtp
. The data from
email
will contain information like e-mail file attachments, carbon copies, etc.
6.2.13. json¶
The json
option tells Meer to store the original JSON/EVE event. This is the
raw event that Meer has read in.
6.2.14. fingerprint¶
The fingerprint
option tells Meer to decode “fingerprint” rules and route the
data differently. Fingerprint rules do not work like normal rules. The data from
these rules is used to passively fingerprint systems for operating systems and types
(client/server). This information can be valuable to determine if an attack might have
been successful or not. Fingerprint rules are located at https://github.com/quadrantsec/fingerprint-rules.
6.2.15. fingerprint_log¶
When fingerprint rules fire, this is the log file that is create and data sent to. This log file format is an JSON (EVE) log file and is meant to be routed to a Elasticsearch back end. The idea is to store this information for historical purposes.
6.2.16. dns¶
The dns
option tells Meer to perform a DNS PTR (reverse) record lookup of the
IP addresses involved in an alert. This option is useful because it records the
DNS record at the time the event occurred.
6.2.17. dns_cache¶
When dns
is enabled, Meer will internally cache records to avoid repetitive
lookups. For example, if 1000 alerts come in from a single IP address, Meer
will look up the DNS PTR record one time and use the cache for the other 999
times. This saves on lookup time and extra stress on the internal DNS server. If you
do not want Meer to cache DNS data, simply set this option to 0. The dns_cache
time is in seconds.
6.2.18. health¶
The health
option is a set of signatures used to monitor the health of Meer and
your Sagan or Suricata instances. When enabled, Meer will treat certain Sagan and
Suricata signatures as “health” indicators rather than normal alerts. When a
“health” signature occurs, Meer updates the sensor
table health
column
with the epoch time the health signature triggered. This can be useful in quickly
determining if a sensor is down or behind (back logged) on alerts.
6.2.19. health_signatures¶
When health
is enabled, this option supplies a list of signature IDs (sid) to
Meer of Suricata or Sagan “health” signatures.
6.2.20. waldo_file¶
The waldo_file
is a file that Meer uses to keep track of its last location within
a EVE/JSON file. This keeps Meer from re-reading data in between stop/starts. This
option is required.
6.2.21. lock_file¶
The lock_file
is used to help avoid multiple Meer processes from processing the
same data. The lock_file should be unique per Meer instance. The lock file contains
the process ID (PID) of instance of Meer. This option is required.
6.2.22. follow_eve¶
The follow_eve
option informs Meer what file to “follow” or “monitor” for new
alerts. You will want to point this to your Sagan or Suricata “alert” EVE output file.
You can think of Meer “monitoring” this file similar to how “tail -f” operates.
This option is required.
7. Output Plugins¶
7.1. SQL output-plugins example¶
Below is an example of the “output-plugins” from the meer.yaml
. This section controls
the SQL output.
output-plugins:
# MySQL/MariaDB output - Stores data from Suricata or Sagan into a semi-
# traditional "Barnyard2/Snort"-like database.
sql:
enabled: yes
driver: mysql # "mysql" or "postgresql"
port: 3306 # Change to 5432 for PostgreSQL
debug: no
server: 127.0.0.1
port: 3306
username: "XXXX"
password: "XXXXXX"
database: "snort_test"
# Automatically reconnect to the database when disconnected.
reconnect: enabled
reconnect_time: 10
# Store decoded JSON data that is similar to Unified2 "extra" data to the
# "extra" table.
extra_data: enabled
# Store extra decoded JSON metadata from Suricata or Sagan. This requires
# your database to have the metadata, flow, http, etc. tables. If all are
# disabled, Meer will store data in strictly a Barnyard2/Snort method.
# If you want to store this decoded information, and you likely do, make
# sure you have the decoders enabled in the "core" section of this Meer
# configuration file!
metadata: enabled
flow: enabled
http: enabled
tls: enabled
ssh: enabled
smtp: enabled
email: enabled
json: enabled
# If you would like Meer to mimic the legacy "reference" tables from
# Snort/Barnyard2, enable it here. If you are using more than one database
# to store Suricata or Sagan data, you will likely want to leave this
# disabled. The legacy reference system is not very efficient and there are
# better ways to keep track of this data. This is also a memory hog and
# performance killer. See tools/reference_handler/reference_handler.pl to
# build a centralized reference table.
reference_system: disabled
sid_file: "/etc/suricata/rules/sid-msg.map" # Created with "create-sidmap"
reference: "/etc/suricata/reference.config"
#sid_file: "/usr/local/etc/sagan-rules/sagan-sid-msg.map"
#reference: "/usr/local/etc/sagan-rules/reference.config"
7.1.1. enabled¶
When this option is set to yes
or no
, it enables or disables the SQL section of
the Meer output plugin.
7.1.2. driver¶
This controls what SQL database driver Meer will use. Valid types are mysql
(for both
MySQL and MariaDB) and postgresql
.
7.1.3. port¶
The port the target SQL server is listening on.
7.1.4. server¶
The IP address of the SQL server.
7.1.5. debug¶
When debug
is enabled, Meer will display SQL statements and transactions to stdout and to the
meer_log
. This can be useful for debugging SQL errors and issues. By default, this is disabled.
7.1.6. username¶
The username to use during authentication with the SQL database.
7.1.7. password¶
The password to use during authentication with the SQL database.
7.1.8. reconnect¶
If Meer encounters an issue with connecting to the SQL database, if this
option is enabled
, Meer will continually try to reconnect until it is
successful.
7.1.9. reconnect_time¶
This is how long to pause, in seconds, before attempting to reconnect to the
SQL database if the reconnect
option is enabled.
7.1.10. extra_data¶
When the extra_data
option is enabled, Meer will record certain information
(XFF, DNS data, SMTP data, etc) in the legacy extra
table.
7.1.11. metadata¶
This option controls Meer’s ability to record decoded alert metadata to the metadata
SQL table. If “metadata” is detected within the EVE/JSON and the metadata
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the metadata
SQL table.
7.1.12. flow¶
This option controls Meer’s ability to record decoded alert flow to the flow
SQL table. If “flow” is detected within the EVE/JSON and the flow
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the flow
SQL table.
7.1.13. http¶
This option controls Meer’s ability to record decoded alert http to the http
SQL table. If “http” is detected within the EVE/JSON and the http
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the http
SQL table.
7.1.14. tls¶
This option controls Meer’s ability to record decoded alert tls to the tls
SQL table. If “tls” is detected within the EVE/JSON and the tls
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the tls
SQL table.
7.1.15. ssh¶
This option controls Meer’s ability to record decoded alert ssh to the ssh
SQL table. If “ssh” is detected within the EVE/JSON and the ssh
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the ssh-client``and ``ssh-server
SQL tables.
smtp ~~~
This option controls Meer’s ability to record decoded alert smtp to the smtp
SQL table. If “smtp” is detected within the EVE/JSON and the smtp
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the smtp
SQL table.
7.1.16. email¶
This option controls Meer’s ability to record decoded alert email to the email
SQL table. If “email” is detected within the EVE/JSON and the email
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the email
SQL tables. This is not to be confused with the smtp
table.
7.1.17. reference_system¶
The reference_system
allows Meer to store alert reference data in a traditional
“Barnyard2” format. If you are using a single database for all events, this
option might be useful to you. If you are using UIs like Snorby, Squeel, etc.
you will likely want to enable this option. If you are using multiple databases,
then consider looking at the “reference_handler.pl” script that ships with Meer.
7.1.18. sid_file¶
The sid_file
is a legacy “signature message map” file that points signature
IDs to their references. If you want to use the legacy reference_system
,
you will need a “signature message map” (sid_file
) for Meer to read.
7.2. “pipe” output¶
Below is an example of the “pipe” output plugin. This takes data being written to the EVE file and puts it into a named pipe (FIFO). This can be useful if you want a third party program (for example, Sagan - https://sagan.io) to analyze the data.
pipe:
enabled: no
pipe_location: /var/sagan/fifo/sagan.fifo
pipe_size: 1048576 # System must support F_GETPIPE_SZ/F_SETPIPE_SZ
metadata: enabled
# Below are the "event_types" from Suricata/Sagan. This tells Meer what to send
# to the named pipe/FIFO.
alert: enabled
files: enabled
flow: enabled
dns: enabled
http: enabled
tls: enabled
ssh: enabled
smtp: enabled
fileinfo: enabled
dhcp: enabled
7.2.1. enabled¶
Enabled/disabled the ‘pipe’ output.
7.2.2. pipe_location¶
Location of the named pipe on the file system.
7.2.3. pipe_size¶
Number of bytes will set the size of the named pipe/FIFO to.
7.2.4. metadata¶
This option controls Meer’s ability to record decoded alert metadata to the named pipe.
If “metadata” is detected within the EVE/JSON and the metadata
decoder is enabled (controlled in the meer-core
), then it will be recorded to the named
pipe.
7.2.5. flow¶
This option controls Meer’s ability to record decoded alert flow to named pipe.
If “flow” is detected within the EVE/JSON and the flow
decoder is enabled (controlled in the meer-core
), then it will be recorded to the
named pipe.
7.2.6. http¶
This option controls Meer’s ability to record decoded alert http to the named pipe.
If “http” is detected within the EVE/JSON and the http
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the named pipe.
7.2.7. tls¶
This option controls Meer’s ability to record decoded alert tls to the named pipe.
If “tls” is detected within the EVE/JSON and the tls
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the named pipe.
7.2.8. ssh¶
This option controls Meer’s ability to record decoded alert ssh to the named pipe.
If “ssh” is detected within the EVE/JSON and the ssh
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the named pipe.
smtp ~~~
This option controls Meer’s ability to record decoded alert smtp to the named pipe.
If “smtp” is detected within the EVE/JSON and the smtp
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the named pipe.
7.2.9. email¶
This option controls Meer’s ability to record decoded alert email to the named pipe.
If “email” is detected within the EVE/JSON and the email
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the named pipe. This is not to be confused with the smtp
table.
7.2.10. fileinfo¶
This option controls Meer’s ability to record decoded alert fileinfo to the named pipe.
If “fileinfo” is detected within the EVE/JSON and the fileinfo
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the named pipe.
7.2.11. dhcp¶
This option controls Meer’s ability to record decoded alert dhcp to the named pipe.
If “dhcp” is detected within the EVE/JSON and the dhcp
decoder is enabled (controlled in the meer-core
), then it will be recorded
to the named pipe.
7.3. “external” output¶
This option allows signatures to call “external” programs. For example, if a signature the
proper “metadata” (metadata: meer external
or a set policy), Meer will fork a copy
of the specified program and pass the EVE via stdin. This feature can be useful for creating
custom firewalling routines or routing data to alternate programs. The “external” program
can be written in any language that suites you.
###########################################################################
# external
#
# EVE data (JSON) is passed via stdin to the external program. The
# external program can be written in any language you choose (shell script,
# Python, Perl, etc).
#
# This can be useful for automatic firewalling, building block lists,
# replicating "snortsam" functionality, etc. See the "tools/external"
# directory for example routines that use this feature.
#
# If this option is enabled, any rule that has the metadata of "meer
# external" (ie - "metadata:meer external") will cause the external script
# to be executed. Execution can also be controlled by Snort metadata
# "policies".
###########################################################################
external:
enabled: no
debug: no
# Execution of an external program based on metadata "policy". When Meer
# encounters a "policy" (security-ips, balanced-ips, connectivity-ips,
# and max-detect-ips), Meer will execute the specified routine.
# Currently only Snort rules have these types of polices. This can be
# useful when you want to execute an external script that will to "block"
# or "firewall" based off the policy types. This section only applies if
# you are using Suricata with Snort rules. Snort's polices are
# below:
# connectivity-ips - You run a lot of real time applications (VOIP,
# financial transactions, etc), and don't want to run any rules that
# could affect the current performance of your sensor. The rules in this
# category make snort happy, additionally this category focuses on the high
# profile most likely to affect the largest number of people type of
# vulnerabilities.
# balanced-ips - You are normal, you run normal stuff and you want normal
# security protections. This is the best policy to start from if you are
# new, old, or just plain average. If you don't have any special
# requirements for super high speeds or super secure networks start here.
# security-ips - You don't care about dropping your bosses email, everything
# in your environment is tightly regulated and you don't tolerate people
# stepping outside of your security policy. This policy hates on IM, P2P,
# vulnerabilities, malware, web apps that cause productivity loss, remote
# access, and just about anything not related to getting work done.
# If you run your network with an iron fist start here.
# I can't seem to find any documentation on what "max-detect-ips" is :(
policy-security-ips: enabled
policy-max-detect-ips: enabled
policy-connectivity-ips: enabled
policy-balanced-ips: enabled
program: "/usr/local/bin/external_program"
7.3.1. enabled¶
Keyword is used to enable/disable external
output.
7.3.2. debug¶
When enabled, this option will display and log debugging information.
7.3.3. policy-security-ips¶
Execute external
program when the policy-security-ips
is encountered.
7.3.4. policy-max-detect-ips¶
Execute external
program when the policy-max-detect-ips
is encountered.
7.3.5. policy-connectivity-ips¶
Execute external
program when the policy-connectivity-ips
is encountered.
7.3.6. policy-balanced-ips¶
Execute external
program when the policy-balanced-ips
is encountered.
7.3.7. program¶
external
program to execute when conditions are met.
7.4. Redis output¶
This controls how Meer logs to a Redis database. Meer can record alert records to
Redis similar to how Suricata with Redis support enabled does. Redis is also used
as a temporary storage engine for client_stats
(Sagan only) and fingerprint
data if enabled.
###########################################################################
# "redis" allows you to send Suricata/Sagan EVE data to a Redis database.
# This will mimic the way Suricata writes EVE data to Redis with the
# exception of "client_stats" which is a Sagan specific processor.
###########################################################################
redis:
enabled: no
debug: no
server: 127.0.0.1
port: 6379
batch: 10 # Batch/pipelining mode. Max is 100. 1 == no batching.
key: "suricata" # Default 'key' or 'channel' to use.
mode: list # How to publish data to Redis. Valid types are list/lpush,
# rpush, channel|publish.
# This controls event_types to send to Redis.
alert: enabled
files: enabled
flow: enabled
dns: enabled
http: enabled
tls: enabled
ssh: enabled
smtp: enabled
fileinfo: enabled
dhcp: enabled
# Fingerprint data can be temporarily stored in a Redis database. When an alert
# fires, this information can be used to determine the targets operating system,
# type (client/server), etc. This can be useful in determining the validity of
# an event. If used in conjunction with the SQL output, the fingerprint data for
# the targeted system is stored in the 'fingerprint' table.
fingerprint: enabled
# This controls sending Sagan client tracking data to Redis. This has no affect
# on Suricata systems.
client_stats: disabled
7.4.1. enabled¶
Enable or disable the Redis output.
7.4.2. debug¶
Enable or disabled Redis debugging.
7.4.3. server¶
The Redis server address you want to store data to.
7.4.4. port¶
Port of the target Redis server.
7.4.5. batch¶
The batch
is the amount of data to collect before sending it to Redis. This has no
affect when using Redis with either client_stats
or fingerprint
data.
7.4.6. key¶
The key
is the default Redis channel or key to use.
7.4.7. mode¶
The mode
controls how data is stored to Redis. Valid options are list
, lpush
,
rpush
, channel
or publish
. The default is list
. The method Meer stores the
data is compatible with Suricata’s Redis output format. Note; This option does not have any
affect on client_stats
or fingerprint
recording.
7.4.8. alert¶
Enable or disable storing alert
data into Redis.
7.4.9. files¶
Enable or disable storing files
data into Redis.
7.4.10. flow¶
Enable or disable storing flow
data into Redis.
7.4.11. dns¶
Enable or disable storing dns
data into Redis.
7.4.12. http¶
Enable or disable storing http
data into Redis.
7.4.13. tls¶
Enable or disable storing tls
data into Redis.
7.4.14. ssh¶
Enable or disable storing ssh
data into Redis.
7.4.15. smtp¶
Enable or disable storing smtp
data into Redis.
7.4.16. fileinfo¶
Enable or disable storing fileinfo
data into Redis.
7.4.17. dhcp¶
Enable or disable storing dhcp
data into Redis.
7.4.18. fingerprint¶
Enable or disable storing fingerprint
data in the Redis database. This is a temporary
storage system for fingerprint
data. This allows correlation between device fingerprints
(ie - operating systems, devices types, etc) with alerts.
7.4.19. client_stats¶
This is a Sagan only option. This option temporarily stores devices that are sending Sagan logs along with an example log entry. This has no affect with Suricata.