Skip to content

Commit

Permalink
Config/runtime docs refinement (#215)
Browse files Browse the repository at this point in the history 
Significant update to the configuration and runtime doc
* Added metadata override detail
* Added missing mount point detail
* Added detail to runtime docs
* Misc fixes
  • Loading branch information
mbonsack authored and Ryan Faircloth committed Dec 6, 2019
1 parent 4ac6be4 commit 0e57952
Show file tree
Hide file tree
Showing 6 changed files with 181 additions and 56 deletions.
145 changes: 108 additions & 37 deletions docs/configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,8 @@ may hide this nuance.

## Archive File Configuration

This feature is designed to support "compliance" archival of all messages. To enable this feature update the Unit file
or docker compose to mount an appropriate host folder to the container folder ``/opt/syslog-ng/var/archive``.
The files will be stored in a folder structure using the naming pattern
This feature is designed to support "compliance" archival of all messages. Instructions for enabling this feature are included
in each "getting started" runtime document. The files will be stored in a folder structure using the naming pattern
``${YEAR}/${MONTH}/${DAY}/${fields.sc4s_vendor_product}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}.log"``.
This pattern will create one file per "vendor_product" per minute with records formatted using syslog-ng's EWMM template.

Expand Down Expand Up @@ -76,7 +75,8 @@ and/or move them to an archival system to avoid disk space failures.

## Syslog Source TLS Certificate Configuration

* Create a folder ``/opt/sc4s/tls``
* Create a folder ``/opt/sc4s/tls`` if not already done as part of the "getting started" process.
* Uncomment the appropriate mount line in the unit or yaml file (again, documented in the "getting started" runtime documents).
* Save the server private key in PEM format with NO PASSWORD to ``/opt/sc4s/tls/server.key``
* Save the server certificate in PEM format to ``/opt/sc4s/tls/server.pem``
* Add the following line to ``/opt/sc4s/env_file``
Expand All @@ -85,44 +85,115 @@ and/or move them to an archival system to avoid disk space failures.
SC4S_SOURCE_TLS_ENABLE=yes
```

## Override index or metadata based on host, ip, or subnet
## Log Path overrides of index or metadata

In some cases it is appropriate to override the default SC4S index or other Splunk metadata (such as an
source, host, or sourcetype) for a given data source. This is accomplished by the use of a lookup file that identifies these
source exceptions based on the log path used by the incoming message. These log path overrides are documented in the associated
"sources" document. For each data source, you will see a table of the form

| key | sourcetype | index | notes |
|----------------|----------------|----------------|----------------|
| foo_bar | fb_log | netfw | none |

In this case the key, `foo_bar`, will be an entry into the `splunk_indexes.csv` file that is populated in `/opt/sc4s/local/context` when SC4S
is run for the first time. The other columns show the default sourcetype and index when not overriden. This file contruct
is best shown with an example. Here is the table for Juniper Netscreen devices, from the "sources" document:

In some cases it is appropriate to re-direct events to an alternate index or append metadata (such as an
indexed field) based on PCI scope, geography, or other criterion. This is accomplished by the use
of a file that uniquely identifies these source exceptions via syslog-ng filters,
which maps to an associated lookup of alternate indexes, sources, or other metadata.
| key | sourcetype | index | notes |
|------------------------|---------------------|----------------|---------------|
| juniper_netscreen | netscreen:firewall | netfw | none |
| juniper_idp | juniper:idp | netfw | none |

* Get the filter and lookup files
```bash
cd /opt/sc4s/default
sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context_templates/compliance_meta_by_source.conf
sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context_templates/compliance_meta_by_source.csv
Here is a snippet from the `splunk_indexes.csv` file:

```
#juniper_sslvpn,index,netfw
juniper_netscreen,index,ns_index
#juniper_nsm,index,netfw
```

The columns in this file are `key`, `metadata`, and `value`. The `key` entries are
by default "commmented out", which is really a half-truth because CSV files don't allow comments. Therefore, to ensure there
is a match from the log path that references this file, be sure to remove the leading `#`. Once this is done, the following changes can be
made by adding one or more rows to the table and specifying one or more of the following `metadata`/`value` pairs for a given `key`:

* `index` to specify an alternate `value` for index
* `source` to specify an alternate `value` for source
* `sourcetype` to specify an alternate `value` for sourcetype (be _very_ careful when changing this; only do so if a downstream
TA is _not_ being used, or a custom TA (built by you) is being used.)

In this case, the `juniper_netscreen` key is "uncommented" (thereby enabling it), and the new index used for that data source will be
`ns_index`.

In general, for most deployments the index should be the only change needed; the defaults for the others should almost
never be overridden (particularly for "Out of the Box" data sources). Even then, care should be taken when considering an alternate
index, as the defaults for SC4S were chosen with best practices in mind.

This `csv` file can also be appended when building custom SC4S log paths (filters). Care should be taken during filter design to choose
appropriate index and sourctype defaults, so that admins are not compelled to override them.


## Override index or metadata based on host, ip, or subnet

In other cases it is appropriate to provide the same overrides but based on PCI scope, geography, or other criterion rather than globally.
This is accomplished by the use of a file that uniquely identifies these source exceptions via syslog-ng filters,
which maps to an associated lookup of alternate indexes, sources, or other metadata. In addition, (indexed) fields can also be
added to futher classify the data.

* The `conf` and `csv` files referenced below will be populated into the `/opt/sc4s/local/context` directory when SC4S is run for the first
time after being set up according to the "getting started" runtime documents.
* Edit the file ``compliance_meta_by_source.conf`` to supply uniquely named filters to identify events subject to override.
* Edit the file ``compliance_meta_by_source.csv`` to supply appropriate the field(s) and values.
The three columns in the table are `filter name`, `field name`, and `value`. `field name` obeys the following convention:
* ``fields.fieldname`` where `fieldname` will become the name of an indexed field with the supplied value
* ``.splunk.index`` to specify an alternate value for index
* ``.splunk.source`` to specify an alternate value for source
* Edit the file ``compliance_meta_by_source.csv`` to supply appropriate field(s) and values.

The three columns in the `csv` file are `filter name`, `field name`, and `value`. Filter names in the `conf` file must match one or more
corresonding `filter name` rows in the `csv` file. The `field name` column obeys the following convention:

* `.splunk.index` to specify an alternate `value` for index
* `.splunk.source` to specify an alternate `value` for source
* `.splunk.sourcetype` to specify an alternate `value` for sourcetype (be _very_ careful when changing this; only do so if a downstream
TA is _not_ being used, or a custom TA (built by you) is being used.)
* `fields.fieldname` where `fieldname` will become the name of an indexed field sent to Splunk with the supplied `value`

This file construct is best shown by an example. Here is a sample ``compliance_meta_by_source.conf`` file:

```
@version: 3.24
filter f_test_test {
host("something-*" type(glob)) or
netmask(192.168.100.1/24)
};
```
and the corresponding ``compliance_meta_by_source.csv`` file:

```
f_test_test,.splunk.index,"pciindex"
f_test_test,fields.compliance,"pci"
```

First off, ensure that the proper version string exists at the top of the `conf` file, and that the filter name(s) in the `conf` file match
one or more rows in the `csv` file. In this case, any incoming message with a hostname starting with `something-` or arriving from a netmask
of `192.168.100.1/24` will match the `f_test_test` filter, and the corresponding entries in the `csv` file will be checked for overrides.
In this case, the new index is `pciindex`, and an indexed field named `compliance` will be sent to Splunk, with it's value set to `pci`.
To add additional overrides, simply add another `filter foo_bar {};` stanza to the `conf` file, and add appropriate entries to the `csv` file
that match the filter name(s) to the overrides you deisre.

* IMPORTANT: The files above are actual syslog-ng config file snippets that get parsed directly by the underlying syslog-ng
process. Take care that your syntax is correct; for more information on proper syslog-ng syntax, see the syslog-ng
[documentation](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.24/administration-guide/57#TOPIC-1298086).
A syntax error will cause the runtime process to abort in the "preflight" phase at startup.

* For the Docker/Podman runtimes, update the docker/podman run command in the systemd unit file or the docker-compose to
include volumes mapping the files above.
* In the Unit file, add the following lines to the `ExecStart` command prior to `$SC4SIMAGE` then restart using the command
``sudo systemctl daemon-reload; sudo systemctl restart sc4s``

``
SC4S_UNIT_VP_CSV=-v /opt/sc4s/default/compliance_meta_by_source.csv:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.csv \
SC4S_UNIT_VP_CONF=-v /opt/sc4s/default/compliance_meta_by_source.conf:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.conf \
``

* For the Docker Swarm runtime, update the docker compose yml to add the following volume mounts to thee sc4s service and
redeploy the updated service using the command:
``docker stack deploy --compose-file docker-compose.yml sc4s``

``
- /opt/sc4s/default/compliance_meta_by_source.csv:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.csv
- /opt/sc4s/default/compliance_meta_by_source.conf:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.conf
``
Finally, to update your changes for the systemd-based runtimes, restart SC4S using the commands:
```
sudo systemctl daemon-reload
sudo systemctl restart sc4s
```

For the Docker Swarm runtime, redeploy the updated service using the command:
```
docker stack deploy --compose-file docker-compose.yml sc4s
```

## Data Durability - Local Disk Buffer Configuration

SC4S provides capability to minimize the number of lost events if the connection to all the Splunk Indexers goes down. This capability utilizes the disk buffering feature of Syslog-ng. SC4S receives a response from the Splunk HTTP Event Collector (HEC) when a message is received successfully. If a confirmation message from the HEC endpoint is not received (or a “server busy” reply, such as a “503” is sent), the load balancer will try the next HEC endpoint in the pool. If all pool members are exhausted (such as would occur if there were a full network outage to the HEC endpoints), events will queue to the local disk buffer on the SC4S Linux host. SC4S will continue attempting to send the failed events while it buffers all new incoming events to disk. If the disk space allocated to disk buffering fills up then SC4S will stop accepting new events and subsequent events will be lost. Once SC4S gets confirmation that events are again being received by one or more indexers, events will then stream from the buffer using FIFO queueing. The number of events in the disk buffer will reduce as long as the incoming event volume is less than the maximum SC4S (with the disk buffer in the path) can handle. When all events have been emptied from the disk buffer, SC4S will resume streaming events directly to Splunk.
Expand Down
2 changes: 2 additions & 0 deletions docs/gettingstarted/byoe-rhel7.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@ mkdir -p /opt/syslog-ng/etc/conf.d/local/config/
cp --verbose -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context/
cp --verbose -R -n /opt/syslog-ng/etc/local_config/* /opt/syslog-ng/etc/conf.d/local/config/
mkdir -p /opt/syslog-ng/var/data/disk-buffer/
mkdir -p /opt/syslog-ng/var/archive/
mkdir -p /opt/syslog-ng/tls/
```

* (Optional) Execute the preconfiguration shell script created above. You may also optionally execute it as part of the unit
Expand Down
20 changes: 16 additions & 4 deletions docs/gettingstarted/docker-swarm-general.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,10 @@ services:
volumes:
- /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z
- /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z
# Uncomment the following line if local disk archiving is desired
# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z
# Uncomment the following line if custom TLS certs are provided
# - /opt/sc4s/tls:/opt/syslog-ng/tls
# - /opt/sc4s/tls:/opt/syslog-ng/tls:z
```

* Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations.
Expand All @@ -76,7 +78,14 @@ of events in the event of network failure to the Splunk infrastructure.
If you are sure, after stopping SC4S, that all data has been sent, these files can be removed. They will be created
again upon restart.

* IMPORTANT: When creating the two directories above, ensure the directories created match the volume mounts specified in the
* Create the subdirectory ``/opt/sc4s/archive``. This will be used as a mount point for local storage of syslog events
(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration"
document for details on the directory structure the archive uses.

* Create the subdirectory ``/opt/sc4s/tls``. This will be used as a mount point for custom TLS certificates
(if the optional mount is uncommented above).

* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the
`docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup.

## Configure the SC4S environment
Expand Down Expand Up @@ -107,12 +116,15 @@ Log paths are preconfigured to utilize a convention of index destinations that a
* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your
environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in
this table that pertain to the individual data source filters that are included with SC4S.
* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further
information is covered in the "Log Path overrides" section of the Configuration document.

## Configure source filtering by source IP or host name

Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps
apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation.

* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start.
* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type.
* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents.
* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`.
Expand All @@ -122,8 +134,8 @@ apply to support such sources. To identify sources that require this step, refer
In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion.
The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to
the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file
lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in
the "Configuration" section.
lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is
covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document.

## Start/Restart SC4S

Expand Down

0 comments on commit 0e57952

Please sign in to comment.