Skip to content

Commit

Permalink
Support generation of multiple HEC destinations dynamically (#553)
Browse files Browse the repository at this point in the history
This change add the ability to create additional HEC destination to use as alternates this allows the administrator to send events from SC4S to more than one instance of Splunk Enterprise, Splunk Enterprise Cloud, or Splunk DSP
  • Loading branch information
Ryan Faircloth authored and GitHub committed Jul 8, 2020
1 parent 22405a5 commit 50f01ea
Show file tree
Hide file tree
Showing 5 changed files with 115 additions and 67 deletions.
52 changes: 40 additions & 12 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,35 +21,63 @@ syslog.

| Variable | Values | Description |
|----------|---------------|-------------|
| SC4S_DEST_SPLUNK_HEC_GLOBAL | yes | Send events to Splunk using HEC |
| SC4S_DEST_SPLUNK_HEC_GLOBAL | yes | Send events to Splunk using HEC. This applies _only_ to the primary HEC destination. |
| SC4S_DEST_SPLUNK_HEC_CIPHER_SUITE | comma separated list | Open SSL cipher suite list |
| SC4S_DEST_SPLUNK_HEC_SSL_VERSION | comma separated list | Open SSL version list |
| SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE | path | Custom trusted cert file |
| SC4S_DEST_SPLUNK_HEC_TLS_VERIFY | yes(default) or no | verify HTTP(s) certificate |
| SC4S_DEST_SPLUNK_HEC_WORKERS | numeric | Number of destination workers (default: 10 threads). This should rarely need to be changed; consult sc4s community for advice on appropriate setting in extreme high- or low-volume environments. |
| SC4S_DEST_SPLUNK_INDEXED_FIELDS | facility,<br>severity,<br>container,<br>loghost,<br>destport,<br>fromhostip,<br>proto<br><br>none | List of sc4s indexed fields that will be included with each event in Splunk (default is the entire list except "none"). Two other indexed fields, `sc4s_vendor_product` and `sc4s_syslog_format`, will also appear along with the fields selected via the list and cannot be turned on or off individually. If no indexed fields are desired (including the two internal ones), set the value to the single value of "none". When setting this variable, separate multiple entries with commas and do not include extra spaces.<br><br>This list maps to the following indexed fields that will appear in all Splunk events:<br>facility: sc4s_syslog_facility<br>severity: sc4s_syslog_severity<br>container: sc4s_container<br>loghost: sc4s_loghost<br>dport: sc4s_destport<br>fromhostip: sc4s_fromhostip<br>proto: sc4s_proto

## Alternate Destination Configuration
* NOTE: When using alternate HEC destinations, the destination operating paramaters outlined above (`CIPHER_SUITE`, `SSL_VERSION`, etc.) can be
individually controlled per `DESTID` (see "Configuration of Additional Splunk HEC Destinations" immediately below). For example, to set the number of workers
for the alternate HEC destination `d_hec_FOO` to 24, set `SC4S_DEST_SPLUNK_HEC_FOO_WORKERS=24`.

Alternate destinations other than HEC can be configured in SC4S. Global and/or source-specific forms of the
variables below can be used to send data to alternate destinations.
## Creation of Additional Splunk HEC Destinations

* NOTE: The administrator is responsible for ensuring that the alternate destinations are configured in the
local mount tree, and that syslog-ng properly parses them.
Additional Splunk HEC destinations can be dynamically created through environment variables. The use of these destinations can then be controlled
along with other user-defined destinations on a global or per-source basis (see "Alternate Destination Use" immediately below).

| Variable | Values | Description |
|----------|---------------|-------------|
| SPLUNK_HEC_ALT_DESTS | Comma or space-separated UPPER case list of destination ids | destination IDs are UPPER case single-word friendly strings used to identify the new destination, which will be named with the destination id appended, for example `d_hec_FOO` |
| SPLUNK_HEC&lt;DESTID&gt;_URL | url | Example: `SPLUNK_HEC_FOO_URL=https://splunk:8088`. `DESTID` must be a member of the list configured in `SPLUNK_HEC_ALT_DESTS` configured above |
| SPLUNK_HEC&lt;DESTID&gt;_TOKEN | string | Example: `SPLUNK_HEC_BAR_TOKEN=&lt;token&gt;`. `DESTID` must be a member of the list configured in `SPLUNK_HEC_ALT_DESTS` configured above |

When set above, the destinations will be created with the `DESTID` appended, for example: `d_hec_FOO`. These destinations can then be
specified below (along with any other destinations created locally) either globally or per source.

* NOTE: The `DESTID` specified in the `URL` and `TOKEN` variables above _must_ match the `DESTID` entries enumerated the `SPLUNK_HEC_ALT_DESTS` list.
Failure to do so will cause destinations to be created without proper HEC parameters.

* NOTE: Do not include `d_hec` in any list of alternate destinations. The configuration of the default HEC destination is configured
separately from that of the alternates below.
* NOTE: Additional Splunk HEC destinations will _not_ be tested at startup. It is the responsiblity of the admin to ensure that additional destinations
are provisioned with the correct URL(s) and tokens to ensure proper connectivity.

* NOTE: The disk and CPU requirements will increase proportionally depending on the number of additional HEC destinations in use (e.g. each HEC
destination will have its own disk buffer).

## Alternate Destination Use

All alternate destinations (including alternate HEC destinations) are configured for use in SC4S through variables. Global and/or source-specific forms of
the variables below can be used to send data to alternate destinations.

* NOTE: The administrator is responsible for ensuring that any non-HEC alternate destinations are configured in the
local mount tree, and that syslog-ng properly parses them.

* NOTE: Do not include the primary HEC destination (`d_hec`) in any list of alternate destinations. The configuration of the primary HEC destination
is configured separately from that of the alternates below. However, _alternate_ HEC destinations (e.g. `d_hec_FOO`) should be configured below, just
like any other user-supplied destination.

| Variable | Values | Description |
|----------|---------------|-------------|
| SC4S_DEST_GLOBAL_ALTERNATES | Comma or space-separated list of syslog-ng destinations | Send all sources to alternate destinations |
| SC4S_DEST_&lt;VENDOR_PRODUCT&gt;_ALTERNATES | Comma or space-separated list of syslog-ng destinations | Send specific sources to alternate syslog-ng destinations using the VENDOR_PRODUCT syntax, e.g. SC4S_DEST_CISCO_ASA_ALTERNATES |
| SC4S_DEST_GLOBAL_ALTERNATES | Comma or space-separated list of destinations | Send all sources to alternate destinations |
| SC4S_DEST_&lt;VENDOR_PRODUCT&gt;_ALTERNATES | Comma or space-separated list of syslog-ng destinations | Send specific sources to alternate syslog-ng destinations using the VENDOR_PRODUCT syntax, e.g. `SC4S_DEST_CISCO_ASA_ALTERNATES` |

## SC4S Disk Buffer Configuration

Disk buffers in SC4S are allocated _per destination_. In the future as more destinations are supported, a separate list of variables
will be used for each. This is why you see the `DEST_SPLUNK_HEC` in the variable names below.
Disk buffers in SC4S are allocated _per destination_. Keep this in mind when using additional destinations that have disk buffering configured. By
default, when alternate HEC destinations are configured as outlined above disk buffering will be configured identically to that of the main HEC
destination (unless overriden individually).

### Important Notes Regarding Disk Buffering:

Expand Down
62 changes: 8 additions & 54 deletions package/etc/conf.d/destinations/splunk_hec.conf.tmpl
Original file line number Diff line number Diff line change
@@ -1,55 +1,9 @@
destination d_hec {
http(
url("{{- getenv "SPLUNK_HEC_URL" | strings.ReplaceAll "/services/collector" "" | strings.ReplaceAll "/event" "" | regexp.ReplaceLiteral "[, ]+" "/services/collector/event " }}/services/collector/event")
method("POST")
log-fifo-size({{- getenv "SC4S_DEST_SPLUNK_HEC_LOG_FIFO_SIZE" "180000000"}})
workers({{- getenv "SC4S_DEST_SPLUNK_HEC_WORKERS" "10"}})
batch-lines({{- getenv "SC4S_DEST_SPLUNK_HEC_BATCH_LINES" "1000"}})
batch-bytes({{- getenv "SC4S_DEST_SPLUNK_HEC_BATCH_BYTES" "4096kb"}})
batch-timeout({{- getenv "SC4S_DEST_SPLUNK_HEC_BATCH_TIMEOUT" "3000"}})
timeout({{- getenv "SC4S_DEST_SPLUNK_HEC_TIMEOUT" "30"}})
user_agent("sc4s/1.0 (events)")
user("sc4s")
headers("{{- getenv "SC4S_DEST_SPLUNK_DEST_SPLUNK_HEC_HEADERS" "Connection: close"}}")
password("{{- getenv "SPLUNK_HEC_TOKEN"}}")
persist-name("splunk_hec")
response-action(400 => drop, 404 => retry)
{{- $context := dict "var_id" "" -}}
{{- tmpl.Exec "t/splunk_hec.t" $context -}}

{{- if eq (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_ENABLE" "yes") "yes"}}

disk-buffer(

{{- if eq (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_RELIABLE" "no") "yes"}}
mem-buf-size({{conv.ToInt64 (math.Round ( math.Div (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_MEMBUFSIZE" "10241024") (getenv "SC4S_DEST_SPLUNK_HEC_WORKERS" "10")))}})
reliable(yes)
{{- else}}
mem-buf-length({{conv.ToInt64 (math.Round ( math.Div (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_MEMBUFLENGTH" "15000") (getenv "SC4S_DEST_SPLUNK_HEC_WORKERS" "10")))}})
reliable(no)
{{- end}}
{{- if ne (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_DIR") ""}}
dir("{{- getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_DIR"}}")
{{- end}}
disk-buf-size({{conv.ToInt64 (math.Round ( math.Div (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_DISKBUFSIZE" "53687091200") (getenv "SC4S_DEST_SPLUNK_HEC_WORKERS" "10")))}})
)
{{- end}}
tls(peer-verify({{- getenv "SC4S_DEST_SPLUNK_HEC_TLS_VERIFY" "yes"}})
{{- if ne (getenv "SC4S_DEST_SPLUNK_HEC_CIPHER_SUITE") ""}}
cipher-suite("{{- getenv "SC4S_DEST_SPLUNK_HEC_CIPHER_SUITE"}}")
{{- end}}
{{- if ne (getenv "SC4S_DEST_SPLUNK_HEC_SSL_VERSION") ""}}
ssl-version("{{- getenv "SC4S_DEST_SPLUNK_HEC_SSL_VERSION"}}")
{{- end}}
ca-file("{{- getenv "SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE" "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"}}"))
body('$(format-json
time=$S_UNIXTIME
host=${HOST}
source=${.splunk.source}
sourcetype=${.splunk.sourcetype}
index=${.splunk.index}
event="$MSG"
{{- if ne (getenv "SC4S_DEST_SPLUNK_INDEXED_FIELDS") "none" }}
fields.*
{{- end }}
)')
);
};
{{- if ne (getenv "SPLUNK_HEC_ALT_DESTS") "" }}
{{- range split (getenv "SPLUNK_HEC_ALT_DESTS" "") "," }}
{{- $context := dict "var_id" (print "_" .) -}}
{{- tmpl.Exec "t/splunk_hec.t" $context -}}
{{- end}}
{{- end}}
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ destination d_hec_internal {
timeout({{- getenv "SC4S_DEST_SPLUNK_HEC_TIMEOUT" "30"}})
user_agent("sc4s/1.0 (events)")
user("sc4s")
headers("{{- getenv "SC4S_DEST_SPLUNK_DEST_SPLUNK_HEC_HEADERS" "Connection: close"}}")
headers("{{- getenv "SC4S_DEST_SPLUNK_HEC_HEADERS" "Connection: close"}}")
password("{{- getenv "SPLUNK_HEC_TOKEN"}}")
persist-name("splunk_hec_internal")
response-action(400 => drop, 404 => retry)
Expand Down
61 changes: 61 additions & 0 deletions package/etc/go_templates/splunk_hec.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{{ define "SPLUNK_HEC" }}
destination d_hec{{ .var_id }} {
{{- $url := (getenv (print "SPLUNK_HEC" .var_id "_URL")) }}
http(
url("{{- $url | strings.ReplaceAll "/services/collector" "" | strings.ReplaceAll "/event" "" | regexp.ReplaceLiteral "[, ]+" "/services/collector/event " }}/services/collector/event")
method("POST")
log-fifo-size({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_LOG_FIFO_SIZE") "180000000"}})
workers({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_WORKERS") "10"}})
batch-lines({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_BATCH_LINES") "1000"}})
batch-bytes({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_BATCH_BYTES") "4096kb"}})
batch-timeout({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_BATCH_TIMEOUT") "3000"}})
timeout({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_TIMEOUT") "30"}})
user_agent("sc4s/1.0 (events)")
user("sc4s")
headers("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_HEADERS") "Connection: close"}}")
password("{{- getenv (print "SPLUNK_HEC" .var_id "_TOKEN")}}")
persist-name("splunk_hec")
response-action(400 => drop, 404 => retry)

{{- if eq (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_ENABLE") "yes") "yes"}}

disk-buffer(

{{- if eq (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_RELIABLE") "no") "yes"}}
mem-buf-size({{conv.ToInt64 (math.Round ( math.Div (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_MEMBUFSIZE") "10241024") (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_WORKERS") "10")))}})
reliable(yes)
{{- else}}
mem-buf-length({{conv.ToInt64 (math.Round ( math.Div (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_MEMBUFLENGTH") "15000") (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_WORKERS") "10")))}})
reliable(no)
{{- end}}
{{- if ne (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_DIR")) ""}}
dir("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_DIR")}}")
{{- end}}
disk-buf-size({{conv.ToInt64 (math.Round ( math.Div (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_DISKBUFSIZE") "53687091200") (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_WORKERS") "10")))}})
)
{{- end}}
tls(peer-verify({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_TLS_VERIFY") "yes"}})
{{- if ne (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_CIPHER_SUITE")) ""}}
cipher-suite("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_CIPHER_SUITE")}}")
{{- end}}
{{- if ne (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_SSL_VERSION")) ""}}
ssl-version("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_SSL_VERSION")}}")
{{- end}}
ca-file("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_TLS_CA_FILE") "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"}}"))
body('$(format-json
time=$S_UNIXTIME
host=${HOST}
source=${.splunk.source}
sourcetype=${.splunk.sourcetype}
index=${.splunk.index}
event="$MSG"
{{- if ne (getenv (print "SC4S_DEST_SPLUNK_INDEXED_FIELDS")) "none" }}
fields.*
{{- end }}
)')
);
};

{{- end -}}

{{- template "SPLUNK_HEC" (.) -}}
5 changes: 5 additions & 0 deletions tests/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,11 @@ services:
environment:
- SPLUNK_HEC_URL=https://splunk:8088
- SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
- SPLUNK_HEC_ALT_DESTS=FOO,BAR
- SPLUNK_HEC_FOO_URL=https://splunk:8088
- SPLUNK_HEC_FOO_TOKEN=${SPLUNK_HEC_TOKEN}
- SPLUNK_HEC_BAR_URL=https://splunk:8088
- SPLUNK_HEC_BAR_TOKEN=${SPLUNK_HEC_TOKEN}
- SC4S_DEST_SPLUNK_SC4S_METRICS_HEC=yes
- SC4S_SOURCE_TLS_ENABLE=no
- SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
Expand Down

0 comments on commit 50f01ea

Please sign in to comment.