From 50f01eab58d4414cf2a4c9d70483127469358312 Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Tue, 7 Jul 2020 20:33:14 -0400 Subject: [PATCH] Support generation of multiple HEC destinations dynamically (#553) This change add the ability to create additional HEC destination to use as alternates this allows the administrator to send events from SC4S to more than one instance of Splunk Enterprise, Splunk Enterprise Cloud, or Splunk DSP --- docs/configuration.md | 52 ++++++++++++---- .../conf.d/destinations/splunk_hec.conf.tmpl | 62 +++---------------- .../splunk_hec_internal.conf.tmpl | 2 +- package/etc/go_templates/splunk_hec.t | 61 ++++++++++++++++++ tests/docker-compose.yml | 5 ++ 5 files changed, 115 insertions(+), 67 deletions(-) create mode 100644 package/etc/go_templates/splunk_hec.t diff --git a/docs/configuration.md b/docs/configuration.md index 53af10e..eb17570 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -21,7 +21,7 @@ syslog. | Variable | Values | Description | |----------|---------------|-------------| -| SC4S_DEST_SPLUNK_HEC_GLOBAL | yes | Send events to Splunk using HEC | +| SC4S_DEST_SPLUNK_HEC_GLOBAL | yes | Send events to Splunk using HEC. This applies _only_ to the primary HEC destination. | | SC4S_DEST_SPLUNK_HEC_CIPHER_SUITE | comma separated list | Open SSL cipher suite list | | SC4S_DEST_SPLUNK_HEC_SSL_VERSION | comma separated list | Open SSL version list | | SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE | path | Custom trusted cert file | @@ -29,27 +29,55 @@ syslog. | SC4S_DEST_SPLUNK_HEC_WORKERS | numeric | Number of destination workers (default: 10 threads). This should rarely need to be changed; consult sc4s community for advice on appropriate setting in extreme high- or low-volume environments. | | SC4S_DEST_SPLUNK_INDEXED_FIELDS | facility,
severity,
container,
loghost,
destport,
fromhostip,
proto

none | List of sc4s indexed fields that will be included with each event in Splunk (default is the entire list except "none"). Two other indexed fields, `sc4s_vendor_product` and `sc4s_syslog_format`, will also appear along with the fields selected via the list and cannot be turned on or off individually. If no indexed fields are desired (including the two internal ones), set the value to the single value of "none". When setting this variable, separate multiple entries with commas and do not include extra spaces.

This list maps to the following indexed fields that will appear in all Splunk events:
facility: sc4s_syslog_facility
severity: sc4s_syslog_severity
container: sc4s_container
loghost: sc4s_loghost
dport: sc4s_destport
fromhostip: sc4s_fromhostip
proto: sc4s_proto -## Alternate Destination Configuration +* NOTE: When using alternate HEC destinations, the destination operating paramaters outlined above (`CIPHER_SUITE`, `SSL_VERSION`, etc.) can be +individually controlled per `DESTID` (see "Configuration of Additional Splunk HEC Destinations" immediately below). For example, to set the number of workers +for the alternate HEC destination `d_hec_FOO` to 24, set `SC4S_DEST_SPLUNK_HEC_FOO_WORKERS=24`. -Alternate destinations other than HEC can be configured in SC4S. Global and/or source-specific forms of the -variables below can be used to send data to alternate destinations. +## Creation of Additional Splunk HEC Destinations -* NOTE: The administrator is responsible for ensuring that the alternate destinations are configured in the -local mount tree, and that syslog-ng properly parses them. +Additional Splunk HEC destinations can be dynamically created through environment variables. The use of these destinations can then be controlled +along with other user-defined destinations on a global or per-source basis (see "Alternate Destination Use" immediately below). + +| Variable | Values | Description | +|----------|---------------|-------------| +| SPLUNK_HEC_ALT_DESTS | Comma or space-separated UPPER case list of destination ids | destination IDs are UPPER case single-word friendly strings used to identify the new destination, which will be named with the destination id appended, for example `d_hec_FOO` | +| SPLUNK_HEC<DESTID>_URL | url | Example: `SPLUNK_HEC_FOO_URL=https://splunk:8088`. `DESTID` must be a member of the list configured in `SPLUNK_HEC_ALT_DESTS` configured above | +| SPLUNK_HEC<DESTID>_TOKEN | string | Example: `SPLUNK_HEC_BAR_TOKEN=<token>`. `DESTID` must be a member of the list configured in `SPLUNK_HEC_ALT_DESTS` configured above | + +When set above, the destinations will be created with the `DESTID` appended, for example: `d_hec_FOO`. These destinations can then be +specified below (along with any other destinations created locally) either globally or per source. + +* NOTE: The `DESTID` specified in the `URL` and `TOKEN` variables above _must_ match the `DESTID` entries enumerated the `SPLUNK_HEC_ALT_DESTS` list. +Failure to do so will cause destinations to be created without proper HEC parameters. -* NOTE: Do not include `d_hec` in any list of alternate destinations. The configuration of the default HEC destination is configured -separately from that of the alternates below. +* NOTE: Additional Splunk HEC destinations will _not_ be tested at startup. It is the responsiblity of the admin to ensure that additional destinations +are provisioned with the correct URL(s) and tokens to ensure proper connectivity. + +* NOTE: The disk and CPU requirements will increase proportionally depending on the number of additional HEC destinations in use (e.g. each HEC +destination will have its own disk buffer). + +## Alternate Destination Use + +All alternate destinations (including alternate HEC destinations) are configured for use in SC4S through variables. Global and/or source-specific forms of +the variables below can be used to send data to alternate destinations. + +* NOTE: The administrator is responsible for ensuring that any non-HEC alternate destinations are configured in the +local mount tree, and that syslog-ng properly parses them. +* NOTE: Do not include the primary HEC destination (`d_hec`) in any list of alternate destinations. The configuration of the primary HEC destination +is configured separately from that of the alternates below. However, _alternate_ HEC destinations (e.g. `d_hec_FOO`) should be configured below, just +like any other user-supplied destination. | Variable | Values | Description | |----------|---------------|-------------| -| SC4S_DEST_GLOBAL_ALTERNATES | Comma or space-separated list of syslog-ng destinations | Send all sources to alternate destinations | -| SC4S_DEST_<VENDOR_PRODUCT>_ALTERNATES | Comma or space-separated list of syslog-ng destinations | Send specific sources to alternate syslog-ng destinations using the VENDOR_PRODUCT syntax, e.g. SC4S_DEST_CISCO_ASA_ALTERNATES | +| SC4S_DEST_GLOBAL_ALTERNATES | Comma or space-separated list of destinations | Send all sources to alternate destinations | +| SC4S_DEST_<VENDOR_PRODUCT>_ALTERNATES | Comma or space-separated list of syslog-ng destinations | Send specific sources to alternate syslog-ng destinations using the VENDOR_PRODUCT syntax, e.g. `SC4S_DEST_CISCO_ASA_ALTERNATES` | ## SC4S Disk Buffer Configuration -Disk buffers in SC4S are allocated _per destination_. In the future as more destinations are supported, a separate list of variables -will be used for each. This is why you see the `DEST_SPLUNK_HEC` in the variable names below. +Disk buffers in SC4S are allocated _per destination_. Keep this in mind when using additional destinations that have disk buffering configured. By +default, when alternate HEC destinations are configured as outlined above disk buffering will be configured identically to that of the main HEC +destination (unless overriden individually). ### Important Notes Regarding Disk Buffering: diff --git a/package/etc/conf.d/destinations/splunk_hec.conf.tmpl b/package/etc/conf.d/destinations/splunk_hec.conf.tmpl index d0a3341..eb759fb 100644 --- a/package/etc/conf.d/destinations/splunk_hec.conf.tmpl +++ b/package/etc/conf.d/destinations/splunk_hec.conf.tmpl @@ -1,55 +1,9 @@ -destination d_hec { - http( - url("{{- getenv "SPLUNK_HEC_URL" | strings.ReplaceAll "/services/collector" "" | strings.ReplaceAll "/event" "" | regexp.ReplaceLiteral "[, ]+" "/services/collector/event " }}/services/collector/event") - method("POST") - log-fifo-size({{- getenv "SC4S_DEST_SPLUNK_HEC_LOG_FIFO_SIZE" "180000000"}}) - workers({{- getenv "SC4S_DEST_SPLUNK_HEC_WORKERS" "10"}}) - batch-lines({{- getenv "SC4S_DEST_SPLUNK_HEC_BATCH_LINES" "1000"}}) - batch-bytes({{- getenv "SC4S_DEST_SPLUNK_HEC_BATCH_BYTES" "4096kb"}}) - batch-timeout({{- getenv "SC4S_DEST_SPLUNK_HEC_BATCH_TIMEOUT" "3000"}}) - timeout({{- getenv "SC4S_DEST_SPLUNK_HEC_TIMEOUT" "30"}}) - user_agent("sc4s/1.0 (events)") - user("sc4s") - headers("{{- getenv "SC4S_DEST_SPLUNK_DEST_SPLUNK_HEC_HEADERS" "Connection: close"}}") - password("{{- getenv "SPLUNK_HEC_TOKEN"}}") - persist-name("splunk_hec") - response-action(400 => drop, 404 => retry) +{{- $context := dict "var_id" "" -}} +{{- tmpl.Exec "t/splunk_hec.t" $context -}} - {{- if eq (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_ENABLE" "yes") "yes"}} - - disk-buffer( - - {{- if eq (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_RELIABLE" "no") "yes"}} - mem-buf-size({{conv.ToInt64 (math.Round ( math.Div (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_MEMBUFSIZE" "10241024") (getenv "SC4S_DEST_SPLUNK_HEC_WORKERS" "10")))}}) - reliable(yes) - {{- else}} - mem-buf-length({{conv.ToInt64 (math.Round ( math.Div (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_MEMBUFLENGTH" "15000") (getenv "SC4S_DEST_SPLUNK_HEC_WORKERS" "10")))}}) - reliable(no) - {{- end}} - {{- if ne (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_DIR") ""}} - dir("{{- getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_DIR"}}") - {{- end}} - disk-buf-size({{conv.ToInt64 (math.Round ( math.Div (getenv "SC4S_DEST_SPLUNK_HEC_DISKBUFF_DISKBUFSIZE" "53687091200") (getenv "SC4S_DEST_SPLUNK_HEC_WORKERS" "10")))}}) - ) - {{- end}} - tls(peer-verify({{- getenv "SC4S_DEST_SPLUNK_HEC_TLS_VERIFY" "yes"}}) - {{- if ne (getenv "SC4S_DEST_SPLUNK_HEC_CIPHER_SUITE") ""}} - cipher-suite("{{- getenv "SC4S_DEST_SPLUNK_HEC_CIPHER_SUITE"}}") - {{- end}} - {{- if ne (getenv "SC4S_DEST_SPLUNK_HEC_SSL_VERSION") ""}} - ssl-version("{{- getenv "SC4S_DEST_SPLUNK_HEC_SSL_VERSION"}}") - {{- end}} - ca-file("{{- getenv "SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE" "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"}}")) - body('$(format-json - time=$S_UNIXTIME - host=${HOST} - source=${.splunk.source} - sourcetype=${.splunk.sourcetype} - index=${.splunk.index} - event="$MSG" - {{- if ne (getenv "SC4S_DEST_SPLUNK_INDEXED_FIELDS") "none" }} - fields.* - {{- end }} - )') - ); -}; +{{- if ne (getenv "SPLUNK_HEC_ALT_DESTS") "" }} +{{- range split (getenv "SPLUNK_HEC_ALT_DESTS" "") "," }} +{{- $context := dict "var_id" (print "_" .) -}} +{{- tmpl.Exec "t/splunk_hec.t" $context -}} +{{- end}} +{{- end}} \ No newline at end of file diff --git a/package/etc/conf.d/destinations/splunk_hec_internal.conf.tmpl b/package/etc/conf.d/destinations/splunk_hec_internal.conf.tmpl index 6352a76..2d9e0dd 100644 --- a/package/etc/conf.d/destinations/splunk_hec_internal.conf.tmpl +++ b/package/etc/conf.d/destinations/splunk_hec_internal.conf.tmpl @@ -10,7 +10,7 @@ destination d_hec_internal { timeout({{- getenv "SC4S_DEST_SPLUNK_HEC_TIMEOUT" "30"}}) user_agent("sc4s/1.0 (events)") user("sc4s") - headers("{{- getenv "SC4S_DEST_SPLUNK_DEST_SPLUNK_HEC_HEADERS" "Connection: close"}}") + headers("{{- getenv "SC4S_DEST_SPLUNK_HEC_HEADERS" "Connection: close"}}") password("{{- getenv "SPLUNK_HEC_TOKEN"}}") persist-name("splunk_hec_internal") response-action(400 => drop, 404 => retry) diff --git a/package/etc/go_templates/splunk_hec.t b/package/etc/go_templates/splunk_hec.t new file mode 100644 index 0000000..104fe8e --- /dev/null +++ b/package/etc/go_templates/splunk_hec.t @@ -0,0 +1,61 @@ +{{ define "SPLUNK_HEC" }} +destination d_hec{{ .var_id }} { + {{- $url := (getenv (print "SPLUNK_HEC" .var_id "_URL")) }} + http( + url("{{- $url | strings.ReplaceAll "/services/collector" "" | strings.ReplaceAll "/event" "" | regexp.ReplaceLiteral "[, ]+" "/services/collector/event " }}/services/collector/event") + method("POST") + log-fifo-size({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_LOG_FIFO_SIZE") "180000000"}}) + workers({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_WORKERS") "10"}}) + batch-lines({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_BATCH_LINES") "1000"}}) + batch-bytes({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_BATCH_BYTES") "4096kb"}}) + batch-timeout({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_BATCH_TIMEOUT") "3000"}}) + timeout({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_TIMEOUT") "30"}}) + user_agent("sc4s/1.0 (events)") + user("sc4s") + headers("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_HEADERS") "Connection: close"}}") + password("{{- getenv (print "SPLUNK_HEC" .var_id "_TOKEN")}}") + persist-name("splunk_hec") + response-action(400 => drop, 404 => retry) + + {{- if eq (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_ENABLE") "yes") "yes"}} + + disk-buffer( + + {{- if eq (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_RELIABLE") "no") "yes"}} + mem-buf-size({{conv.ToInt64 (math.Round ( math.Div (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_MEMBUFSIZE") "10241024") (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_WORKERS") "10")))}}) + reliable(yes) + {{- else}} + mem-buf-length({{conv.ToInt64 (math.Round ( math.Div (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_MEMBUFLENGTH") "15000") (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_WORKERS") "10")))}}) + reliable(no) + {{- end}} + {{- if ne (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_DIR")) ""}} + dir("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_DIR")}}") + {{- end}} + disk-buf-size({{conv.ToInt64 (math.Round ( math.Div (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_DISKBUFF_DISKBUFSIZE") "53687091200") (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_WORKERS") "10")))}}) + ) + {{- end}} + tls(peer-verify({{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_TLS_VERIFY") "yes"}}) + {{- if ne (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_CIPHER_SUITE")) ""}} + cipher-suite("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_CIPHER_SUITE")}}") + {{- end}} + {{- if ne (getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_SSL_VERSION")) ""}} + ssl-version("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_SSL_VERSION")}}") + {{- end}} + ca-file("{{- getenv (print "SC4S_DEST_SPLUNK_HEC" .var_id "_TLS_CA_FILE") "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"}}")) + body('$(format-json + time=$S_UNIXTIME + host=${HOST} + source=${.splunk.source} + sourcetype=${.splunk.sourcetype} + index=${.splunk.index} + event="$MSG" + {{- if ne (getenv (print "SC4S_DEST_SPLUNK_INDEXED_FIELDS")) "none" }} + fields.* + {{- end }} + )') + ); +}; + +{{- end -}} + +{{- template "SPLUNK_HEC" (.) -}} diff --git a/tests/docker-compose.yml b/tests/docker-compose.yml index e5daca8..2959a84 100644 --- a/tests/docker-compose.yml +++ b/tests/docker-compose.yml @@ -29,6 +29,11 @@ services: environment: - SPLUNK_HEC_URL=https://splunk:8088 - SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN} + - SPLUNK_HEC_ALT_DESTS=FOO,BAR + - SPLUNK_HEC_FOO_URL=https://splunk:8088 + - SPLUNK_HEC_FOO_TOKEN=${SPLUNK_HEC_TOKEN} + - SPLUNK_HEC_BAR_URL=https://splunk:8088 + - SPLUNK_HEC_BAR_TOKEN=${SPLUNK_HEC_TOKEN} - SC4S_DEST_SPLUNK_SC4S_METRICS_HEC=yes - SC4S_SOURCE_TLS_ENABLE=no - SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no