Release/1.1.0 (#189)

* Feature/cisco ise (#178) This merge aggregates Cisco ISE messages from multiple segments into a single event in Splunk * Support for archival file destinations (#179) This merge add support for file archival in syslog-ng EWMM format * Feature/improvedmetrics (#185) * This change correct an incorrect URL issue preventing metrics flow and cleans up related documentation and filters anon metrics * Update Metrics for Splunk 8 * Resolve splunk sdk for CI unit tests move * Add disk buffer envrionment variables table (#184) * Add disk buffer envrionment variables table Add env var table to docs Adjust default disk buffer to 50 GB * Disk Buffer refinement Perform internal math to convert specified buffer sizes to values appropriate for internal (syslog-ng) destination options, which are set per worker. * Add Ubiquiti Networks Unifi product range (#188) * Resolve fallback events in msg rather than JSON mode * Add support for Ubiquiti networks unifi product line AP, Switch, USG (firewalls)
CSVD · Nov 12, 2019 · 8d007eb · 8d007eb
1 parent 3f8c370
commit 8d007eb
Show file tree

Hide file tree

Showing 60 changed files with 1,195 additions and 435 deletions.
diff --git a/.env.template b/.env.template
@@ -12,10 +12,6 @@ SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SPLUNK_PASSWORD=Changed@11
 SPLUNK_START_ARGS=--accept-license
 SPLUNK_HEC_URL=https://splunk:8088/services/collector/event
-SPLUNK_HEC_STATSURL=https://splunk:8088/services/collector/event
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 #SPLUNK_APPS_URL=https://splunkbase.splunk.com/app/2757/release/6.1.1/download,https://splunkbase.splunk.com/app/3245/release/1.0/download,https://splunkbase.splunk.com/app/1620/release/3.4.0/download,https://splunkbase.splunk.com/app/1467/release/2.5.8/download,https://splunkbase.splunk.com/app/2846/release/1.6.0/download,https://splunkbase.splunk.com/app/2847/release/1.2.0/download
 #SPLUNKBASE_USERNAME=username
 #SPLUNKBASE_PASSWORD=password
diff --git a/docker-compose-ci.yml b/docker-compose-ci.yml
@@ -33,7 +33,6 @@ services:
       - splunk
     environment:
       - SPLUNK_HEC_URL=${SPLUNK_HEC_URL}
-      - SPLUNK_HEC_STATSURL=${SPLUNK_HEC_STATSURL}
       - SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
       - SPLUNK_CONNECT_METHOD=${SPLUNK_CONNECT_METHOD}
       - SPLUNK_DEFAULT_INDEX=${SPLUNK_DEFAULT_INDEX}

diff --git a/docker-compose-debug.yml b/docker-compose-debug.yml
@@ -33,7 +33,6 @@ services:
       - splunk
     environment:
       - SPLUNK_HEC_URL=${SPLUNK_HEC_URL}
-      - SPLUNK_HEC_STATSURL=${SPLUNK_HEC_STATSURL}
       - SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
       - SPLUNK_CONNECT_METHOD=${SPLUNK_CONNECT_METHOD}
       - SPLUNK_DEFAULT_INDEX=${SPLUNK_DEFAULT_INDEX}

diff --git a/docker-compose-demo.yml b/docker-compose-demo.yml
@@ -34,7 +34,6 @@ services:
       - splunk
     environment:
       - SPLUNK_HEC_URL=${SPLUNK_HEC_URL}
-      - SPLUNK_HEC_STATSURL=${SPLUNK_HEC_STATSURL}
       - SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
       - SPLUNK_CONNECT_METHOD=${SPLUNK_CONNECT_METHOD}
       - SPLUNK_DEFAULT_INDEX=${SPLUNK_DEFAULT_INDEX}

diff --git a/docker-compose-perf.yml b/docker-compose-perf.yml
@@ -26,7 +26,6 @@ services:
       - splunk
     environment:
       - SPLUNK_HEC_URL=${SPLUNK_HEC_URL}
-      - SPLUNK_HEC_STATSURL=${SPLUNK_HEC_STATSURL}
       - SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
       - SPLUNK_CONNECT_METHOD=${SPLUNK_CONNECT_METHOD}
       - SPLUNK_DEFAULT_INDEX=${SPLUNK_DEFAULT_INDEX}

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -29,7 +29,8 @@ services:
         RH_ORG: ${RH_ORG}
         RH_ACTIVATION: ${RH_ACTIVATION}
     hostname: sc4s
-    command: -det
+#When this is enabled test_common will fail
+#    command: -det
     ports:
       - "514:514"
       - "601:601"
@@ -43,14 +44,16 @@ services:
       - splunk
     environment:
       - SPLUNK_HEC_URL=${SPLUNK_HEC_URL}
-      - SPLUNK_HEC_STATSURL=${SPLUNK_HEC_STATSURL}
       - SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
-      - SPLUNK_CONNECT_METHOD=${SPLUNK_CONNECT_METHOD}
-      - SPLUNK_DEFAULT_INDEX=${SPLUNK_DEFAULT_INDEX}
-      - SPLUNK_METRICS_INDEX=${SPLUNK_DEFAULT_INDEX}
       - SC4S_SOURCE_TLS_ENABLE=no
       - SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
+      - SC4S_LISTEN_DEFAULT_TCP_PORT=514
+      - SC4S_LISTEN_DEFAULT_UDP_PORT=514
+#      - SC4S_LISTEN_DEFAULT_TLS_PORT=6514
       - SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
+      - SC4S_LISTEN_CHECKPOINT_SPLUNK_TCP_PORT=6000
+#      - SC4S_ARCHIVE_CHECKPOINT=yes
+      - SC4S_ARCHIVE_GLOBAL=yes
     volumes:
       - ./tls:/opt/syslog-ng/tls
   splunk:

diff --git a/docs/configuration.md b/docs/configuration.md
@@ -21,11 +21,51 @@ and variables needed to properly configure SC4S for your environment.
 | SC4S_DEST_SPLUNK_HEC_SSL_VERSION |  comma separated list | Open SSL version list |
 | SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE | path | Custom trusted cert file |
 
+## SC4S Disk Buffer Configuration
+
+Disk buffers in SC4S are allocated _per destination_.  In the future as more destinations are supported, a separate list of variables
+will be used for each.  This is why you see the `DEST_SPLUNK_HEC` in the variable names below.
+* NOTE:  "Reliable" disk buffering offeres little advantage over "normal" disk buffering, at a significant performance penalty.
+For this reason, normal disk buffering is recommended.
+* NOTE:  If you add destinations locally in your configuration, pay attention to the _cumulative_ buffer requirements when allocating local
+disk.
+* NOTE:  The values for the variables below represent the _total_ sizes of the buffers for the destination.  These sizes are divded by the
+number of workers (threads) when setting the actual syslog-ng buffer options, because the buffer options apply to each worker rather than the
+entire destination.  Pay careful attention to this when using the "BYOE" version of SC4S, where direct access to the syslog-ng config files
+may hide this nuance.
+
+| Variable | Values/Default   | Description |
+|----------|---------------|-------------|
+| SC4S_DEST_SPLUNK_HEC_DISKBUFF_ENABLE | yes(default) or no | Enable local disk buffering  |
+| SC4S_DEST_SPLUNK_HEC_DISKBUFF_RELIABLE | yes or no(default) | Enable reliable/normal disk buffering (normal recommended) |
+| SC4S_DEST_SPLUNK_HEC_DISKBUFF_MEMBUFSIZE | bytes (10241024) | Memory buffer size in bytes (used with reliable disk buffering) |
+| SC4S_DEST_SPLUNK_HEC_DISKBUFF_MEMBUFLENGTH |messages (15000) | Memory buffer size in message count (used with normal disk buffering) |
+| SC4S_DEST_SPLUNK_HEC_DISKBUFF_DISKBUFSIZE | bytes (53687091200) | size of local disk buffer in bytes (default 50 GB) |
+
+## Archive File Configuration
+
+This feature is designed to support "compliance" archival of all messages. To enable this feature update the Unit file
+or docker compose to mount an appropriate host folder to the container folder ``/opt/syslog-ng/var/archive``.
+The files will be stored in a folder structure using the naming pattern
+``${YEAR}/${MONTH}/${DAY}/${fields.sc4s_vendor_product}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}.log"``.
+This pattern will create one file per "vendor_product" per minute with records formatted using syslog-ng's EWMM template. 
+
+**WARNING POTENTIAL OUTAGE CAUSING CONSEQUENCE**
+
+SC4S does not prune the files that are created. The administrator must provide a means of log rotation to prune files
+and/or move them to an archival system to avoid disk space failures.
+
+| Variable | Values        | Description |
+|----------|---------------|-------------|
+| SC4S_ARCHIVE_GLOBAL | yes or undefined | Enable archive of all vendor_products |
+| SC4S_ARCHIVE_LISTEN_<VENDOR_PRODUCT> | yes(default) or undefined | See sources section of documentation enables selective archival |
+
+
 ## Syslog Source Configuration
 
 | Variable | Values/Default | Description |
 |----------|----------------|-------------|
-| SC4S_SOURCE_TLS_ENABLE | no(default) or yes | Enable a TLS listener on port 6514 |
+| SC4S_LISTEN_DEFAULT_TLS_PORT | undefined or 6514 | Enable a TLS listener on port 6514 |
 | SC4S_SOURCE_TLS_OPTIONS | See openssl | List of SSl/TLS protocol versions to support | 
 | SC4S_SOURCE_TLS_CIPHER_SUITE | See openssl | List of Ciphers to support |
 | SC4S_SOURCE_TCP_MAX_CONNECTIONS | 2000 | Max number of TCP Connections |

diff --git a/docs/gettingstarted/byoe-rhel7.md b/docs/gettingstarted/byoe-rhel7.md
@@ -129,12 +129,9 @@ sudo bash /opt/sc4s/bin/preconfig.sh
 
 ```dotenv
 SYSLOGNG_OPTS=-f /opt/syslog-ng/etc/syslog-ng.conf 
-SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
+SPLUNK_HEC_URL=https://splunk.smg.aws:8088
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SC4S_DEST_SPLUNK_HEC_WORKERS=6
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```

diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md
@@ -65,12 +65,9 @@ of events in the event of network failure to the Splunk infrastructure.
 Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
 
 ```dotenv
-SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
+SPLUNK_HEC_URL=https://splunk.smg.aws:8088
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SC4S_DEST_SPLUNK_HEC_WORKERS=6
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
@@ -179,12 +176,9 @@ match this value to the total number of indexers behind the load balancer.
 uncomment the last line in the example below.
 
 ```dotenv
-SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
+SPLUNK_HEC_URL=https://splunk.smg.aws:8088
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SC4S_DEST_SPLUNK_HEC_WORKERS=6
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no

diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md
@@ -92,12 +92,9 @@ again upon restart.
 Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
 
 ```dotenv
-SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
+SPLUNK_HEC_URL=https://splunk.smg.aws:8088
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SC4S_DEST_SPLUNK_HEC_WORKERS=6
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
@@ -208,12 +205,9 @@ match this value to the total number of indexers behind the load balancer.
 uncomment the last line in the example below.
 
 ```dotenv
-SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
+SPLUNK_HEC_URL=https://splunk.smg.aws:8088
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SC4S_DEST_SPLUNK_HEC_WORKERS=6
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no

diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md
@@ -71,12 +71,9 @@ unit file above.  Failure to do this will cause SC4S to abort at startup.
 Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
 
 ```dotenv
-SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
+SPLUNK_HEC_URL=https://splunk.smg.aws:8088
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SC4S_DEST_SPLUNK_HEC_WORKERS=6
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
@@ -182,12 +179,9 @@ match this value to the total number of indexers behind the load balancer.
 uncomment the last line in the example below.
 
 ```dotenv
-SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
+SPLUNK_HEC_URL=https://splunk.smg.aws:8088
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SC4S_DEST_SPLUNK_HEC_WORKERS=6
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no

diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md
@@ -71,12 +71,9 @@ unit file above.  Failure to do this will cause SC4S to abort at startup.
 Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
 
 ```dotenv
-SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
+SPLUNK_HEC_URL=https://splunk.smg.aws:8088
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SC4S_DEST_SPLUNK_HEC_WORKERS=6
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
@@ -182,12 +179,9 @@ match this value to the total number of indexers behind the load balancer.
 uncomment the last line in the example below.
 
 ```dotenv
-SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
+SPLUNK_HEC_URL=https://splunk.smg.aws:8088
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
 SC4S_DEST_SPLUNK_HEC_WORKERS=6
-SPLUNK_CONNECT_METHOD=hec
-SPLUNK_DEFAULT_INDEX=main
-SPLUNK_METRICS_INDEX=em_metrics
 SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no

diff --git a/docs/sources.md b/docs/sources.md
@@ -191,6 +191,52 @@ Use the following search to validate events are present, for NX-OS, WLC and ACI
 index=<asconfigured> sourcetype=cisco:ios | stats count by host
 ```
 
+## Product - ISE
+
+| Ref            | Link                                                                                                    |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| Splunk Add-on  | https://splunkbase.splunk.com/app/1915/                                                                 |
+| Product Manual | https://www.cisco.com/c/en/us/td/docs/security/ise/2-6/Cisco_ISE_Syslogs/Cisco_ISE_Syslogs/Cisco_ISE_Syslogs_chapter_00.html |
+
+
+### Sourcetypes
+
+| sourcetype     | notes                                                                                                   |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| cisco:ise:syslog     | Aggregation used                                                                                                    |
+
+### Sourcetype and Index Configuration
+
+| key            | sourcetype     | index          | notes          |
+|----------------|----------------|----------------|----------------|
+| cisco_ise     | cisco:ise:syslog    | netauth          | None     |
+
+
+### Filter type
+
+PATTERN MATCH
+
+### Setup and Configuration
+
+* No special steps required
+
+### Options
+
+| Variable       | default        | description    |
+|----------------|----------------|----------------|
+| SC4S_LISTEN_CISCO_ISE_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format |
+| SC4S_LISTEN_CISCO_ISE_UDP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format |
+
+### Verification
+
+Use the following search to validate events are present
+
+```
+index=<asconfigured> sourcetype=cisco:ise:syslog
+```
+
+Verify timestamp, and host values match as expected    
+
 ## Product - Meraki Product Line MR, MS, MX, MV
 
 | Ref            | Link                                                                                                    |
@@ -237,9 +283,6 @@ Use the following search to validate events are present
 index=<asconfigured> sourcetype=merkai
 ```
 
-Verify timestamp, and host values match as expected    
-
-
 Verify timestamp, and host values match as expected    
 
 # Vendor - Forcepoint
@@ -905,6 +948,80 @@ index=<asconfigured> sourcetype=bluecoat:proxysg:access:kv | stats count by host
 ```
 
 
+# Vendor - Ubiquiti - Unifi
+
+All Ubiquity Unfi firewalls, switches, and access points share a common syslog configuration via the NMS.
+
+
+* Login to NMS
+* Navigate to settings
+* Navigate to Site 
+* Enable Remote syslog server
+* Enter hostname and port
+* Update ``vi /opt/sc4s/local/context/vendor_product_by_source.conf `` update the host or ip mask for ``f_ubiquiti_unifi_fw`` to identify USG firewalls
+
+## Product - Unifi Switch and Access Points 
+
+Unifi devices are managed using the Network Management Controller
+
+
+| Ref            | Link                                                                                                    |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| Splunk Add-on  | https://splunkbase.splunk.com/app/4107/                                                                 |
+| Product Manual | https://https://help.ubnt.com/    |
+
+
+### Sourcetypes
+
+| sourcetype     | notes                                                                                                   |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| ubnt  | Used when no sub source type is required by add on |
+| ubnt:fw  | USG events |
+| ubnt:threat | USG IDS events    |
+| ubnt:switch  | Unifi Switches |
+| ubnt:wireless  | Access Point logs |
+
+
+### Sourcetype and Index Configuration
+
+| key            | sourcetype     | index          | notes          |
+|----------------|----------------|----------------|----------------|
+| ubiquiti_unifi      | ubnt     | netops          | none          |
+| ubiquiti_unifi_fw      | ubnt:fw       | netfw          | none          |
+| ubiquiti_unifi_link      | ubnt:link     | netops          | none          |
+| ubiquiti_unifi_sudo      | ubnt:sudo     | netops          | none          |
+| ubiquiti_unifi_switch | ubnt:switch | netops          | none          |
+| ubiquiti_unifi_threat      | ubnt:threat      | netids          | none          |
+| ubiquiti_unifi_wireless | ubnt:wireless | netops          | none          |
+
+
+### Filter type
+
+MSG Parse: This filter parses message content
+
+### Setup and Configuration
+
+* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer.
+* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source.
+* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration
+    * Select TCP or SSL transport option
+    * Ensure the format of the event is customized per Splunk documentation
+
+### Options
+
+| Variable       | default        | description    |
+|----------------|----------------|----------------|
+| SC4S_LISTEN_ZSCALER_NSS_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+
+### Verification
+
+An active proxy will generate frequent events. Use the following search to validate events are present per source device
+
+```
+index=<asconfigured> sourcetype=zscalernss-* | stats count by host
+```
+
+
 # Vendor - Zscaler
 
 ## Product - All Products