Develop (#44)

CSVD · Aug 14, 2019 · d03f341 · d03f341
1 parent 19b805c
commit d03f341
Show file tree

Hide file tree

Showing 21 changed files with 692 additions and 31 deletions.
diff --git a/.env.template b/.env.template
@@ -16,6 +16,6 @@ SPLUNK_HEC_STATSURL=https://splunk:8088/services/collector/event
 SPLUNK_CONNECT_METHOD=hec
 SPLUNK_DEFAULT_INDEX=main
 SPLUNK_METRICS_INDEX=metrics
-SPLUNK_APPS_URL=https://splunkbase.splunk.com/app/2757/release/6.1.1/download,https://splunkbase.splunk.com/app/3245/release/1.0/download,https://splunkbase.splunk.com/app/1620/release/3.4.0/download,https://splunkbase.splunk.com/app/1467/release/2.5.8/download,https://splunkbase.splunk.com/app/2846/release/1.6.0/download
+SPLUNK_APPS_URL=https://splunkbase.splunk.com/app/2757/release/6.1.1/download,https://splunkbase.splunk.com/app/3245/release/1.0/download,https://splunkbase.splunk.com/app/1620/release/3.4.0/download,https://splunkbase.splunk.com/app/1467/release/2.5.8/download,https://splunkbase.splunk.com/app/2846/release/1.6.0/download,https://splunkbase.splunk.com/app/2847/release/1.2.0/download
 SPLUNKBASE_USERNAME=username
 SPLUNKBASE_PASSWORD=password
diff --git a/.idea/misc.xml b/.idea/misc.xml
diff --git a/.idea/splunk-connect-for-syslog.iml b/.idea/splunk-connect-for-syslog.iml
diff --git a/README.md b/README.md
@@ -41,8 +41,6 @@ index = main
 
 # License
 
-
-
 Configuration and documentation licensed subject to [CC0](LICENSE-CC0)
 
 Code and scripts licensed subject to [BSD-2-Clause](LICENSE-BSD2) 
diff --git a/docs/FortiGate_event.png b/docs/FortiGate_event.png
diff --git a/docs/FortiGate_traffic.png b/docs/FortiGate_traffic.png
diff --git a/docs/FortiGate_utm.png b/docs/FortiGate_utm.png
diff --git a/docs/gettingstarted.md b/docs/gettingstarted.md
@@ -0,0 +1,58 @@
+
+# Pre-req
+
+* Linux host with Docker 19.x or newer with Docker Swarm enabled
+    * [Getting Started](https://docs.docker.com/get-started/)
+* A Splunk index for metrics typically "em_metrics"
+* One or more Splunk indexes for events collected by SC4S
+* Splunk HTTP event collector enabled with a token dedicated for SC4S
+    * [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q)
+    * [Splunk Enterprise Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
+* A network load balancer (NLB) configured for round robin. Note: Special consideration may be required when more advanced products are used. The optimal configuration of the load balancer will round robin each http POST request (not each connection)
+
+# Setup
+
+* Create a directory on the server for configuration
+* Create a docker-compose.yml file based on the following template
+
+```yaml
+version: "3"
+services:
+  sc4s:
+    image: rfaircloth/scs:latest
+    hostname: sc4s
+    ports:
+      - "514"
+      - "601"
+      - "514/udp"
+      - "5514"
+      - "5514/udp"
+    stdin_open: true
+    tty: true
+    environment:
+      - SPLUNK_HEC_URL=https://foo:8088/services/collector/event
+      - SPLUNK_HEC_TOKEN=<token>
+      - SPLUNK_CONNECT_METHOD=hec
+      - SPLUNK_DEFAULT_INDEX=<defaultindex>
+      - SPLUNK_METRICS_INDEX=em_metrics
+    volumes:
+    - splunk_index.csv:/opt/syslog-ng/etc/context/splunk_index.csv
+```
+
+* Download the latest context.csv file to the current directory
+```bash
+wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context/splunk_index.csv
+```
+
+* Edit splunk_index.csv review the index configuration and revise as required for sourcertypes utilized in your environment.
+
+* Start SC4S
+
+```bash
+docker stack deploy --compose-file docker-compose.yml sc4s
+```
+
+
+## Scale out
+
+Additional hosts can be deployed for syslog collection from additional network zones and locations
diff --git a/docs/index.md b/docs/index.md
@@ -0,0 +1,24 @@
+# Welcome to Splunk Connect for Syslog
+
+Splunk Connect for Syslog is an open source packaged solution for 
+getting data into Splunk using syslog-ng Open Source Edition (Syslog-NG OSE) and the Splunk 
+HTTP event Collector. 
+
+## Project Goals
+
+* Bring a tested configuration and build of syslog-ng OSE to the market that will function consistently regardless of the underlying host's linux distribution
+* Provide a container with the tested configuration for Docker/K8s that can be more easily deployed than upstream packages directly on a customer OS
+* Provide validated (testable and tested) implementations of filter and parse functions for common vendor products
+* Reduce latency and improve scale by balancing event distribution across Splunk Indexers
+
+
+
+## License
+
+* Configuration and documentation licensed subject to [CC0](LICENSE-CC0)
+
+* Code and scripts licensed subject to [BSD-2-Clause](LICENSE-BSD2) 
+
+* Third Party Red Hat Universal Base Image see [License](https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf)
+
+* Third Party Syslog-NG (OSE) [License](https://github.com/balabit/syslog-ng)
diff --git a/docs/performance.md b/docs/performance.md
@@ -0,0 +1,27 @@
+# Performance
+
+## Tested Configuration
+
+* SC4S instance requesting 8 cores and 4 GB of memory with K8S scheduler. 
+* 6 Splunk Indexers clustered in Single site
+* 1 loggen test client using the following command 
+* AWS instance type c5n.4xlarge
+
+```bash
+/opt/syslog-ng/bin/loggen -i --rate=1000 --interval=180 -P -F --sdata="[test name=\"stress17\"]" -s 800 --active-connections=10 sc4s 514
+```
+
+## Result  
+
+The single syslog-ng container in this test is able to provided effective balancing and routing of events equivalent 632 GB per day 
+
+```
+average rate = 9717.58 msg/sec, count=1749420, time=180.026, (average) msg size=800, bandwidth=7591.86 kB/sec
+
+```
+
+
+## Limitations
+
+* Splunk Enterprise's implementation of the http event collection server will respond to the client with a status code 200 and fail to commit the events to disk during a rolling restart in our testing 20-30 events per indexer may be lost
+
diff --git a/docs/sources.md b/docs/sources.md
@@ -0,0 +1,170 @@
+# Vendor - Cisco
+
+## Product - ASA (Pre Firepower)
+
+| Ref            | Link                                                                                                    |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| Splunk Add-on  | https://splunkbase.splunk.com/app/1620/                                                                 |
+| Product Manual | https://www.cisco.com/c/en/us/td/docs/security/asa/asa82/configuration/guide/config/monitor_syslog.html |
+
+
+## Sourcetypes
+
+| sourcetype     | notes                                                                                                   |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| cisco:asa      | None                                                                                                    |
+| cisco:pix      | Not supported                                                                                           |
+| cisco:fwsm     | Not supported                                                                                           |
+
+## Filter type
+
+MSG Parse: This filter parses message content
+
+## Setup and Configuration
+
+* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer.
+* Review and update the splunk_index.csv file and set the index as required.
+* Follow vendor configuration steps per Product Manual above ensure:
+    * Log Level is 6 "Informational"
+    * Protocol is TCP/IP
+    * permit-hostdown is on
+    * device-id is hostname and included
+    * timestamp is included
+
+## Verification
+
+Use the following search to validate events are present
+
+```
+index=<asconfigured> sourcetype=cisco:asa
+```
+
+Verify timestamp, and host values match as expected    
+
+# Vendor - Fortinet
+
+## Product - Fortigate
+
+| Ref            | Link                                                                                                    |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| Splunk Add-on  | https://splunkbase.splunk.com/app/2846/                                                                 |
+| Product Manual | https://docs.fortinet.com/product/fortigate/6.2                                                         |
+
+
+## Sourcetypes
+
+| sourcetype     | notes                                                                                                   |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| fgt_log        | The catch all sourcetype is not used                                                                                                    |
+| fgt_traffic    | None                                                                                         |
+| fgt_utm        | None                                                                                          |
+| fgt_event      | None 
+
+## Filter type
+
+MSG Parse: This filter parses message content
+
+## Setup and Configuration
+
+* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer.
+* Review and update the splunk_index.csv file and set the index as required.
+* Refer to the admin manual for specific details of configuration to send Reliable syslog using RFC 3195 format, a typical logging configuration will include the following features.
+
+```
+config log memory filter
+
+set forward-traffic enable
+
+set local-traffic enable
+
+set sniffer-traffic disable
+
+set anomaly enable
+
+set voip disable
+
+set multicast-traffic enable
+
+set dns enable
+
+end
+
+config system global
+
+set cli-audit-log enable
+
+end
+
+config log setting
+
+set neighbor-event enable
+
+end
+
+```
+
+## Verification
+
+An active firewall will generate frequent events, in addition fortigate has the ability to test logging functionality using a build in command
+
+```
+diag log test
+```
+
+Verify timestamp, and host values match as expected    
+
+```
+index=<asconfigured> (sourcetype=fgt_log OR sourcetype=fgt_traffic OR sourcetype=fgt_utm) 
+```
+
+###UTM Message type
+
+![FortiGate UTM message](FortiGate_utm.png)
+
+### Traffic Message Type
+
+![FortiGate Traffic message](FortiGate_traffic.png)
+
+###Event Message Type
+![FortiGate Event message](FortiGate_event.png)
+
+Verify timestamp, and host values match as expected    
+
+# Vendor - PaloAlto
+
+## Product - NGFW
+
+| Ref            | Link                                                                                                    |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| Splunk Add-on  | https://splunkbase.splunk.com/app/2757/                                                                 |
+| Product Manual | https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/configure-syslog-monitoring.html                                                         |
+
+
+## Sourcetypes
+
+| sourcetype     | notes                                                                                                   |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| pan:log        | None                                                                                                    |
+| pan:traffic    | None                                                                                         |
+| pan:threat     | None                                                                                          |
+
+## Filter type
+
+MSG Parse: This filter parses message content
+
+## Setup and Configuration
+
+* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer.
+* Review and update the splunk_index.csv file and set the index as required.
+* Refer to the admin manual for specific details of configuration
+    * Select TCP or SSL transport option
+    * Select IETF Format
+    * Ensure the format of the event is not customized
+
+## Verification
+
+An active firewall will generate frequent events use the following search to validate events are present per source device
+
+```
+index=<asconfigured> sourcetype=pan:*| stats count by host
+```
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
@@ -0,0 +1,9 @@
+#Troubleshooting 
+
+## Syslog-ng Metrics 
+
+## Syslog-NG Events
+
+## Container Events
+
+# Monitoring
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -0,0 +1,8 @@
+site_name: Splunk Connect for Syslog
+theme: readthedocs
+nav:
+  - Home: 'index.md'
+  - Performance: 'performance.md'
+  - Getting Started: 'gettingstarted.md'
+  - Sources: 'sources.md'
+  - Troubleshooting: 'troubleshooting.md'
diff --git a/package/etc/conf.d/blocks/b_parsers.conf b/package/etc/conf.d/blocks/b_parsers.conf
@@ -47,9 +47,9 @@ block rewrite r_set_splunk(sourcetype(`splunk-sourcetype`)
       set("`vendor`", value(".splunk.vendor"));
       set("`category`", value(".splunk.category"));
       set("`filename`", value(".splunk.filename"));
-      set($FACILITY, value("fields.facility"));
-      set($LEVEL, value("fields.severity"));
-      set($LOGHOST, value("fields.syslog-server"));
+      set($FACILITY, value("fields.syslog_facility"));
+      set($LEVEL, value("fields.syslog_severity"));
+      set($LOGHOST, value("fields.syslog_server"));
   };
 
 #Used when for normal filters index and sourcetype must be set in a prior re-write or context
@@ -63,7 +63,7 @@ block rewrite r_set_splunk_basic(
       set("`vendor`", value(".splunk.vendor"));
       set("`category`", value(".splunk.category"));
       set("`filename`", value(".splunk.filename"));
-      set($FACILITY, value("fields.facility"));
-      set($LEVEL, value("fields.severity"));
-      set($LOGHOST, value("fields.syslog-server"));
+      set($FACILITY, value("fields.syslog_facility"));
+      set($LEVEL, value("fields.syslog_severity"));
+      set($LOGHOST, value("fields.syslog_server"));
   };
diff --git a/package/etc/conf.d/destinations/d_destinations.conf b/package/etc/conf.d/destinations/d_destinations.conf
@@ -17,15 +17,16 @@ destination d_hec {
          method("POST")
          log-fifo-size(`splunk-log-fifo-size`)
          workers(`SYSLOGNG_HEC_WORKERS`)
-         batch-lines(500)
-         batch-bytes(512Kb)
+         batch-lines(1000)
+         batch-bytes(2048Kb)
          batch-timeout(3)
          timeout(15)
          user_agent("syslog-ng User Agent")
          user("syslog-ng")
+         headers("Connection: close")
          password("`SPLUNK_HEC_TOKEN`")
          persist-name("splunk")
-         disk-buffer(mem-buf-length(500)
+         disk-buffer(mem-buf-length(15000)
                      disk-buf-size(20000)
                      reliable(no)
                      dir("/opt/syslog-ng/var/data/disk-buffer/"))
@@ -44,7 +45,7 @@ destination d_hec {
                  source=${HOST_FROM}
                  sourcetype=${.splunk.sourcetype}
                  index=${.splunk.index}
-                 event=$(template ${.splunk.template})
+                 event=$(template ${.splunk.template} $(template `splunk-default-template`))
                  fields.*)')
         );
     };