Manual Merge (#599)

* [filterchange] Cisco Nexus (#594) Adjust airOS and Nexsus to address event with no HOST but with TZ indicator * [doc] Palo filter does not support UDP due to limitations of message size (#591) * [filteradd] Carbon Black Protection CEF format (#590) * [fix] noise in SC4S logs from goss (#595) * Troubleshooting/problem >solution (#593) (#596) * Troubleshooting/problem >solution (#593) * Troubleshooting guide format problem solution * Minor chnages * minor changes * Revert "Troubleshooting/problem >solution (#593)" (#597) This reverts commit 20a1116. Co-authored-by: nandinivij <61885842+nandinivij@users.noreply.github.com> Co-authored-by: nandinivij <61885842+nandinivij@users.noreply.github.com>
CSVD · Jul 30, 2020 · f79c368 · f79c368
1 parent 1ba2e4d
commit f79c368
Show file tree

Hide file tree

Showing 7 changed files with 65 additions and 10 deletions.
diff --git a/docs/gettingstarted/index.md b/docs/gettingstarted/index.md
@@ -37,6 +37,7 @@ using the SC4S defaults. SC4S can be easily customized to use different indexes
 
 * email
 * epav
+* epintel
 * netauth
 * netdlp
 * netdns
@@ -46,6 +47,7 @@ using the SC4S defaults. SC4S can be easily customized to use different indexes
 * netwaf
 * netproxy
 * netipam
+* oswin
 * oswinsec
 * osnix
 * em_metrics (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index)

diff --git a/docs/sources/PaloaltoNetworks/index.md b/docs/sources/PaloaltoNetworks/index.md
@@ -50,7 +50,6 @@ MSG Parse: This filter parses message content
 | Variable       | default        | description    |
 |----------------|----------------|----------------|
 | SC4S_LISTEN_PALOALTO_PANOS_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
-| SC4S_LISTEN_PALOALTO_PANOS_UDP_PORT      | empty string      | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
 | SC4S_ARCHIVE_PALOALTO_PANOS | no | Enable archive to disk for this specific source |
 | SC4S_DEST_PALOALTO_PANOS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | 
 

diff --git a/docs/sources/VMWare/index.md b/docs/sources/VMWare/index.md
@@ -1,5 +1,61 @@
 # Vendor - Dell - VMware
 
+## Product - Carbon Black Protection
+
+| Ref            | Link                                                                                                    |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| Splunk Add-on CEF | none                                                  |
+| Splunk Add-on Source Specific | https://bitbucket.org/SPLServices/ta-cef-imperva-incapsula/downloads/                                                               |
+
+
+### Sourcetypes
+
+| sourcetype     | notes                                                                                                   |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| cef        | Common sourcetype                                                                                                 |
+
+### Source
+
+| source     | notes                                                                                                   |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| carbonblack:protection:cef       | Note this method of onboarding is not recommended for a more complete experience utilize the json format supported by he product with hec or s3                                                                            |
+
+### Index Configuration
+
+| key            | source     | index          | notes          |
+|----------------|----------------|----------------|----------------|
+| Carbon Black_Protection      | carbonblack:protection:cef      | epintel          | none          |
+
+### Filter type
+
+MSG Parse: This filter parses message content
+
+### Options
+
+Note listed for reference processing utilizes the Microsoft ArcSight log path as this format is a subtype of CEF
+
+| Variable       | default        | description    |
+|----------------|----------------|----------------|
+| SC4S_LISTEN_CEF_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
+| SC4S_LISTEN_CEF_UDP_PORT      | empty string      | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
+| SC4S_ARCHIVE_CEF | no | Enable archive to disk for this specific source |
+| SC4S_DEST_CEF_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |
+
+* NOTE:  Set only _one_ set of CEF variables for the entire SC4S deployment, regardless of how
+many ports are in use by this CEF source (or any others).  See the "Common Event Format" source
+documentation for more information.
+
+### Verification
+
+An active site will generate frequent events use the following search to check for new events
+
+Verify timestamp, and host values match as expected
+
+```
+index=<asconfigured> (sourcetype=cef source="carbonblack:protection:cef")
+```
+
+
 ## Product - vSphere - ESX NSX (Controller, Manager, Edge)
 
 

diff --git a/package/etc/conf.d/filters/cisco/cisco_syslog.conf b/package/etc/conf.d/filters/cisco/cisco_syslog.conf
@@ -45,22 +45,19 @@ parser cisco-parser-ex{
         } elif {
             # Cisco Nexus Switch
             filter {
-                message('^<(?<pri>\d*)>:? ?20\d\d (\w\w\w \d+ \d\d:\d\d:\d\d(?:.\d{3,3})?) ([^ ]+) ((%[^\: ]+)\:? ?.*)'
+                message('^<(?<pri>\d*)>:? ?20\d\d (\w\w\w \d+ \d\d:\d\d:\d\d(?:.\d{3,3})?)(?: [A-Z]{3}:)? ([^ ]+)? ?((%[^\: ]+)\:? ?.*)'
                 flags(store-matches));
             };            
             rewrite {
                 set(
                     "${3}",
-                    value("HOST")                
+                    value("HOST")     
+                    condition(not match('^\d+$', value('3')) and match('^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$|^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$' value('3')))           
                 );            
                 set(
                     "${4}",
                     value("MESSAGE")                
                 );
-                set(
-                    "${5}",
-                    value("PROGRAM")                
-                );
             };
             parser { date-parser-nofilter(format(
                 '%b %d %H:%M:%S.%f',

diff --git a/package/etc/context_templates/splunk_metadata.csv.example b/package/etc/context_templates/splunk_metadata.csv.example
@@ -2,8 +2,8 @@ bluecoat_proxy,index,netproxy
 brocade_syslog,index,netops
 ArcSight_ArcSight,index,main
 ArcSight_ArcSight,source,ArcSight:ArcSight
-Carbon Black_Protection,sourcetype,carbonblack:protection:cef
-Carbon Black_Protection,index,cb:cef
+Carbon Black_Protection,source,carbonblack:protection:cef
+Carbon Black_Protection,index,epintel
 Cyber-Ark_Vault,index,netauth
 Cyber-Ark_Vault,sourcetype,cyberark:epv:cef
 CyberArk_PTA,index,main

diff --git a/package/sbin/entrypoint.sh b/package/sbin/entrypoint.sh
@@ -108,7 +108,7 @@ echo sc4s version=$(cat /VERSION) >/opt/syslog-ng/var/log/syslog-ng.out
 # Use gomplate to pick up default listening ports for health check
 echo starting goss
 gomplate --file /goss.yaml.tmpl --out /goss.yaml
-goss -g /goss.yaml serve --format json >/dev/null &
+goss -g /goss.yaml serve --format json >/dev/null 2>/dev/null &
 
 echo syslog-ng starting
 /opt/syslog-ng/bin/persist-tool add /opt/syslog-ng/etc/reset_persist -o /opt/syslog-ng/var

diff --git a/tests/test_cisco_ios.py b/tests/test_cisco_ios.py
@@ -44,6 +44,7 @@
     "{{ mark }}22191: {{ host }}: 022546: {{ bsd }}.{{ millisec }} CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:dfa_service_admin  logged command:!exec: enable",
     "{{ mark }}{{ host }}: {{ year }} {{ bsd }} CDT: %MODULE-2-MOD_SOMEPORTS_FAILED: Module 13 (Serial number: JAF12345678) reported failure on ports Eth13/17-20 (Ethernet) due to hardware not accessible in device DEV_CLP_FWD(device error 0xca804200)",
     "{{ mark }}{{ host }}: {{ year }} {{ bsd }}.{{ millisec }} CDT: %MODULE-2-MOD_SOMEPORTS_FAILED: Module 13 (Serial number: JAF12345678) reported failure on ports Eth13/17-20 (Ethernet) due to hardware not accessible in device DEV_CLP_FWD(device error 0xca804200)",
+    "{{ mark }}: 2020 {{ bsd }} EDT: %L2FM-4-L2FM_MAC_MOVE: Mac e4c7.2266.f741 in vlan 1159 has moved from  100.16.4513 to  {{ host }}"
 ]
 testdata_badtime = [
     "{{ mark }}{{ seq }}: {{ host }}: 6340004: *{{ bsd }}: %SEC-6-IPACCESSLOGP: list INET-BLOCK permitted tcp 192.168.20.252(55244) -> 10.54.3.178(44818), 1 packet",