Merge pull request #285 from splunk/update/CEF-data-sources

Refactor old MICROFOCUS_ARCSIGHT log path to CEF

docs/sources/CommonEventFormat/index.md

@@ -0,0 +1,70 @@
+# Vendor - Common Event Format Data Sources
+
+## Product - Various products that send CEF-format messages via syslog
+
+Each CEF product should have their own source entry in this documentation set.  In a departure
+from normal configuration, all CEF products should use the "CEF" version of the unique port and
+archive envrionmetn variable settings (rather than a unique one per product), as the CEF log path
+handles all products sending events to SC4S in the CEF format. Examples of this include Arcsight,
+Imperva, and Cyberark.  Therefore, the CEF environment varialbes for unique port, archive, etc.
+should be set only _once_.
+
+If your deployment has multiple CEF devices that send to more than one port,
+set the CEF unique port variable(s) to just one of the ports in use.  Then, map the others with
+container networking to the port chosen.  Example: If you have three CEF devices, sending on TCP
+ports 2000,2001, and 2002, set `SC4S_LISTEN_CEF_TCP_PORT=2000`.  Then, map the other two with
+container networking, e.g. `-p 2000:2000 -p 2001:2000 -p 2002:2000`.  This will route all
+three ports to TCP port 2000 inside the container, and the single CEF log path will properly
+process data from all three devices.
+
+The source documentation included below is a reference baseline for any product that sends data
+using the CEF log path.
+
+
+| Ref            | Link                                                                                                    |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| Splunk Add-on CEF | https://bitbucket.org/SPLServices/ta-cef-for-splunk/downloads/                                                              |
+| Product Manual | https://docs.imperva.com/bundle/cloud-application-security/page/more/log-configuration.htm                                                        |
+
+
+### Sourcetypes
+
+| sourcetype     | notes                                                                                                   |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| cef        | Common sourcetype                                                                                                 |
+
+### Typical Source
+
+| source     | notes                                                                                                   |
+|----------------|---------------------------------------------------------------------------------------------------------|
+| Varies        | Varies                                                                                               |
+
+### Typical Index Configuration
+
+| key            | source     | index          | notes          |
+|----------------|----------------|----------------|----------------|
+| Vendor_Product      | Varies      | main          | none          |
+
+### Filter type
+
+MSG Parse: This filter parses message content
+
+### Options
+
+| Variable       | default        | description    |
+|----------------|----------------|----------------|
+| SC4S_LISTEN_CEF_UDP_PORT      | empty string      | Enable a UDP port for this specific vendor product using the number defined |
+| SC4S_LISTEN_CEF_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+| SC4S_LISTEN_CEF_TLS_PORT      | empty string      | Enable a TLS  port for this specific vendor product using the number defined |
+| SC4S_ARCHIVE_CEF | no | Enable archive to disk for this specific source |
+| SC4S_DEST_CEF_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | 
+
+### Verification
+
+An active site will generate frequent events use the following search to check for new events
+
+Verify timestamp, and host values match as expected    
+
+```
+index=<asconfigured> (sourcetype=cef source=<asconfigured>)
+```

docs/sources/CyberArk/index.md

@@ -28,7 +28,11 @@ MSG Parse: This filter parses message content
 
 | Variable       | default        | description    |
 |----------------|----------------|----------------|
-| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+| SC4S_LISTEN_CEF_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+
+* NOTE:  Set only _one_ set of CEF variables for the entire SC4S deployment, regardless of how
+many ports are in use by this CEF source (or any others).  See the "Common Event Format" source
+documentation for more information.
 
 ### Verification
 
@@ -68,7 +72,11 @@ MSG Parse: This filter parses message content
 
 | Variable       | default        | description    |
 |----------------|----------------|----------------|
-| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+| SC4S_LISTEN_CEF_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+
+* NOTE:  Set only _one_ set of CEF variables for the entire SC4S deployment, regardless of how
+many ports are in use by this CEF source (or any others).  See the "Common Event Format" source
+documentation for more information.
 
 ### Verification
 

docs/sources/Imperva/index.md

@@ -25,7 +25,7 @@
 
 | key            | source     | index          | notes          |
 |----------------|----------------|----------------|----------------|
-| cef_Incapsula_SIEMintegration      | Imperva:Incapsula      | netwaf          | none          |
+| Incapsula_SIEMintegration      | Imperva:Incapsula      | netwaf          | none          |
 
 ### Filter type
 
@@ -37,10 +37,14 @@ Note listed for reference processing utilizes the Microsoft ArcSight log path as
 
 | Variable       | default        | description    |
 |----------------|----------------|----------------|
-| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
-| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_UDP_PORT      | empty string      | Enable a UDP port for this specific vendor product using the number defined |
-| SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT | no | Enable archive to disk for this specific source |
-| SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | 
+| SC4S_LISTEN_CEF_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+| SC4S_LISTEN_CEF_UDP_PORT      | empty string      | Enable a UDP port for this specific vendor product using the number defined |
+| SC4S_ARCHIVE_CEF | no | Enable archive to disk for this specific source |
+| SC4S_DEST_CEF_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | 
+
+* NOTE:  Set only _one_ set of CEF variables for the entire SC4S deployment, regardless of how
+many ports are in use by this CEF source (or any others).  See the "Common Event Format" source
+documentation for more information.
 
 ### Verification
 
@@ -50,4 +54,4 @@ Verify timestamp, and host values match as expected
 
 ```
 index=<asconfigured> (sourcetype=cef source="Imperva:Incapsula")
-```
+```

docs/sources/Microfocus/index.md

@@ -1,6 +1,6 @@
-# Vendor - Microfocus ArcSight
+# Vendor - MicroFocus Arcsight
 
-## Product - Internal Agent Events
+## Product - Arcsight Internal Agent
 
 | Ref            | Link                                                                                                    |
 |----------------|---------------------------------------------------------------------------------------------------------|
@@ -24,7 +24,7 @@
 
 | key            | source     | index          | notes          |
 |----------------|----------------|----------------|----------------|
-| cef_ArcSight_ArcSight      | ArcSight:ArcSight      | main          | none          |
+| ArcSight_ArcSight      | ArcSight:ArcSight      | main          | none          |
 
 ### Filter type
 
@@ -34,7 +34,12 @@ MSG Parse: This filter parses message content
 
 | Variable       | default        | description    |
 |----------------|----------------|----------------|
-| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+| SC4S_LISTEN_CEF_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT      | empty string      | Deprecated equivalent of above variable.  This is included for backward compatibility and will be removed in a future version. _Do not use_ in new installations. |
+
+* NOTE:  Set only _one_ set of CEF variables for the entire SC4S deployment, regardless of how
+many ports are in use by this CEF source (or any others).  See the "Common Event Format" source
+documentation for more information.
 
 ### Verification
 
@@ -46,7 +51,7 @@ Verify timestamp, and host values match as expected
 index=<asconfigured> (sourcetype=cef source="ArcSight:ArcSight")
 ```
 
-## Product - Microsoft Windows
+## Product - Arcsight Microsoft Windows (CEF)
 
 | Ref            | Link                                                                                                    |
 |----------------|---------------------------------------------------------------------------------------------------------|
@@ -72,8 +77,8 @@ index=<asconfigured> (sourcetype=cef source="ArcSight:ArcSight")
 
 | key            | source     | index          | notes          |
 |----------------|----------------|----------------|----------------|
-| cef_Microsoft_System or Application Event      | CEFEventLog:System or Application Event      | oswin          | none          |
-| cef_Microsoft_Microsoft Windows      | CEFEventLog:Microsoft Windows      | oswinsec         | none          |
+| Microsoft_System or Application Event      | CEFEventLog:System or Application Event      | oswin          | none          |
+| Microsoft_Microsoft Windows      | CEFEventLog:Microsoft Windows      | oswinsec         | none          |
 
 ### Filter type
 
@@ -83,10 +88,15 @@ MSG Parse: This filter parses message content
 
 | Variable       | default        | description    |
 |----------------|----------------|----------------|
-| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
-| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_UDP_PORT      | empty string      | Enable a UDP port for this specific vendor product using the number defined |
-| SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT | no | Enable archive to disk for this specific source |
-| SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | 
+| SC4S_LISTEN_CEF_TCP_PORT      | empty string      | Enable a TCP port for this specific vendor product using the number defined |
+| SC4S_LISTEN_CEF_UDP_PORT      | empty string      | Enable a UDP port for this specific vendor product using the number defined |
+| SC4S_ARCHIVE_CEF | no | Enable archive to disk for this specific source |
+| SC4S_DEST_CEF_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | 
+| SC4S_WWW_XXX_MICROFOCUS_ARCSIGHT_YYY_ZZZ | no | Deprecated equivalents of the above variables.  These are included for backward compatibility, and will be removed in a future version. _Do not use_ in new installations. | 
+
+* NOTE:  Set only _one_ set of CEF variables for the entire SC4S deployment, regardless of how
+many ports are in use by this CEF source (or any others).  See the "Common Event Format" source
+documentation for more information.
 
 ### Verification
 
@@ -96,4 +106,4 @@ Verify timestamp, and host values match as expected
 
 ```
 index=<asconfigured> (sourcetype=cef (source="CEFEventLog:Microsoft Windows" OR source="CEFEventLog:System or Application Event"))
-```
+```

mkdocs.yml

@@ -14,14 +14,15 @@ nav:
       - About: sources/index.md
       - Checkpoint: sources/Checkpoint/index.md
       - Cisco: sources/Cisco/index.md
+      - 'Common Event Format': sources/CommonEventFormat/index.md
       - CyberArk: sources/CyberArk/index.md
       - Forcepoint: sources/Forcepoint/index.md
       - Fortinet: sources/Fortinet/index.md
       - Imperva: sources/Imperva/index.md
       - Juniper: sources/Juniper/index.md
-      - Nix: sources/nix/index.md
       - Microfocus: sources/Microfocus/index.md
-      - 'Paloalto Networks': sources/PaloaltoNetworks/index.md
+      - Nix: sources/nix/index.md
+      - 'Palo Alto Networks': sources/PaloaltoNetworks/index.md
       - Proofpoint: sources/Proofpoint/index.md
       - Symantec: sources/Symantec/index.md
       - Ubiquiti: sources/Ubiquiti/index.md

package/etc/conf.d/filters/common_event_format/cef.conf

@@ -0,0 +1,4 @@
+
+filter f_cef {
+    program(CEF);
+};

package/etc/conf.d/log_paths/lp-microfocus_arcsight.conf.tmpl → package/etc/conf.d/log_paths/lp-common_event_format.conf.tmpl

@@ -1,9 +1,9 @@
-# Microfocus ArcSight
+# Common Event Format
 {{- /* The following provides a unique port source configuration if env var(s) are set */}}
-{{- $context := dict "port_id" "MICROFOCUS_ARCSIGHT" "parser" "rfc3164" }}
+{{- $context := dict "port_id" "CEF" "parser" "rfc3164" }}
 {{- tmpl.Exec "t/source_network.t" $context }}
 
-parser p_microfocus_arcsight_header {
+parser p_cef_header {
     csv-parser(
         columns("fields.sc4s_cef_version", "fields.cef_device_vendor", "fields.cef_device_product", "fields.cef_device_version", "fields.cef_device_event_class", "fields.cef_name", "fields.cef_severity", MESSAGE)
         delimiters(chars("|"))
@@ -15,19 +15,19 @@ parser p_microfocus_arcsight_header {
 
 };
 
-parser p_microfocus_arcsight_ts_rt {
+parser p_cef_ts_rt {
     date-parser(format("%s") template("${.cef.rt}")
     );
 };
-parser p_microfocus_arcsight_ts_end {
+parser p_cef_ts_end {
     date-parser(format("%s") template("${.cef.end}")
     );
 };
 
-parser p_microfocus_arcsight_source {
+parser p_cef_source {
     add-contextual-data(
         selector("${fields.cef_device_vendor}_${fields.cef_device_product}"),
-        database("conf.d/context/microfocus_arcsight_source.csv")
+        database("conf.d/context/common_event_format_source.csv")
         ignore-case(yes)
         prefix(".splunk.")
         default-selector("unknown")
@@ -36,18 +36,18 @@ parser p_microfocus_arcsight_source {
 
 log {
     junction {
-{{- if or (or (getenv  (print "SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT")) (getenv  (print "SC4S_LISTEN_MICROFOCUS_ARCSIGHT_UDP_PORT"))) (getenv  (print "SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TLS_PORT")) }}
+{{- if or (or (getenv  (print "SC4S_LISTEN_CEF_TCP_PORT")) (getenv  (print "SC4S_LISTEN_CEF_UDP_PORT"))) (getenv  (print "SC4S_LISTEN_CEF_TLS_PORT")) }}
         channel {
-        # Listen on the specified dedicated port(s) for MICROFOCUS_ARCSIGHT traffic
-            source (s_MICROFOCUS_ARCSIGHT);
+        # Listen on the specified dedicated port(s) for CEF traffic
+            source (s_CEF);
             flags (final);
 	    };
 {{- end}}
         channel {
-        # Listen on the default port (typically 514) for MICROFOCUS_ARCSIGHT traffic
+        # Listen on the default port (typically 514) for CEF traffic
             source (s_DEFAULT);
             filter(f_is_rfc3164);
-            filter(f_microfocus_arcsight);
+            filter(f_cef);
             flags(final);
         };
     };
@@ -56,7 +56,7 @@ log {
         r_set_splunk_dest_default(sourcetype("cef"), index("main"))
     };
 
-    parser (p_microfocus_arcsight_header);
+    parser (p_cef_header);
 
     rewrite {
         set("${fields.cef_device_vendor}_${fields.cef_device_product}", value("fields.sc4s_vendor_product"));
@@ -70,13 +70,13 @@ log {
     # If we have an rt or end field that is best we use the If trick here so if this parser fails
     # We don't get sent to fallback.
     if {
-        parser (p_microfocus_arcsight_ts_rt);
+        parser (p_cef_ts_rt);
     } elif {
-        parser (p_microfocus_arcsight_ts_end);
+        parser (p_cef_ts_end);
     } else {}; #Do nothing this is allows for both rt and end to be missing and still pass with the message ts
 
     #CEF TAs use the source as their bounds in props.conf
-    parser(p_microfocus_arcsight_source);
+    parser(p_cef_source);
 
     parser (compliance_meta_by_source);
 
@@ -85,11 +85,11 @@ log {
     #if we don't
     rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); };
 
-{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC" "no")) }}
+{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CEF_HEC" "no")) }}
     destination(d_hec);
 {{- end}}
 
-{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT" "no")) }}
+{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CEF" "no")) }}
     destination(d_archive);
 {{- end}}
 

package/etc/context_templates/splunk_index.csv

@@ -1,8 +1,10 @@
 #bluecoat_proxy,index,netproxy
-#cef_ArcSight_ArcSight,index,netwaf
-#cef_Incapsula_SIEMintegration,index,netwaf
-#cef_Microsoft_Microsoft Windows,index,oswinsec
-#cef_Microsoft_System or Application Event,index,oswin
+#ArcSight_ArcSight,index,netwaf
+#Cyber-Ark_Vault,index,netauth
+#CyberArk_PTA,index,main
+#Incapsula_SIEMintegration,index,netwaf
+#Microsoft_Microsoft Windows,index,oswinsec
+#Microsoft_System or Application Event,index,oswin
 #checkpoint_splunk,index,netops
 #checkpoint_splunk_dlp,index,netdlp
 #checkpoint_splunk_email,index,email

package/sbin/entrypoint.sh

@@ -1,6 +1,14 @@
 #!/usr/bin/env bash
 source scl_source enable rh-python36
 
+# The MICROFOCUS_ARCSIGHT unique port environment variables are currently deprecated
+# This will be removed when the MICROFOCUS_ARCSIGHT unique port environment variables are removed in version 2.0
+if [ ${SC4S_LISTEN_MICROFOCUS_ARCSIGHT_UDP_PORT} ]; then export SC4S_LISTEN_CEF_UDP_PORT=$SC4S_LISTEN_MICROFOCUS_ARCSIGHT_UDP_PORT; fi
+if [ ${SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT} ]; then export SC4S_LISTEN_CEF_TCP_PORT=$SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT; fi
+if [ ${SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TLS_PORT} ]; then export SC4S_LISTEN_CEF_TLS_PORT=$SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TLS_PORT; fi
+if [ ${SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT} ]; then export SC4S_ARCHIVE_CEF=$SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT; fi
+if [ ${SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC} ]; then export SC4S_DEST_CEF_HEC=$SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC; fi
+
 cd /opt/syslog-ng
 
 gomplate $(find . -name *.tmpl | sed -E 's/^(\/.*\/)*(.*)\..*$/--file=\2.tmpl --out=\2/') --template t=etc/go_templates/