Skip to content

Commit

Permalink
Merge pull request #335 from splunk/feature/zscaler-lss
Browse files Browse the repository at this point in the history
Feature/zscaler lss
  • Loading branch information
Ryan Faircloth authored and GitHub committed Mar 6, 2020
2 parents 7deb4f9 + 6aadb8b commit 9930103
Show file tree
Hide file tree
Showing 40 changed files with 691 additions and 61 deletions.
17 changes: 17 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,23 @@ syslog.
| SC4S_DEST_SPLUNK_HEC_TLS_VERIFY | yes(default) or no | verify HTTP(s) certificate |
| SC4S_DEST_SPLUNK_HEC_WORKERS | numeric | Number of destination workers (threads). Set this to the number of HEC endpoints up to a max of 32. |

## Alternate Destination Configuration

Alternate destinations other than HEC can be configured in SC4S. Global and/or source-specific forms of the
variables below can be used to send data to alternate destinations.

* NOTE: The administrator is responsible for ensuring that the alternate destinations are configured in the
local mount tree, and that syslog-ng properly parses them.

* NOTE: Do not include `d_hec` in any list of alternate destinations. The configuration of the default HEC destination is configured
separately from that of the alternates below.


| Variable | Values | Description |
|----------|---------------|-------------|
| SC4S_DEST_GLOBAL_ALTERNATES | Comma or space-separated list of syslog-ng destinations | Send all sources to alternate destinations |
| SC4S_DEST_<SOURCE>\_ALTERNATES | Comma or space-separated list of syslog-ng destiinations | Send specific sources to alternate syslog-ng destinations, e.g. SC4S_DEST_CISCO_ASA_ALTERNATES |

## SC4S Disk Buffer Configuration

Disk buffers in SC4S are allocated _per destination_. In the future as more destinations are supported, a separate list of variables
Expand Down
70 changes: 63 additions & 7 deletions docs/sources/Zscaler/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Vendor - Zscaler

## Product - All Products
## Product - ZIA

The ZScaler product manual includes and extensive section of configuration for multiple Splunk TCP input ports around page
26. When using SC4S these ports are not required and should not be used. Simply configure all outputs from the NSS to utilize
Expand All @@ -20,9 +20,6 @@ the IP or host name of the SC4S instance and port 514
| zscalernss-alerts | Requires format customization add ``\tvendor=Zscaler\tproduct=alerts`` immediately prior to the ``\n`` in the NSS Alert Web format. See Zscaler manual for more info. |
| zscalernss-dns | Requires format customization add ``\tvendor=Zscaler\tproduct=dns`` immediately prior to the ``\n`` in the NSS DNS format. See Zscaler manual for more info. |
| zscalernss-web | None |
| zscalernss-zpa-app | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. |
| zscalernss-zpa-auth | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa_auth`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. |
| zscalernss-zpa-connector | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa_auth_connector`` immediately prior to the ``\n`` in the LSS Connector format. See Zscaler manual for more info. |
| zscalernss-fw | Requires format customization add ``\tvendor=Zscaler\tproduct=fw`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. |


Expand All @@ -34,9 +31,6 @@ the IP or host name of the SC4S instance and port 514
| zscalernss_dns | zscalernss-dns | netdns | none |
| zscalernss_fw | zscalernss-fw | netfw | none |
| zscalernss_web | zscalernss-web | netproxy | none |
| zscalernss-zpa-app | zscalernss_zpa-app | netids | none |
| zscalernss-zpa-auth | zscalernss_zpa_auth | netauth | none |
| zscalernss-zpa-connector | zscalernss_zpa_connector | netops | none |


### Filter type
Expand Down Expand Up @@ -67,3 +61,65 @@ An active proxy will generate frequent events. Use the following search to valid
```
index=<asconfigured> sourcetype=zscalernss-* | stats count by host
```

## Product - LSS

The ZScaler product manual includes and extensive section of configuration for multiple Splunk TCP input ports around page
26. When using SC4S these ports are not required and should not be used. Simply configure all outputs from the LSS to utilize
the IP or host name of the SC4S instance and port 514


| Ref | Link |
|----------------|---------------------------------------------------------------------------------------------------------|
| Splunk Add-on | https://splunkbase.splunk.com/app/3865/ |
| Product Manual | https://community.zscaler.com/t/zscaler-splunk-app-design-and-installation-documentation/4728 |


### Sourcetypes

| sourcetype | notes |
|----------------|---------------------------------------------------------------------------------------------------------|
| zscalerlss-zpa-app | None |
| zscalerlss-zpa-auth | None |
| zscalerlss-zpa-bba | None |
| zscalerlss-zpa-connector | None |


### Sourcetype and Index Configuration

| key | sourcetype | index | notes |
|----------------|----------------|----------------|----------------|
| zscalernss-zpa-app | zscalerlss_zpa-app | netproxy | none |
| zscalernss-zpa-auth | zscalerlss_zpa_auth | netauth | none |
| zscalernss-zpa-bba | zscalerlss_zpa_auth | netproxy | none |
| zscalernss-zpa-connector | zscalerlss_zpa_connector | netproxy | none |


### Filter type

MSG Parse: This filter parses message content

### Setup and Configuration

* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer.
* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source.
* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration
* Select TCP or SSL transport option
* Ensure the format of the event is customized per Splunk documentation

### Options

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_ZSCALER_LSS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_ZSCALER_LSS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_ARCHIVE_ZSCALER_LSS | no | Enable archive to disk for this specific source |
| SC4S_DEST_ZSCALER_LSS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

### Verification

An active proxy will generate frequent events. Use the following search to validate events are present per source device

```
index=<asconfigured> sourcetype=zscalernss-* | stats count by host
```
19 changes: 19 additions & 0 deletions package/etc/conf.d/conflib/_common/syslog_format.conf
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,23 @@ filter f_is_no_parse{

rewrite set_rfc3164_no_version_string{
subst('(^<\d+>)\d', $1, value("MESSAGE"));
};
filter f_is_rfc3164_json{
match("rfc3164_json" value("fields.sc4s_syslog_format"))
};
rewrite set_rfc3164_json{
set("rfc3164_json" value("fields.sc4s_syslog_format"));
};

filter f_is_tcp_json{
match("tcp_json" value("fields.sc4s_syslog_format"))
};
rewrite set_tcp_json{
set("tcp_json" value("fields.sc4s_syslog_format"));
};

filter f_msg_is_tcp_json{
match("rfc3164_json" value("fields.sc4s_syslog_format"))
or
match("tcp_json" value("fields.sc4s_syslog_format"))
};
7 changes: 6 additions & 1 deletion package/etc/conf.d/filters/zscaler/nss.conf
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
filter f_zscaler_nss {
message('\tvendor=Zscaler\t');
};
or message('^ZscalerNSS:');
};
# filter f_zscaler_lss {
# match()

# };
10 changes: 9 additions & 1 deletion package/etc/conf.d/log_paths/lp-checkpoint_splunk.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -88,5 +88,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CHECKPOINT_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CHECKPOINT_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
};
10 changes: 9 additions & 1 deletion package/etc/conf.d/log_paths/lp-cisco_acs.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,15 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CISCO_ACS_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CISCO_ACS_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};

};
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-cisco_apic.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -52,5 +52,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CISCO_APIC_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CISCO_APIC_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-cisco_asa.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CISCO_ASA_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CISCO_ASA_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-cisco_asa_legacy.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -41,5 +41,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CISCO_ASA_LEGACY_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CISCO_ASA_LEGACY_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-cisco_ios.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CISCO_IOS_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CISCO_IOS_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-cisco_ise.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,14 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CISCO_ISE_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CISCO_ISE_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};

Expand Down
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-cisco_meraki.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CISCO_MERAKI_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CISCO_MERAKI_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-cisco_nxos.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CISCO_NXOS_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CISCO_NXOS_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-cisco_ucm.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -58,5 +58,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CISCO_UCM_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CISCO_UCM_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-citrix-netscaler.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CITRIX_NETSCALER_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CITRIX_NETSCALER_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-common_event_format.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -97,5 +97,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_CEF_ALTERNATES")) }}
{{ getenv "SC4S_DEST_CEF_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_FORCEPOINT_WEBPROTECT_ALTERNATES")) }}
{{ getenv "SC4S_DEST_FORCEPOINT_WEBPROTECT_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-fortinet.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -114,5 +114,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_FORTINET_ALTERNATES")) }}
{{ getenv "SC4S_DEST_FORTINET_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-infoblox.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -70,5 +70,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_INFOBLOX_ALTERNATES")) }}
{{ getenv "SC4S_DEST_INFOBLOX_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-juniper_idp.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_JUNIPER_IDP_ALTERNATES")) }}
{{ getenv "SC4S_DEST_JUNIPER_IDP_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
8 changes: 8 additions & 0 deletions package/etc/conf.d/log_paths/lp-juniper_junos.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -56,5 +56,13 @@ log {
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_JUNIPER_JUNOS_ALTERNATES")) }}
{{ getenv "SC4S_DEST_JUNIPER_JUNOS_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
Loading

0 comments on commit 9930103

Please sign in to comment.