Skip to content

Commit

Permalink
Merge pull request #338 from splunk/master
Browse files Browse the repository at this point in the history
Merge back
  • Loading branch information
Ryan Faircloth authored and GitHub committed Mar 6, 2020
2 parents c53001d + 9930103 commit de96be8
Show file tree
Hide file tree
Showing 7 changed files with 404 additions and 56 deletions.
70 changes: 63 additions & 7 deletions docs/sources/Zscaler/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Vendor - Zscaler

## Product - All Products
## Product - ZIA

The ZScaler product manual includes and extensive section of configuration for multiple Splunk TCP input ports around page
26. When using SC4S these ports are not required and should not be used. Simply configure all outputs from the NSS to utilize
Expand All @@ -20,9 +20,6 @@ the IP or host name of the SC4S instance and port 514
| zscalernss-alerts | Requires format customization add ``\tvendor=Zscaler\tproduct=alerts`` immediately prior to the ``\n`` in the NSS Alert Web format. See Zscaler manual for more info. |
| zscalernss-dns | Requires format customization add ``\tvendor=Zscaler\tproduct=dns`` immediately prior to the ``\n`` in the NSS DNS format. See Zscaler manual for more info. |
| zscalernss-web | None |
| zscalernss-zpa-app | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. |
| zscalernss-zpa-auth | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa_auth`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. |
| zscalernss-zpa-connector | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa_auth_connector`` immediately prior to the ``\n`` in the LSS Connector format. See Zscaler manual for more info. |
| zscalernss-fw | Requires format customization add ``\tvendor=Zscaler\tproduct=fw`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. |


Expand All @@ -34,9 +31,6 @@ the IP or host name of the SC4S instance and port 514
| zscalernss_dns | zscalernss-dns | netdns | none |
| zscalernss_fw | zscalernss-fw | netfw | none |
| zscalernss_web | zscalernss-web | netproxy | none |
| zscalernss-zpa-app | zscalernss_zpa-app | netids | none |
| zscalernss-zpa-auth | zscalernss_zpa_auth | netauth | none |
| zscalernss-zpa-connector | zscalernss_zpa_connector | netops | none |


### Filter type
Expand Down Expand Up @@ -67,3 +61,65 @@ An active proxy will generate frequent events. Use the following search to valid
```
index=<asconfigured> sourcetype=zscalernss-* | stats count by host
```

## Product - LSS

The ZScaler product manual includes and extensive section of configuration for multiple Splunk TCP input ports around page
26. When using SC4S these ports are not required and should not be used. Simply configure all outputs from the LSS to utilize
the IP or host name of the SC4S instance and port 514


| Ref | Link |
|----------------|---------------------------------------------------------------------------------------------------------|
| Splunk Add-on | https://splunkbase.splunk.com/app/3865/ |
| Product Manual | https://community.zscaler.com/t/zscaler-splunk-app-design-and-installation-documentation/4728 |


### Sourcetypes

| sourcetype | notes |
|----------------|---------------------------------------------------------------------------------------------------------|
| zscalerlss-zpa-app | None |
| zscalerlss-zpa-auth | None |
| zscalerlss-zpa-bba | None |
| zscalerlss-zpa-connector | None |


### Sourcetype and Index Configuration

| key | sourcetype | index | notes |
|----------------|----------------|----------------|----------------|
| zscalernss-zpa-app | zscalerlss_zpa-app | netproxy | none |
| zscalernss-zpa-auth | zscalerlss_zpa_auth | netauth | none |
| zscalernss-zpa-bba | zscalerlss_zpa_auth | netproxy | none |
| zscalernss-zpa-connector | zscalerlss_zpa_connector | netproxy | none |


### Filter type

MSG Parse: This filter parses message content

### Setup and Configuration

* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer.
* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source.
* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration
* Select TCP or SSL transport option
* Ensure the format of the event is customized per Splunk documentation

### Options

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_ZSCALER_LSS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_ZSCALER_LSS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_ARCHIVE_ZSCALER_LSS | no | Enable archive to disk for this specific source |
| SC4S_DEST_ZSCALER_LSS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

### Verification

An active proxy will generate frequent events. Use the following search to validate events are present per source device

```
index=<asconfigured> sourcetype=zscalernss-* | stats count by host
```
19 changes: 19 additions & 0 deletions package/etc/conf.d/conflib/_common/syslog_format.conf
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,23 @@ filter f_is_no_parse{

rewrite set_rfc3164_no_version_string{
subst('(^<\d+>)\d', $1, value("MESSAGE"));
};
filter f_is_rfc3164_json{
match("rfc3164_json" value("fields.sc4s_syslog_format"))
};
rewrite set_rfc3164_json{
set("rfc3164_json" value("fields.sc4s_syslog_format"));
};

filter f_is_tcp_json{
match("tcp_json" value("fields.sc4s_syslog_format"))
};
rewrite set_tcp_json{
set("tcp_json" value("fields.sc4s_syslog_format"));
};

filter f_msg_is_tcp_json{
match("rfc3164_json" value("fields.sc4s_syslog_format"))
or
match("tcp_json" value("fields.sc4s_syslog_format"))
};
7 changes: 6 additions & 1 deletion package/etc/conf.d/filters/zscaler/nss.conf
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
filter f_zscaler_nss {
message('\tvendor=Zscaler\t');
};
or message('^ZscalerNSS:');
};
# filter f_zscaler_lss {
# match()

# };
98 changes: 98 additions & 0 deletions package/etc/conf.d/log_paths/lp-zscaler_lss.conf.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Zscaler

{{- /* The following provides a unique port source configuration if env var(s) are set */}}
{{- $context := dict "port_id" "ZSCALER_LSS" "parser" "common" }}
{{- tmpl.Exec "t/source_network.t" $context }}

log {
junction {
{{- if or (or (getenv (print "SC4S_LISTEN_ZSCALER_LSS_TCP_PORT")) (getenv (print "SC4S_LISTEN_ZSCALER_LSS_UDP_PORT"))) (getenv (print "SC4S_LISTEN_ZSCALER_LSS_TLS_PORT")) }}
channel {
# Listen on the specified dedicated port(s) for ZSCALER_LSS traffic
source (s_ZSCALER_LSS);
flags (final);
};
{{- end}}
channel {
# Listen on the default port (typically 514) for ZSCALER_LSS traffic
source (s_DEFAULT);
filter(f_msg_is_tcp_json);
flags(final);
};
};
parser {
#.jsonLog.Timestamp Mar 04 20:37:53 2020
date-parser(
format("%b %d %H:%M:%S %Y",
"%h %d %H:%M:%S %Y",
"%b %d %k:%M:%S %Y",
"%h %d %k:%M:%S %Y")
template("${.json.LogTimestamp}")
time-zone({{- getenv "SC4S_DEFAULT_TIMEZONE" "GMT"}})
flags(guess-timezone)
);

};
if {
filter {
match('.' value('.json.ClientZEN'))
and match('.' value('.json.AppGroup'))
and match('.' value('.json.Application'))
};
rewrite { r_set_splunk_dest_default(sourcetype("zscalerlss-zpa-app"), index("netproxy"))};
parser { p_add_context_splunk(key("zscaler_lss")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };
} elif {
filter {
match('.' value('.json.Exporter'))
and match('.' value('.json.Customer'))
and match('.' value('.json.ConnectionID'))
};
rewrite { r_set_splunk_dest_default(sourcetype("zscalerlss-zpa-bba"), index("netproxy"))};
parser { p_add_context_splunk(key("zscaler_lss")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };
} elif {
filter {
match('.' value('.json.Connector'))
and match('.' value('.json.Customer'))
and match('.' value('.json.ConnectorGroup'))
};
rewrite { r_set_splunk_dest_default(sourcetype("zscalerlss-zpa-connector"), index("netproxy"))};
parser { p_add_context_splunk(key("zscaler_lss")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };
} elif {
filter {
match('.' value('.json.SAMLAttributes'))
and match('.' value('.json.Customer'))
};
rewrite { r_set_splunk_dest_default(sourcetype("zscalerlss-zpa-auth"), index("netauth"))};
parser { p_add_context_splunk(key("zscaler_lss")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };
};


parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };

{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_ZSCALER_LSS_HEC" "no")) }}
destination(d_hec);
{{- end}}

{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_ZSCALER_LSS" "no")) }}
destination(d_archive);
{{- end}}

{{- if (print (getenv "SC4S_DEST_GLOBAL_ALTERNATES")) }}
{{ getenv "SC4S_DEST_GLOBAL_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

{{- if (print (getenv "SC4S_DEST_ZSCALER_LSS_ALTERNATES")) }}
{{ getenv "SC4S_DEST_ZSCALER_LSS_ALTERNATES" | regexp.ReplaceLiteral "^" "destination(" | regexp.ReplaceLiteral "[, ]+" ");\n destination(" }});
{{- end }}

flags(flow-control,final);
};
83 changes: 37 additions & 46 deletions package/etc/conf.d/log_paths/lp-zscaler_nss.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -20,57 +20,48 @@ log {
flags(final);
};
};

parser { date-parser(format("%Y-%m-%d %H:%M:%S") template('$(substr "$LEGACY_MSGHDR$MSG" "0" "19")') flags(guess-timezone)); };
rewrite {
set("zscaler_nss", value("fields.sc4s_vendor_product"));
subst("^[^\t]+\t", "", value("MESSAGE"), flags("global"));
};
parser {
#basic parsing
kv-parser(prefix(".kv.") pair-separator("\t") template("${MSG}"));
};

if (match("alerts" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-alerts"), index("main"))};
parser { p_add_context_splunk(key("zscaler_alerts")); };
} elif (match("dns" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-dns"), index("netdns"))};
parser { p_add_context_splunk(key("zscaler_dns")); };
} elif (match("fw" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-fw"), index("netfw"))};
parser { p_add_context_splunk(key("zscaler_fw")); };
} elif (match("NSS" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-web"), index("netproxy"))};
parser { p_add_context_splunk(key("zscaler_web")); };
} elif (match("audit" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-zia-audit"), index("netops"))};
parser { p_add_context_splunk(key("zscaler_zia_audit")); };
} elif (match("sandbox" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-zia-sandbox"), index("main"))};
parser { p_add_context_splunk(key("zscaler_zia_sandbox")); };
} elif (match("zpa" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-zpa-app"), index("netids"))};
parser { p_add_context_splunk(key("zscaler_zpa")); };
} elif (match("zpa_auth" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-zpaauth"), index("netauth"))};
parser { p_add_context_splunk(key("zscaler_zpa_auth")); };
} elif (match("zpa_auth_connector" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-zpa-connector"), index("netops"))};
parser { p_add_context_splunk(key("zscaler_zpa_connector")); };
} elif (match("zpa_bba" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-bba"), index("main"))};
parser { p_add_context_splunk(key("zscaler_zpa_bba")); };
if (message('^ZscalerNSS:')) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-alerts"), index("netops"))};
parser { p_add_context_splunk(key("zscaler_alerts")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };
} else {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-unknown"), index("main"))};
parser { date-parser(format("%Y-%m-%d %H:%M:%S") template('$(substr "$LEGACY_MSGHDR$MSG" "0" "19")') flags(guess-timezone)); };
rewrite {
set("zscaler_nss", value("fields.sc4s_vendor_product"));
subst("^[^\t]+\t", "", value("MESSAGE"), flags("global"));
};
parser {
p_add_context_splunk(key("zscaler_nss"));
#basic parsing
kv-parser(prefix(".kv.") pair-separator("\t") template("${MSG}"));
};
};

parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };

if (match("dns" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-dns"), index("netdns"))};
parser { p_add_context_splunk(key("zscaler_dns")); };
} elif (match("fw" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-fw"), index("netfw"))};
parser { p_add_context_splunk(key("zscaler_fw")); };
} elif (match("NSS" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-web"), index("netproxy"))};
parser { p_add_context_splunk(key("zscaler_web")); };
} elif (match("audit" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-zia-audit"), index("netops"))};
parser { p_add_context_splunk(key("zscaler_zia_audit")); };
} elif (match("sandbox" value(".kv.product"))) {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-zia-sandbox"), index("main"))};
parser { p_add_context_splunk(key("zscaler_zia_sandbox")); };
} else {
rewrite { r_set_splunk_dest_default(sourcetype("zscalernss-unknown"), index("main"))};
parser {
p_add_context_splunk(key("zscaler_nss"));
};
};

parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };
};
{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_ZSCALER_NSS_HEC" "no")) }}
destination(d_hec);
{{- end}}
Expand Down
39 changes: 39 additions & 0 deletions package/etc/go_templates/source_network.t
Original file line number Diff line number Diff line change
Expand Up @@ -96,11 +96,28 @@ source s_{{ .port_id }} {
rewrite (r_cisco_ucm_message);
{{ else if eq .parser "no_parse" }}
rewrite(set_no_parse);
{{ else if eq .parser "tcp_json" }}
filter { message('^{') and message('}$') };
parser {
json-parser(
prefix('.json.')
);
};
rewrite(set_tcp_json);
{{ else }}
if {
filter(f_citrix_netscaler_message);
parser(p_citrix_netscaler_date);
rewrite(r_citrix_netscaler_message);
} elif {
#JSON over IP its not syslog but it can work
filter { message('^{') and message('}$') };
parser {
json-parser(
prefix('.json.')
);
};
rewrite(set_tcp_json);
} elif {
filter(f_rfc5424_strict);
parser {
Expand Down Expand Up @@ -135,6 +152,28 @@ source s_{{ .port_id }} {
syslog-parser(time-zone({{- getenv "SC4S_DEFAULT_TIMEZONE" "GMT"}}) flags(guess-timezone {{- if (conv.ToBool (getenv "SC4S_SOURCE_STORE_RAWMSG" "no")) }} store-raw-message {{- end}}));
};
rewrite(set_rfc3164);
if {
filter { message('^{') and message('}$') };
parser {
json-parser(
prefix('.json.')
);
};
rewrite(set_rfc3164_json);
} elif {
filter { match('^{' value('LEGACY_MSGHDR')) and message('}$') };
parser {
json-parser(
prefix('.json.')
template('${LEGACY_MSGHDR}${MSG}')
);
};
rewrite {
set('${LEGACY_MSGHDR}${MSG}' value('MSG'));
unset(value('LEGACY_MSGHDR'));
};
rewrite(set_rfc3164_json);
};
};
{{ end }}
rewrite(r_set_splunk_default);
Expand Down
Loading

0 comments on commit de96be8

Please sign in to comment.