Skip to content

Commit

Permalink
Merge branch 'feature/checkpoint-noise' of https://github.com/splunk/…
Browse files Browse the repository at this point in the history
…splunk-connect-for-syslog into feature/checkpoint-noise
  • Loading branch information
rfaircloth-splunk committed May 8, 2020
2 parents caaf5dc + c7ce0cd commit 11f31d8
Show file tree
Hide file tree
Showing 16 changed files with 571 additions and 49 deletions.
2 changes: 2 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and variables needed to properly configure SC4S for your environment.
| SPLUNK_HEC_URL | url | URL(s) of the Splunk endpoint, can be a single URL space seperated list |
| SPLUNK_HEC_TOKEN | string | Splunk HTTP Event Collector Token |
| SC4S_GLOBAL_DNS_USE | yes or no(default) | use reverse DNS to identify hosts when HOST is not valid in the syslog header |
| SC4S_CONTAINER_HOST | string | variable passed to the container to identify the actual log host for container implementations |

* NOTE: Do _not_ configure HEC Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http
destination does not support this feature. Moreover, HEC Ack would significantly degrade performance for streaming data such as
Expand All @@ -26,6 +27,7 @@ syslog.
| SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE | path | Custom trusted cert file |
| SC4S_DEST_SPLUNK_HEC_TLS_VERIFY | yes(default) or no | verify HTTP(s) certificate |
| SC4S_DEST_SPLUNK_HEC_WORKERS | numeric | Number of destination workers (default: 10 threads). This should rarely need to be changed; consult sc4s community for advice on appropriate setting in extreme high- or low-volume environments. |
| SC4S_DEST_SPLUNK_INDEXED_FIELDS | facility,severity,container,loghost,dport,fromhostip,proto | list of sc4s indexed fields default list is all fields )

## Alternate Destination Configuration

Expand Down
1 change: 1 addition & 0 deletions docs/gettingstarted/docker-systemd-general.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ ExecStartPre=/usr/bin/docker run \
--name SC4S_preflight \
--rm $SC4S_IMAGE -s
ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 \
-e "SC4S_CONTAINER_HOST=$(hostname -s)" \
--env-file=/opt/sc4s/env_file \
"$SC4S_PERSIST_VOLUME" \
"$SC4S_LOCAL_CONFIG_MOUNT" \
Expand Down
1 change: 1 addition & 0 deletions docs/gettingstarted/podman-systemd-general.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ ExecStartPre=/usr/bin/podman run \
--name SC4S_preflight \
--rm $SC4S_IMAGE -s
ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 \
-e "SC4S_CONTAINER_HOST=$(hostname -s)" \
--env-file=/opt/sc4s/env_file \
"$SC4S_PERSIST_VOLUME" \
"$SC4S_LOCAL_CONFIG_MOUNT" \
Expand Down
37 changes: 27 additions & 10 deletions docs/sources/Symantec/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,27 @@

### Sourcetypes

| sourcetype | notes |
|----------------|---------------------------------------------------------------------------------------------------------|
| symantec:ep:syslog | Warning the syslog method of accepting EP logs has been reported to show high data loss and is not Supported by Splunk |

### Sourcetype and Index Configuration

| key | sourcetype | index | notes |
|----------------|----------------|----------------|----------------|
| symantec_ep | symantec:ep:syslog | epav | none |
| sourcetype | notes |
|--------------------------------|---------------------------------------------------------------------------------------------------------|
| symantec:ep:syslog | Warning the syslog method of accepting EP logs has been reported to show high data loss and is not Supported by Splunk |
| symantec:ep:admin:syslog | none |
| symantec:ep:agent:syslog | none |
| symantec:ep:agt:system:syslog | none |
| symantec:ep:behavior:syslog | none |
| symantec:ep:packet:syslog | none |
| symantec:ep:policy:syslog | none |
| symantec:ep:proactive:syslog | none |
| symantec:ep:risk:syslog | none |
| symantec:ep:scan:syslog | none |
| symantec:ep:scm:system:syslog | none |
| symantec:ep:security:syslog | none |
| symantec:ep:traffic:syslog | none |

### Index Configuration

| key | index | notes |
|----------------|----------------|----------------|
| symantec_ep | epav | none |


### Filter type
Expand All @@ -31,7 +43,12 @@ MSG Parse: This filter parses message content
* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source.
* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration
* Select TCP or SSL transport option
* Ensure the format of the event is customized per Splunk documentation
* Ensure the format of the event is customized as follows

```
<111>1 $(date)T$(x-bluecoat-hour-utc):$(x-bluecoat-minute-utc):$(x-bluecoat-second-utc).000z $(x-bluecoat-appliance-name) bluecoat - splunk_format - c-ip=$(c-ip) Content-Type=$(quot)$(rs(Content-Type))$(quot) cs-auth-group=$(cs-auth-group) cs-bytes=$(cs-bytes) cs-categories=$(quot)$(cs-categories)$(quot) cs-host=$(cs-host) cs-ip=$(cs-ip) cs-method=$(cs-method) cs-uri-extension=$(cs-uri-extension) cs-uri-path=$(cs-uri-path) cs-uri-port=$(cs-uri-port) cs-uri-query=$(quot)$(cs-uri-query)$(quot) cs-uri-scheme=$(cs-uri-scheme) cs-User-Agent=$(quot)$(cs(User-Agent))$(quot) cs-username=$(cs-username) dnslookup-time=$(dnslookup-time) duration=$(duration) rs-status=$(rs-status) rs-version=$(rs-version) rs_Content_Type=$(rs-Content-Type) s-action=$(s-action) s-ip=$(s-ip) service.name=$(service.name) service.group=$(service.group) s-supplier-ip=$(s-supplier-ip) s-supplier-name=$(s-supplier-name) sc-bytes=$(sc-bytes) sc-filter-result=$(sc-filter-result) sc-filter-result=$(sc-filter-result) sc-status=$(sc-status) time-taken=$(time-taken) x-bluecoat-appliance-name=$(x-bluecoat-appliance-name) x-bluecoat-appliance-primary-address=$(x-bluecoat-appliance-primary-address) x-bluecoat-application-name=$(x-bluecoat-application-name) x-bluecoat-application-operation=$(x-bluecoat-application-operation) x-bluecoat-proxy-primary-address=$(x-bluecoat-proxy-primary-address) x-bluecoat-transaction-uuid=$(x-bluecoat-transaction-uuid) x-exception-id=$(x-exception-id) x-virus-id=$(x-virus-id) c-url=$(quot)$(url)$(quot) cs-Referer=$(quot)$(cs(Referer))$(quot) c-cpu=$(c-cpu) c-uri-pathquery=$(c-uri-pathquery) connect-time=$(connect-time) cs-auth-groups=$(cs-auth-groups) cs-headerlength=$(cs-headerlength) cs-threat-risk=$(cs-threat-risk) r-ip=$(r-ip) r-supplier-ip=$(r-supplier-ip) rs-time-taken=$(rs-time-taken) rs-server=$(rs(server)) s-connect-type=$(s-connect-type) s-icap-status=$(s-icap-status) s-sitename=$(s-sitename) s-source-port=$(s-source-port) s-supplier-country=$(s-supplier-country) sc-Content-Encoding=$(sc(Content-Encoding)) sr-Accept-Encoding=$(sr(Accept-Encoding)) x-auth-credential-type=$(x-auth-credential-type) x-cookie-date=$(x-cookie-date) x-cs-certificate-subject=$(x-cs-certificate-subject) x-cs-connection-negotiated-cipher=$(x-cs-connection-negotiated-cipher) x-cs-connection-negotiated-cipher-size=$(x-cs-connection-negotiated-cipher-size) x-cs-connection-negotiated-ssl-version=$(x-cs-connection-negotiated-ssl-version) x-cs-ocsp-error=$(x-cs-ocsp-error) x-cs-Referer-uri=$(x-cs(Referer)-uri) x-cs-Referer-uri-address=$(x-cs(Referer)-uri-address) x-cs-Referer-uri-extension=$(x-cs(Referer)-uri-extension) x-cs-Referer-uri-host=$(x-cs(Referer)-uri-host) x-cs-Referer-uri-hostname=$(x-cs(Referer)-uri-hostname) x-cs-Referer-uri-path=$(x-cs(Referer)-uri-path) x-cs-Referer-uri-pathquery=$(x-cs(Referer)-uri-pathquery) x-cs-Referer-uri-port=$(x-cs(Referer)-uri-port) x-cs-Referer-uri-query=$(x-cs(Referer)-uri-query) x-cs-Referer-uri-scheme=$(x-cs(Referer)-uri-scheme) x-cs-Referer-uri-stem=$(x-cs(Referer)-uri-stem) x-exception-category=$(x-exception-category) x-exception-category-review-message=$(x-exception-category-review-message) x-exception-company-name=$(x-exception-company-name) x-exception-contact=$(x-exception-contact) x-exception-details=$(x-exception-details) x-exception-header=$(x-exception-header) x-exception-help=$(x-exception-help) x-exception-last-error=$(x-exception-last-error) x-exception-reason=$(x-exception-reason) x-exception-sourcefile=$(x-exception-sourcefile) x-exception-sourceline=$(x-exception-sourceline) x-exception-summary=$(x-exception-summary) x-icap-error-code=$(x-icap-error-code) x-rs-certificate-hostname=$(x-rs-certificate-hostname) x-rs-certificate-hostname-category=$(x-rs-certificate-hostname-category) x-rs-certificate-observed-errors=$(x-rs-certificate-observed-errors) x-rs-certificate-subject=$(x-rs-certificate-subject) x-rs-certificate-validate-status=$(x-rs-certificate-validate-status) x-rs-connection-negotiated-cipher=$(x-rs-connection-negotiated-cipher) x-rs-connection-negotiated-cipher-size=$(x-rs-connection-negotiated-cipher-size) x-rs-connection-negotiated-ssl-version=$(x-rs-connection-negotiated-ssl-version) x-rs-ocsp-error=$(x-rs-ocsp-error)
```

### Options

Expand Down
10 changes: 10 additions & 0 deletions docs/sources/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,13 @@ A key aspect of SC4S is to properly set Splunk metadata prior to the data arrivi

It is understood that default values will need to be changed in many installations. Each source documented in this section has a table entitled "Sourcetype and Index Configuration", which highlights the default index and sourcetype for each source. See the section "SC4S metadata configuration" in the "Configuration" page for more information on how to override the default values in this table.

## Unique listening ports

SC4S supports unique listening ports for each source technology/log path (e.g. Cisco ASA), which is useful when the device is
sending data on a port different from the typical default syslog port (UDP port 514). In some cases, when the source device emits data that
is not able to be distinguished from other device types, a unique port is sometimes required. The specific environment variables used for
setting "unique ports" are outlined in each source document in this section.

In most cases only one "unique port" is needed for each source. However, SC4S also supports multiple network listening ports per source,
which can be useful for a narrow set of compliance use cases. When configuring a source port variable to enable multiple ports, use a
comma-separated list with no spaces (e.g. `SC4S_LISTEN_CISCO_ASA_UDP_PORT=5005,6005`).
2 changes: 1 addition & 1 deletion docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ don't expect, check to see that the index is created in Splunk, or that a `lastC
cause for almost _all_ `400` errors.
* If you continue to the individual log entries in these directories, you will see entries of the form
```bash
curl -k -u "sc4s HEC debug:a778f63a-5dff-4e3c-a72c-a03183659e94" "https://splunk.smg.aws:8088/services/collector/event" -d '{"time":"1584556114.271","sourcetype":"sc4s:events","source":"SC4S:s_internal","index":"main","host":"e3563b0ea5d8","fields":{"sc4s_syslog_severity":"notice","sc4s_syslog_facility":"syslog","sc4s_log_host":"e3563b0ea5d8","sc4s_fromhostip":"127.0.0.1"},"event":"syslog-ng starting up; version='3.26.1'"}'
curl -k -u "sc4s HEC debug:a778f63a-5dff-4e3c-a72c-a03183659e94" "https://splunk.smg.aws:8088/services/collector/event" -d '{"time":"1584556114.271","sourcetype":"sc4s:events","source":"SC4S:s_internal","index":"main","host":"e3563b0ea5d8","fields":{"sc4s_syslog_severity":"notice","sc4s_syslog_facility":"syslog","sc4s_loghost":"e3563b0ea5d8","sc4s_fromhostip":"127.0.0.1"},"event":"syslog-ng starting up; version='3.26.1'"}'
```
* These commands, with minimal modifications (e.g. multiple URLs specified or elements that needs shell escapes) can be run directly on the
command line to determine what, exactly, the HEC endpoint is returning. This can be used to refine th index or other parameter to correct the
Expand Down
23 changes: 0 additions & 23 deletions package/etc/conf.d/conflib/_splunk/splunkfields.conf

This file was deleted.

42 changes: 42 additions & 0 deletions package/etc/conf.d/conflib/_splunk/splunkfields.conf.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#Used to set indexed fields we will always use to global defaults
rewrite r_set_splunk_default {
set("SC4S:$SOURCE", value(".splunk.source"));
{{- if (split (getenv "SC4S_DEST_SPLUNK_INDEXED_FIELDS" "facility,severity,container,loghost,dport,fromhostip,proto") ",") has "facility" }}
set($FACILITY, value("fields.sc4s_syslog_facility"));
{{- end}}
{{- if (split (getenv "SC4S_DEST_SPLUNK_INDEXED_FIELDS" "facility,severity,container,loghost,dport,fromhostip,proto") ",") has "severity" }}
set($LEVEL, value("fields.sc4s_syslog_severity"));
{{- end}}
{{- if (split (getenv "SC4S_DEST_SPLUNK_INDEXED_FIELDS" "facility,severity,container,loghost,dport,fromhostip,proto") ",") has "loghost" }}
{{- if (getenv "SC4S_CONTAINER_HOST") }}
set("{{ getenv "SC4S_CONTAINER_HOST" }}", value("fields.sc4s_loghost"));
{{- end}}
{{- end}}
{{- if (split (getenv "SC4S_DEST_SPLUNK_INDEXED_FIELDS" "facility,severity,container,loghost,dport,fromhostip,proto") ",") has "container" }}
set($LOGHOST, value("fields.sc4s_container"));
{{- end}}
{{- if (split (getenv "SC4S_DEST_SPLUNK_INDEXED_FIELDS" "facility,severity,container,loghost,dport,fromhostip,proto") ",") has "fromhostip" }}
set($SOURCEIP, value("fields.sc4s_fromhostip"));
{{- end}}
{{- if (split (getenv "SC4S_DEST_SPLUNK_INDEXED_FIELDS" "facility,severity,container,loghost,destport,fromhostip,proto") ",") has "destport" }}
set($DESTPORT, value("fields.sc4s_destport"));
{{- end}}
{{- if (split (getenv "SC4S_DEST_SPLUNK_INDEXED_FIELDS" "facility,severity,container,loghost,destport,fromhostip,proto") ",") has "proto" }}
set($PROTO, value("fields.sc4s_proto"));
{{- end}}
};
#used by each log-path to set index and sourcetype which may be
#overridden by user defined values
block rewrite r_set_splunk_dest_default(
index()
source("${.splunk.source}")
sourcetype()
template(`splunk-template`)
) {
set("`index`", value(".splunk.index"));
set("`source`", value(".splunk.source"));
set("`sourcetype`", value(".splunk.sourcetype"));
};



48 changes: 48 additions & 0 deletions package/etc/conf.d/filters/symantec/ep.conf
Original file line number Diff line number Diff line change
@@ -1,3 +1,51 @@
filter f_symantec_ep {
program("SymantecServer")
};

filter f_symantec_ep_proactive {
message(',Detection\stype:')
};

filter f_symantec_ep_risk {
message(',Risk\sname:')
};

filter f_symantec_ep_agt_system {
message(',Category:\s\d+,')
};

filter f_symantec_ep_packet {
message(',(?:Inbound|Outbound|Unknown),Application:')
};

filter f_symantec_ep_traffic {
message(',(?:Inbound|Outbound|Unknown),Begin(?:\sTime)?:')
};

filter f_symantec_ep_security {
message('CIDS\sSignature\sSubID:')
};

filter f_symantec_ep_scan {
message('Scan\sID:\s\d+')
};

filter f_symantec_ep_behavior {
message('Begin(?:\sTime)?:\s[^,]*,End(?:\sTime)?:')
};

filter f_symantec_ep_policy {
message('Admin:\s[^,]+,.*[Pp]olicy')
};

filter f_symantec_ep_admin {
message('Domain(?:\sName)?:\s[^,]{0,25},Admin:')
};

filter f_symantec_ep_agent {
message('(?:,The\smanagement\sserver|,The\sclient)')
};

filter f_symantec_ep_scm_system {
message('Site:\s[^,]+,Server(?:\sName)?:\s[^,]+,')
};
73 changes: 68 additions & 5 deletions package/etc/conf.d/log_paths/lp-symantec_ep.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,78 @@ log {
};
};


if {
filter(f_symantec_ep_proactive);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:proactive:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_risk);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:risk:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_agt_system);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:agt:system:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_packet);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:packet:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_traffic);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:traffic:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_security);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:security:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_scan);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:scan:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_behavior);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:behavior:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_policy);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:policy:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_admin);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:admin:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_agent);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:agent:syslog"), index("epav"))
};
} elif {
filter(f_symantec_ep_scm_system);
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:scm:system:syslog"), index("epav"))
};
} else {
rewrite {
r_set_splunk_dest_default(sourcetype("symantec:ep:syslog"), index("epav"))
};
};
rewrite {
set("symantec_ep_syslog", value("fields.sc4s_vendor_product"));
r_set_splunk_dest_default(sourcetype("symantec:ep:syslog"), index("epav"))
set("Symantec Endpoint Protection", value("fields.sc4s_vendor_product"));
};
parser { p_add_context_splunk(key("symantec_ep_syslog")); };
parser { p_add_context_splunk(key("symantec_ep")); };

parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); };

{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_SYMANTEC_EP_HEC" "no")) }}
destination(d_hec);
Expand Down
5 changes: 4 additions & 1 deletion package/etc/conf.d/log_paths/lp-symantec_proxy.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ log {
channel {
# Listen on the default port (typically 514) for SYMANTEC_PROXY traffic
source (s_DEFAULT);
filter(f_is_rfc5424_noversion);
filter(f_symantec_bluecoat_proxy);
flags(final);
};
Expand All @@ -24,6 +23,10 @@ log {
rewrite {
set("bluecoat_proxy", value("fields.sc4s_vendor_product"));
r_set_splunk_dest_default(sourcetype("bluecoat:proxysg:access:kv"), index("netproxy"))
subst(
"([-_a-zA-Z\(\)]+=(\"-\"|-| ))",
"", value(MESSAGE)
);
};

parser {p_add_context_splunk(key("bluecoat_proxy")); };
Expand Down
2 changes: 1 addition & 1 deletion package/etc/context_templates/splunk_index.csv.example
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@
#sc4s_events,index,main
#sc4s_fallback,index,main
#sc4s_metrics,index,em_metrics
#symanrtec_ep,index,epav
#symantec_ep,index,epav
#vmware_nsx,index,main
#zscaler_alerts,index,main
#zscaler_dns,index,netdns
Expand Down
Loading

0 comments on commit 11f31d8

Please sign in to comment.