Skip to content

Commit

Permalink
Merge pull request #260 from splunk/release/1.5.0
Browse files Browse the repository at this point in the history
Release/1.5.0
  • Loading branch information
Ryan Faircloth authored and GitHub committed Jan 3, 2020
2 parents 980e2d4 + 277b928 commit 47a592d
Show file tree
Hide file tree
Showing 44 changed files with 393 additions and 646 deletions.
96 changes: 57 additions & 39 deletions docs/configuration.md

Large diffs are not rendered by default.

15 changes: 8 additions & 7 deletions docs/performance.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Performance
# Performance and Sizing
Performance testing against our lab configuration produces the following results and limitations.

## Tested Configuration
Expand Down Expand Up @@ -34,9 +34,10 @@ and forget protocol making it sensitive to performance. Given this it is highly
performance with your hardware and production data samples. The syslog-ng loggen tool available in the SC4S container
and the commands above can be utilized in this effort.

Deployment Size | Hardware Spec | Average EPS with average msg size 800 bytes
-- | -- | --
Small | 2 X 3.1 ghz cores1 GB of memory | 2K msg/sec
Medium | 4 X 3.1 ghz cores2 GB of memory | 4.5K msg/sec
Large | 8 X 3.1 ghz cores4 GB of memory | 9K msg/sec
XL | 16 X 3.1 ghz cores8 GB of memory | 18K msg/sec
| Deployment Size | Hardware Spec | Average EPS<br>800-byte msg |
| -------- | --------- | -------- |
| Small | 4 X 3.1 ghz<br> 1 GB memory | 3.5K msg/sec |
| Medium | 8 X 3.1 ghz<br> 2 GB memory | 7K msg/sec |
| Large | 12 X 3.1 ghz<br> 4 GB memory | 10K msg/sec |
| XL | 16 X 3.1 ghz<br> 8 GB memory | 14K msg/sec |

16 changes: 15 additions & 1 deletion docs/sources/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,21 @@
# Introduction
When using Splunk Connect for Syslog to onboard a data source, the SC4S filter performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source.
When using Splunk Connect for Syslog to onboard a data source, the SC4S filter (or "log path") performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source.

SC4S "unique" filters are based either on the port upon which events arrive or the hostname/CIDR block from which they are sent. The "soup" filters run for events that arrive on port 514 (default for syslog), and contain regex and other syslog-specific parsers to identify events from a specific source, apply the correct sourcetype, and set other metadata. Data sources which generate events that are not unique enough to accurately identify with soup filters _must_ employ the "unique" filters (port/hostname/CIDR block) instead -- the soup filters are unavailable for these sources.

## The SC4S "fallback" sourcetype

If SC4S receives an event on port 514 which has no soup filter, that event will be given a "fallback" sourcetype. If you see events in Splunk with the fallback sourcetype, then you should figure out what source the events are from and determine why these events are not being sourcetyped correctly. The most common reason for events categorized as "fallback" is the lack of a SC4S filter for that source, and in some cases a misconfigured relay which alters the integrity of the message format. In most cases this means a new SC4S filter must be developed. In this situation you can either build a filter or file an issue with the community to request help.

The "fallback" sourcetype is formatted in JSON to allow the administrator to see the constituent syslog-ng "macros" (fields) that have been autmaticially parsed by the syslog-ng server An RFC3164 (legacy BSD syslog) "on the wire" raw message is usually (but unfortunately not always) comprised of the following syslog-ng macros, in this order and spacing:
```
<$PRI> $HOST $LEGACY_MSGHDR$MESSAGE
```
These fields can be very useful in building a new filter for that sourcetype. In addition, the indexed field `sc4s_syslog_format` is helpful in determining if the incoming message is standard RFC3164. A value of anything other than `rfc3164` or `rfc5424_strict` indicates a vendor purturbation of standard syslog, which will warrant more careful examination when building a filter.

## Splunk Connect for Syslog and Splunk metadata

A key aspect of SC4S is to properly set Splunk metadata prior to the data arriving in Splunk (and before any TA processing takes place. The filters will apply the proper index, source, sourcetype, host, and timestamp metadata automatically by individual data source. Proper values for this metadata (including a recommended index) are included with all "out-of-the-box" log paths included with SC4S and are chosen to properly interface with the corresponding TA in Splunk. The administrator will need to ensure all recommneded indexes be created to accept this data if the defaults are not changed.

It is understood that default values will need to be changed in many installations. Each source documented in this section has a table entitled "Sourcetype and Index Configuration", which highlights the default index and sourcetype for each source. See the section "SC4S metadata configuration" in the "Configuration" page for more information on how to override the default values in this table.

2 changes: 1 addition & 1 deletion package/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ RUN dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noar
dnf config-manager --set-enabled PowerTools -y; \
dnf update -y

RUN dnf group install "Development Tools" ;\
RUN dnf group install "Development Tools" -y ;\
dnf install findutils autoconf \
autoconf automake ca-certificates git libtool pkgconfig bison byacc file \
flex pcre-devel glib2-devel openssl-devel libcurl-devel \
Expand Down
14 changes: 7 additions & 7 deletions package/etc/conf.d/conflib/_common/syslog_format.conf
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ filter f_rfc5424_noversion{
filter f_rfc5424_epochtime{
message('^(?<SYSLOGMSG>(?<HEADER>(?<PRI><\d{1,3}>)(?<VERSION>[1-9][0-9]?) (?<TIMESTAMP>(?<EPOCH>\d{10})(?:.(?<TIMESECFRAC>\d{1,9})?)) (?<FROMHOST>[^ ]+) ))');
};
filter f_rfc5424_bsdtime{
filter f_rfc3164_version{
message('^(?<SYSLOGMSG>(?<HEADER>(?<PRI><\d{1,3}>)(?<VERSION>[1-9][0-9]?) (?<TIMESTAMP>[A-Za-z]{3} \d\d \d\d:\d\d:\d\d) (?<FROMHOST>[^ ]+) ))');
};
rewrite set_rfcnonconformant{
Expand All @@ -31,11 +31,11 @@ rewrite set_rfc5424_epochtime{
filter f_is_rfc5424_epochtime{
match("rfc5424_epochtime" value("fields.sc4s_syslog_format"))
};
rewrite set_rfc5424_bsdtime{
set("rfc5424_bsdtime" value("fields.sc4s_syslog_format"));
rewrite set_rfc3164_version{
set("rfc3164_version" value("fields.sc4s_syslog_format"));
};
filter f_is_rfc5424_bsdtime{
match("rfc5424_bsdtime" value("fields.sc4s_syslog_format"))
filter f_is_rfc3164_version{
match("rfc3164_version" value("fields.sc4s_syslog_format"))
};
rewrite set_rfc3164{
set("rfc3164" value("fields.sc4s_syslog_format"));
Expand All @@ -56,6 +56,6 @@ filter f_is_no_parse{
match("no_parse" value("fields.sc4s_syslog_format"))
};

rewrite set_rfc3894_bsdtime{
subst('^<\d+>(\d)', "", value("MESSAGE"));
rewrite set_rfc3164_no_version_string{
subst('(^<\d+>)\d', $1, value("MESSAGE"));
};
34 changes: 16 additions & 18 deletions package/etc/conf.d/conflib/_common/templates.conf
Original file line number Diff line number Diff line change
Expand Up @@ -57,27 +57,25 @@ template t_hdr_sdata_msg {
# JSON; for JSON pretty-printing (for debugging)
# ===============================================================================================

template t_JSON {
template("$(format-json --scope all-nv-pairs
--exclude fields.*
--exclude .splunk.*
--exclude HOST
--exclude HOST_FROM
--exclude SOURCE
)");
};
template t_JSON_3164 {
template('$(format-json --scope rfc3164
--pair PRI="<$PRI>"
--key LEGACY_MSGHDR
--exclude FACILITY
--exclude PRIORITY
)');
};

# ===============================================================================================
# JSON; for JSON pretty-printing (for RFC5424)
# ===============================================================================================

template t_JSON_5424 {
template("$(format-json --scope all-nv-pairs
--exclude fields.*
--exclude .splunk.*
--exclude HOST
--exclude HOST_FROM
--exclude RAWMSG
--exclude SOURCE
)");
};
template('$(format-json --scope rfc5424
--pair PRI="<$PRI>"
--key ISODATE
--exclude DATE
--exclude FACILITY
--exclude PRIORITY
)');
};
3 changes: 2 additions & 1 deletion package/etc/conf.d/filters/checkpoint/splunk.conf
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
filter f_checkpoint_splunk {
match('\|(?:origin_sic_name|originsicname)\=[cC][nN]|\|product\=SmartConsole\|' value("RAWMSG") type("pcre"));
match('\|(?:origin_sic_name|originsicname)\=[cC][nN]|\|product\=SmartConsole\|' value("MSG") type("pcre")) or
match('\|(?:origin_sic_name|originsicname)\=[cC][nN]|\|product\=SmartConsole\|' value("LEGACY_MSGHDR") type("pcre"));
};

filter f_checkpoint_splunk_alerts {
Expand Down
2 changes: 1 addition & 1 deletion package/etc/conf.d/filters/paloalto/panos.conf
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
filter f_is_palalto_format{
match("rfc5424_bsdtime" value("fields.sc4s_syslog_format"))
match("rfc3164_version" value("fields.sc4s_syslog_format"))
or
match("rfc3164" value("fields.sc4s_syslog_format"))
};
Expand Down
12 changes: 7 additions & 5 deletions package/etc/conf.d/log_paths/internal.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,25 @@ log {
);
};

{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_INTERNAL_METRICS_HEC" "no") | conv.ToBool) }}
{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_INTERNAL_METRICS_HEC" "no")) }}
destination(d_hecmetrics);
{{- end}}

} else {

{{- if eq (getenv "SC4S_DEBUG_STDOUT" "yes") "yes"}}
destination(d_stdout);
{{- end}}
rewrite { r_set_splunk_dest_default(sourcetype("sc4s:events"), index("main"))};
parser {p_add_context_splunk(key("sc4s_events")); };

{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_INTERNAL_EVENTS_HEC" "no") | conv.ToBool) }}
{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_INTERNAL_EVENTS_HEC" "no")) }}
destination(d_hec_internal);
{{- end}}

{{- if eq (getenv "SC4S_DEBUG_STDOUT" "yes") "yes"}}
destination(d_stdout);
{{- end}}

};
flags(flow-control,final);
};
{{- end}}
{{- tmpl.Exec "log_path" "yes" }}
22 changes: 5 additions & 17 deletions package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# Checkpoint
# Generate the custom port if defined
{{ $context := dict "port_id" "CHECKPOINT_SPLUNK" "parser" "common" }}
{{ tmpl.Exec "t/source_network.t" $context }}
Expand All @@ -22,12 +23,9 @@ log {
set("${PROGRAM}", value(".PROGRAM"));
subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM"));
};
rewrite {
r_set_splunk_dest_default(sourcetype("nix:syslog"), index("netops"), source("program:${.PROGRAM}") )
};
rewrite { r_set_splunk_dest_default(sourcetype("nix:syslog"), index("netops"), source("program:${.PROGRAM}")) };
parser { p_add_context_splunk(key("checkpoint_os")); };


} else {
parser {
kv-parser(prefix(".kv.") pair-separator("|") template("${MSGHDR} ${MSG}"));
Expand Down Expand Up @@ -74,23 +72,13 @@ log {
};

parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); };

#We want to unset the fields we won't need, as this is copied into the
#disk queue for network destinations. This can be very disk expensive
#if we don't
rewrite {
set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG"));
unset(value("RAWMSG"));
unset(value("PROGRAM"));
unset(value("LEGACY_MSGHDR"));
groupunset(values(".kv.*"));
};
{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CHECKPOINT_SPLUNK_HEC" "no") | conv.ToBool) }}
{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CHECKPOINT_SPLUNK_HEC" "no")) }}
destination(d_hec);
{{- end}}


{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CHECKPOINT_SPLUNK") }}
{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CHECKPOINT_SPLUNK" "no")) }}
destination(d_archive);
{{- end}}

Expand Down
24 changes: 5 additions & 19 deletions package/etc/conf.d/log_paths/p_rfc3164-cisco_acs.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -68,29 +68,15 @@ log {

parser {p_add_context_splunk(key("cisco_acs")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };

#We want to unset the fields we won't need, as this is copied into the
#disk queue for network destinations. This can be very disk expensive
#if we don't
rewrite {
set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG"));
unset(value("RAWMSG"));
unset(value("PROGRAM"));
unset(value("PID"));
unset(value("LEGACY_MSGHDR"));
unset(value("EPOCH"));
unset(value("VERSION"));
unset(value("TIMESECFRAC"));
groupunset(values("ACS.*"));
};

{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ACS_HEC" "no") | conv.ToBool) }}
destination(d_hec);
{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_ACS_HEC" "no")) }}
destination(d_hec);
{{- end}}

{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ACS") }}
{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_ACS" "no")) }}
destination(d_archive);
{{- end}}
{{- end}}

flags(flow-control,final);
};
Expand Down
16 changes: 3 additions & 13 deletions package/etc/conf.d/log_paths/p_rfc3164-cisco_asa.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,14 @@ log {
r_set_splunk_dest_default(sourcetype("cisco:asa"), index("netfw"))
};
parser {p_add_context_splunk(key("cisco_asa")); };

parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };

#We want to unset the fields we won't need, as this is copied into the
#disk queue for network destinations. This can be very disk expensive
#if we don't
rewrite {
set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG"));
unset(value("RAWMSG"));
unset(value("PROGRAM"));
unset(value("LEGACY_MSGHDR"));
};

{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ASA_HEC" "no") | conv.ToBool) }}
{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_ASA_HEC" "no")) }}
destination(d_hec);
{{- end}}

{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ASA_LEGACY") }}
{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_ASA_LEGACY" "no")) }}
destination(d_archive);
{{- end}}

Expand Down
21 changes: 4 additions & 17 deletions package/etc/conf.d/log_paths/p_rfc3164-cisco_ios.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -18,28 +18,15 @@ log {
guess-time-zone();
r_set_splunk_dest_default(sourcetype("cisco:ios"), index("netops"))
};
parser {
p_add_context_splunk(key("cisco_ios"));
};

parser { p_add_context_splunk(key("cisco_ios")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };

#We want to unset the fields we won't need, as this is copied into the
#disk queue for network destinations. This can be very disk expensive
#if we don't
rewrite {
set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG"));
unset(value("RAWMSG"));
unset(value("PROGRAM"));
unset(value("LEGACY_MSGHDR"));
groupunset(values(".cisco.*"));
};

{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_IOS_HEC" "no") | conv.ToBool) }}
{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_IOS_HEC" "no")) }}
destination(d_hec);
{{- end}}

{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_IOS") }}
{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_IOS" "no")) }}
destination(d_archive);
{{- end}}

Expand Down
24 changes: 5 additions & 19 deletions package/etc/conf.d/log_paths/p_rfc3164-cisco_ise.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -68,29 +68,15 @@ log {

parser {p_add_context_splunk(key("cisco_ise")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); };

#We want to unset the fields we won't need, as this is copied into the
#disk queue for network destinations. This can be very disk expensive
#if we don't
rewrite {
set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG"));
unset(value("RAWMSG"));
unset(value("PROGRAM"));
unset(value("PID"));
unset(value("LEGACY_MSGHDR"));
unset(value("EPOCH"));
unset(value("VERSION"));
unset(value("TIMESECFRAC"));
groupunset(values("ISE.*"));
};

{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ISE_HEC" "no") | conv.ToBool) }}
destination(d_hec);
{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_ISE_HEC" "no")) }}
destination(d_hec);
{{- end}}

{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ISE") }}
{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_ISE" "no")) }}
destination(d_archive);
{{- end}}
{{- end}}

flags(flow-control,final);
};
Expand Down
20 changes: 4 additions & 16 deletions package/etc/conf.d/log_paths/p_rfc3164-cisco_nxos.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -19,27 +19,15 @@ log {
r_set_splunk_dest_default(sourcetype("cisco:ios"), index("netops"), template("t_hdr_msg"))
};

parser {
p_add_context_splunk(key("cisco_nx_os"));
};
parser { p_add_context_splunk(key("cisco_nx_os")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); };

#We want to unset the fields we won't need, as this is copied into the
#disk queue for network destinations. This can be very disk expensive
#if we don't
rewrite {
set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG"));
unset(value("RAWMSG"));
unset(value("PROGRAM"));
unset(value("LEGACY_MSGHDR"));
};

{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_NXOS_HEC" "no") | conv.ToBool) }}
{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_NXOS_HEC" "no")) }}
destination(d_hec);
{{- end}}


{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_NXOS") }}
{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_NXOS" "no")) }}
destination(d_archive);
{{- end}}

Expand Down
Loading

0 comments on commit 47a592d

Please sign in to comment.