diff --git a/docs/configuration.md b/docs/configuration.md index f7ab084..f19488b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -30,10 +30,11 @@ will be used for each. This is why you see the `DEST_SPLUNK_HEC` in the variabl For this reason, normal disk buffering is recommended. * NOTE: If you add destinations locally in your configuration, pay attention to the _cumulative_ buffer requirements when allocating local disk. +* Be sure to factor in the syslog-ng overhead (approx. 1.7x) when calculating the total buffer size needed. See the "Data Resilience" section below for more information. * NOTE: The values for the variables below represent the _total_ sizes of the buffers for the destination. These sizes are divded by the number of workers (threads) when setting the actual syslog-ng buffer options, because the buffer options apply to each worker rather than the entire destination. Pay careful attention to this when using the "BYOE" version of SC4S, where direct access to the syslog-ng config files -may hide this nuance. +may hide this nuance. To determine the proper size of the disk buffer, consult the "Data Resilience" section below. | Variable | Values/Default | Description | |----------|---------------|-------------| @@ -48,12 +49,12 @@ may hide this nuance. This feature is designed to support "compliance" archival of all messages. Instructions for enabling this feature are included in each "getting started" runtime document. The files will be stored in a folder structure using the naming pattern ``${YEAR}/${MONTH}/${DAY}/${fields.sc4s_vendor_product}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}.log"``. -This pattern will create one file per "vendor_product" per minute with records formatted using syslog-ng's EWMM template. +This pattern will create one file per minute for each "vendor_product", with records formatted using syslog-ng's EWMM template. **WARNING POTENTIAL OUTAGE CAUSING CONSEQUENCE** SC4S does not prune the files that are created. The administrator must provide a means of log rotation to prune files -and/or move them to an archival system to avoid disk space failures. +and/or move them to an archival system to avoid exhaustion of disk space. | Variable | Values | Description | |----------|---------------|-------------| @@ -86,20 +87,13 @@ and/or move them to an archival system to avoid disk space failures. SC4S_SOURCE_TLS_ENABLE=yes ``` -## Log Path overrides of index or metadata +## SC4S metadata configuration -In some cases it is appropriate to override the default SC4S index or other Splunk metadata (such as an -source, host, or sourcetype) for a given data source. This is accomplished by the use of a lookup file that identifies these -source exceptions based on the log path used by the incoming message. These log path overrides are documented in the associated -"sources" document. For each data source, you will see a table of the form +### Log Path overrides of index or metadata -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| foo_bar | fb_log | netfw | none | +A key aspect of SC4S is to properly set Splunk metadata prior to the data arriving in Splunk (and before any TA processing takes place). The filters will apply the proper index, source, sourcetype, host, and timestamp metadata automatically by individual data source. Proper values for this metadata, including a recommended index and output format (template), are included with all "out-of-the-box" log paths included with SC4S and are chosen to properly interface with the corresponding TA in Splunk. The administrator will need to ensure all recommneded indexes be created to accept this data if the defaults are not changed. -In this case the key, `foo_bar`, will be an entry into the `splunk_indexes.csv` file that is populated in `/opt/sc4s/local/context` when SC4S -is run for the first time. The other columns show the default sourcetype and index when not overriden. This file contruct -is best shown with an example. Here is the table for Juniper Netscreen devices, from the "sources" document: +It is understood that default values will need to be changed in many installations. To accomodate this, each filter consults a lookup file that is mounted to the container (by default `/opt/sc4s/local/context/splunk_index.csv`) and is populated with defaults on the first run of SC4S after being set up according to the "getting started" runtime documents. This is a CSV file containing a "key" that is referenced in the log path for each data source. These keys are documented in the individual source files in this section, and allow one to override Splunk metadata either in whole or part. The use of this file is best shown by example. Here is the "Sourcetype and Index Configuration" table from the Juniper Netscreen source documentation page in this section: | key | sourcetype | index | notes | |------------------------|---------------------|----------------|---------------| @@ -108,39 +102,40 @@ is best shown with an example. Here is the table for Juniper Netscreen devices, Here is a snippet from the `splunk_indexes.csv` file: -* Get the filter and lookup files ```bash -cd /opt/sc4s/default -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context_templates/compliance_meta_by_source.conf -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context_templates/compliance_meta_by_source.csv -``` #juniper_sslvpn,index,netfw juniper_netscreen,index,ns_index #juniper_nsm,index,netfw + ``` -The columns in this file are `key`, `metadata`, and `value`. The `key` entries are -by default "commmented out", which is really a half-truth because CSV files don't allow comments. Therefore, to ensure there +The columns in this file are `key`, `metadata`, and `value`. By default, the keys in this file are "commented out", but in reality CSV files +cannot have comments so the `#` simply causes a mismatch to the key reference, effectively "commenting" it out. Therefore, to ensure there is a match from the log path that references this file, be sure to remove the leading `#`. Once this is done, the following changes can be -made by adding one or more rows to the table and specifying one or more of the following `metadata`/`value` pairs for a given `key`: +made by modifying and/or adding rows in the table and specifying one or more of the following `metadata`/`value` pairs for a given `key`: * `index` to specify an alternate `value` for index * `source` to specify an alternate `value` for source - * `sourcetype` to specify an alternate `value` for sourcetype (be _very_ careful when changing this; only do so if a downstream + * `host` to specify an alternate `value` for host + * `sourcetype` to specify an alternate `value` for sourcetype (be _very_ careful when changing this; only do so if an upstream TA is _not_ being used, or a custom TA (built by you) is being used.) + * `sc4s_template` to specify an alternate `value` for the syslog-ng template that will be used to format the event that will be + indexed by Splunk. Changing this carries the same warning as the sourcetype above; this will affect the upstream TA. The template + choices are documented elsewhere in this "Configuration" section. In this case, the `juniper_netscreen` key is "uncommented" (thereby enabling it), and the new index used for that data source will be `ns_index`. -In general, for most deployments the index should be the only change needed; the defaults for the others should almost -never be overridden (particularly for "Out of the Box" data sources). Even then, care should be taken when considering an alternate -index, as the defaults for SC4S were chosen with best practices in mind. +In general, for most deployments the index should be the only change needed; other default metadata should almost +never be overridden (particularly for the "Out of the Box" data sources). Even then, care should be taken when considering any alternates, +as the defaults for SC4S were chosen with best practices in mind. -This `csv` file can also be appended when building custom SC4S log paths (filters). Care should be taken during filter design to choose -appropriate index and sourctype defaults, so that admins are not compelled to override them. +The `splunk_indexes.csv` file should also be appended to (with a "commented out" default for the index) when building custom SC4S log paths +(filters). Care should be taken during filter design to choose appropriate index, sourctype and template defaults, so that admins are not +compelled to override them. -## Override index or metadata based on host, ip, or subnet +### Override index or metadata based on host, ip, or subnet (compliance overrides) In other cases it is appropriate to provide the same overrides but based on PCI scope, geography, or other criterion rather than globally. This is accomplished by the use of a file that uniquely identifies these source exceptions via syslog-ng filters, @@ -148,7 +143,8 @@ which maps to an associated lookup of alternate indexes, sources, or other metad added to futher classify the data. * The `conf` and `csv` files referenced below will be populated into the `/opt/sc4s/local/context` directory when SC4S is run for the first -time after being set up according to the "getting started" runtime documents. +time after being set up according to the "getting started" runtime documents, in a similar fashion to `splunk_indexes.csv`. After this first-time population of the files takes place, they can be edited (and SC4S restarted) for the changes to take effect. To get started: + * Edit the file ``compliance_meta_by_source.conf`` to supply uniquely named filters to identify events subject to override. * Edit the file ``compliance_meta_by_source.csv`` to supply appropriate field(s) and values. @@ -164,7 +160,6 @@ corresonding `filter name` rows in the `csv` file. The `field name` column obey This file construct is best shown by an example. Here is a sample ``compliance_meta_by_source.conf`` file: ``` -@version: 3.24 filter f_test_test { host("something-*" type(glob)) or netmask(192.168.100.1/24) @@ -177,7 +172,7 @@ f_test_test,.splunk.index,"pciindex" f_test_test,fields.compliance,"pci" ``` -First off, ensure that the proper version string exists at the top of the `conf` file, and that the filter name(s) in the `conf` file match +First off, ensure that the filter name(s) in the `conf` file match one or more rows in the `csv` file. In this case, any incoming message with a hostname starting with `something-` or arriving from a netmask of `192.168.100.1/24` will match the `f_test_test` filter, and the corresponding entries in the `csv` file will be checked for overrides. In this case, the new index is `pciindex`, and an indexed field named `compliance` will be sent to Splunk, with it's value set to `pci`. @@ -200,7 +195,23 @@ For the Docker Swarm runtime, redeploy the updated service using the command: docker stack deploy --compose-file docker-compose.yml sc4s ``` -## Data Durability - Local Disk Buffer Configuration +## Splunk Connect for Syslog output templates (syslog-ng templates) + +Splunk Connect for Syslog utilizes the syslog-ng template mechanism to format the output payload (event) that will be sent to Splunk. These templates can format the messages in a number of ways (straight text, JSON, etc.) as well as utilize the many syslog-ng "macros" (fields) to specify what gets placed in the payload that is delivered to the destination. Here is a list of the templates used in SC4S, which can be used in the metadata override section immediately above. New templates can also be added by the administrator in the "local" section for local destinations; pay careful attention to the syntax as the templates are "live" syslog-ng config code. + +| Template name | Template contents | Notes | +|------------------|------------------------------------------|------------------------------------------------------------------| +| t_standard | ${DATE} ${HOST} ${MSGHDR}${MESSAGE} | Standard template for most RFC3164 (standard syslog) traffic | +| t_msg_only | ${MSGONLY} | syslog-ng $MSG is sent, no headers (host, timestamp, etc.) | +| t_msg_trim | $(strip $MSGONLY) | As above with whitespace stripped | +| t_everything | ${ISODATE} ${HOST} ${MSGHDR}${MESSAGE} | Standard template with ISO date format | +| t_hdr_msg | ${MSGHDR}${MESSAGE} | Useful for non-compliant syslog messages | +| t_legacy_hdr_msg | ${LEGACY_MSGHDR}${MESSAGE} | Useful for non-compliant syslog messages | +| t_hdr_sdata_msg | ${MSGHDR}${MSGID} ${SDATA} ${MESSAGE} | Text-based representation of RFC5424-compliant syslog messages | +| t_JSON_3164 | $(format-json --scope rfc3164
--pair PRI="<$PRI>"
--key LEGACY_MSGHDR
--exclude FACILITY
--exclude PRIORITY) | JSON output of all RFC3164-based syslog-ng macros. Useful with the "fallback" sourcetype to aid in new filter development. | +| t_JSON_5424 | $(format-json --scope rfc5424
--pair PRI="<$PRI>"
--key ISODATE
--exclude DATE
--exclude FACILITY
--exclude PRIORITY) | JSON output of all RFC5424-based syslog-ng macros; for use with RFC5424-compliant traffic. | + +## Data Resilience - Local Disk Buffer Configuration SC4S provides capability to minimize the number of lost events if the connection to all the Splunk Indexers goes down. This capability utilizes the disk buffering feature of Syslog-ng. SC4S receives a response from the Splunk HTTP Event Collector (HEC) when a message is received successfully. If a confirmation message from the HEC endpoint is not received (or a “server busy” reply, such as a “503” is sent), the load balancer will try the next HEC endpoint in the pool. If all pool members are exhausted (such as would occur if there were a full network outage to the HEC endpoints), events will queue to the local disk buffer on the SC4S Linux host. SC4S will continue attempting to send the failed events while it buffers all new incoming events to disk. If the disk space allocated to disk buffering fills up then SC4S will stop accepting new events and subsequent events will be lost. Once SC4S gets confirmation that events are again being received by one or more indexers, events will then stream from the buffer using FIFO queueing. The number of events in the disk buffer will reduce as long as the incoming event volume is less than the maximum SC4S (with the disk buffer in the path) can handle. When all events have been emptied from the disk buffer, SC4S will resume streaming events directly to Splunk. @@ -210,15 +221,22 @@ SC4S has disk buffering enabled by default and it is strongly recommended that y Without disk buffering enabled SC4S can handle up to 345K EPS (800 bytes/event avg) With “Normal” disk buffering enabled SC4S can handle up to 60K EPS (800 bytes/event avg) -- This is still a lot of data! -To guard against data loss it is important to configure the appropriate type and amount of storage for SC4S disk buffering. To estimate the storage allocation its best to start with your estimated maximum events per second that each SC4S server will experience. Based on the maximum throughput of SC4S with disk buffering enabled, the conservative estimate for maximum events per second is 60K (however, you should use the maximum rate in your environment for this calculation, not the max rate SC4S can handle). Next is your average estimated event size based on your data sources. It is common industry practice to estimate log events as 800 bytes on average. And the final input to the sizing estimation would be the maximum length of connectivity downtime you want disk buffering to be able to handle. This measure is very much dependent on your risk tolerance. For example, to protect against a full day of lost connectivity from SC4S to all your indexers at maximum throughput the calculation would look like the following... +To guard against data loss it is important to configure the appropriate type and amount of storage for SC4S disk buffering. To estimate the storage allocation, follow these steps: +* Start with your estimated maximum events per second that each SC4S server will experience. Based on the maximum throughput of SC4S with disk buffering enabled, the conservative estimate for maximum events per second would be 60K (however, you should use the maximum rate in your environment for this calculation, not the max rate SC4S can handle). +* Next is your average estimated event size based on your data sources. It is common industry practice to estimate log events as 800 bytes on average. +* Then, factor in the maximum length of connectivity downtime you want disk buffering to be able to handle. This measure is very much dependent on your risk tolerance. +* Lastly, syslog-ng imposes significant overhead to maintain its internal data structures (primarily macros) so that the data can be properly "played back" upon network restoration. This overhead currently runs at about 1.7x above the total storage size for the raw messages themselves, and can be higher for "fallback" data sources due to the overlap of syslog-ng macros (data fields) containing some or all of the original message. + + +For example, to protect against a full day of lost connectivity from SC4S to all your indexers at maximum throughput the calculation would look like the following... -60,000 EPS * 86400 seconds * 800 bytes = 3.77186 TB of storage +60,000 EPS * 86400 seconds * 800 bytes * 1.7 = 6.4 TB of storage To configure storage allocation for the SC4S disk buffering, do the following... -Edit the file /opt/sc4s/default/env_file -Add the SC4S_DEST_SPLUNK_HEC_DISKBUFF_DISKBUFSIZE variable to the file and set the value to the number of bytes based on your estimation (e.g. 4147200000000 in the example above) -Splunk does not recommend reducing the disk allocation below 500 GB -Restart SC4S +* Edit the file /opt/sc4s/default/env_file +* Add the SC4S_DEST_SPLUNK_HEC_DISKBUFF_DISKBUFSIZE variable to the file and set the value to the number of bytes based on your estimation (e.g. 7050240000000 in the example above) +* Splunk does not recommend reducing the disk allocation below 500 GB +* Restart SC4S Given that in a connectivity outage to the Indexers events will be saved and read from disk until the buffer is emptied, it is ideal to use the fastest type of storage available. For this reason, NVMe storage is recommended for SC4S disk buffering. diff --git a/docs/performance.md b/docs/performance.md index 14f1b9c..5197570 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -1,4 +1,4 @@ -# Performance +# Performance and Sizing Performance testing against our lab configuration produces the following results and limitations. ## Tested Configuration @@ -34,9 +34,10 @@ and forget protocol making it sensitive to performance. Given this it is highly performance with your hardware and production data samples. The syslog-ng loggen tool available in the SC4S container and the commands above can be utilized in this effort. -Deployment Size | Hardware Spec | Average EPS with average msg size 800 bytes --- | -- | -- -Small | 2 X 3.1 ghz cores1 GB of memory | 2K msg/sec -Medium | 4 X 3.1 ghz cores2 GB of memory | 4.5K msg/sec -Large | 8 X 3.1 ghz cores4 GB of memory | 9K msg/sec -XL | 16 X 3.1 ghz cores8 GB of memory | 18K msg/sec +| Deployment Size | Hardware Spec | Average EPS
800-byte msg | +| -------- | --------- | -------- | +| Small | 4 X 3.1 ghz
1 GB memory | 3.5K msg/sec | +| Medium | 8 X 3.1 ghz
2 GB memory | 7K msg/sec | +| Large | 12 X 3.1 ghz
4 GB memory | 10K msg/sec | +| XL | 16 X 3.1 ghz
8 GB memory | 14K msg/sec | + diff --git a/docs/sources/index.md b/docs/sources/index.md index c32c773..8a416bf 100644 --- a/docs/sources/index.md +++ b/docs/sources/index.md @@ -1,7 +1,21 @@ # Introduction -When using Splunk Connect for Syslog to onboard a data source, the SC4S filter performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source. +When using Splunk Connect for Syslog to onboard a data source, the SC4S filter (or "log path") performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source. SC4S "unique" filters are based either on the port upon which events arrive or the hostname/CIDR block from which they are sent. The "soup" filters run for events that arrive on port 514 (default for syslog), and contain regex and other syslog-specific parsers to identify events from a specific source, apply the correct sourcetype, and set other metadata. Data sources which generate events that are not unique enough to accurately identify with soup filters _must_ employ the "unique" filters (port/hostname/CIDR block) instead -- the soup filters are unavailable for these sources. +## The SC4S "fallback" sourcetype + If SC4S receives an event on port 514 which has no soup filter, that event will be given a "fallback" sourcetype. If you see events in Splunk with the fallback sourcetype, then you should figure out what source the events are from and determine why these events are not being sourcetyped correctly. The most common reason for events categorized as "fallback" is the lack of a SC4S filter for that source, and in some cases a misconfigured relay which alters the integrity of the message format. In most cases this means a new SC4S filter must be developed. In this situation you can either build a filter or file an issue with the community to request help. +The "fallback" sourcetype is formatted in JSON to allow the administrator to see the constituent syslog-ng "macros" (fields) that have been autmaticially parsed by the syslog-ng server An RFC3164 (legacy BSD syslog) "on the wire" raw message is usually (but unfortunately not always) comprised of the following syslog-ng macros, in this order and spacing: +``` +<$PRI> $HOST $LEGACY_MSGHDR$MESSAGE +``` +These fields can be very useful in building a new filter for that sourcetype. In addition, the indexed field `sc4s_syslog_format` is helpful in determining if the incoming message is standard RFC3164. A value of anything other than `rfc3164` or `rfc5424_strict` indicates a vendor purturbation of standard syslog, which will warrant more careful examination when building a filter. + +## Splunk Connect for Syslog and Splunk metadata + +A key aspect of SC4S is to properly set Splunk metadata prior to the data arriving in Splunk (and before any TA processing takes place. The filters will apply the proper index, source, sourcetype, host, and timestamp metadata automatically by individual data source. Proper values for this metadata (including a recommended index) are included with all "out-of-the-box" log paths included with SC4S and are chosen to properly interface with the corresponding TA in Splunk. The administrator will need to ensure all recommneded indexes be created to accept this data if the defaults are not changed. + +It is understood that default values will need to be changed in many installations. Each source documented in this section has a table entitled "Sourcetype and Index Configuration", which highlights the default index and sourcetype for each source. See the section "SC4S metadata configuration" in the "Configuration" page for more information on how to override the default values in this table. + diff --git a/package/Dockerfile b/package/Dockerfile index 7ea7def..77740bb 100644 --- a/package/Dockerfile +++ b/package/Dockerfile @@ -18,7 +18,7 @@ RUN dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noar dnf config-manager --set-enabled PowerTools -y; \ dnf update -y -RUN dnf group install "Development Tools" ;\ +RUN dnf group install "Development Tools" -y ;\ dnf install findutils autoconf \ autoconf automake ca-certificates git libtool pkgconfig bison byacc file \ flex pcre-devel glib2-devel openssl-devel libcurl-devel \ diff --git a/package/etc/conf.d/conflib/_common/syslog_format.conf b/package/etc/conf.d/conflib/_common/syslog_format.conf index 49e6529..b8802a7 100644 --- a/package/etc/conf.d/conflib/_common/syslog_format.conf +++ b/package/etc/conf.d/conflib/_common/syslog_format.conf @@ -7,7 +7,7 @@ filter f_rfc5424_noversion{ filter f_rfc5424_epochtime{ message('^(?(?
(?<\d{1,3}>)(?[1-9][0-9]?) (?(?\d{10})(?:.(?\d{1,9})?)) (?[^ ]+) ))'); }; -filter f_rfc5424_bsdtime{ +filter f_rfc3164_version{ message('^(?(?
(?<\d{1,3}>)(?[1-9][0-9]?) (?[A-Za-z]{3} \d\d \d\d:\d\d:\d\d) (?[^ ]+) ))'); }; rewrite set_rfcnonconformant{ @@ -31,11 +31,11 @@ rewrite set_rfc5424_epochtime{ filter f_is_rfc5424_epochtime{ match("rfc5424_epochtime" value("fields.sc4s_syslog_format")) }; -rewrite set_rfc5424_bsdtime{ - set("rfc5424_bsdtime" value("fields.sc4s_syslog_format")); +rewrite set_rfc3164_version{ + set("rfc3164_version" value("fields.sc4s_syslog_format")); }; -filter f_is_rfc5424_bsdtime{ - match("rfc5424_bsdtime" value("fields.sc4s_syslog_format")) +filter f_is_rfc3164_version{ + match("rfc3164_version" value("fields.sc4s_syslog_format")) }; rewrite set_rfc3164{ set("rfc3164" value("fields.sc4s_syslog_format")); @@ -56,6 +56,6 @@ filter f_is_no_parse{ match("no_parse" value("fields.sc4s_syslog_format")) }; -rewrite set_rfc3894_bsdtime{ - subst('^<\d+>(\d)', "", value("MESSAGE")); +rewrite set_rfc3164_no_version_string{ + subst('(^<\d+>)\d', $1, value("MESSAGE")); }; \ No newline at end of file diff --git a/package/etc/conf.d/conflib/_common/templates.conf b/package/etc/conf.d/conflib/_common/templates.conf index 97db7ef..46b63b2 100644 --- a/package/etc/conf.d/conflib/_common/templates.conf +++ b/package/etc/conf.d/conflib/_common/templates.conf @@ -57,27 +57,25 @@ template t_hdr_sdata_msg { # JSON; for JSON pretty-printing (for debugging) # =============================================================================================== -template t_JSON { - template("$(format-json --scope all-nv-pairs - --exclude fields.* - --exclude .splunk.* - --exclude HOST - --exclude HOST_FROM - --exclude SOURCE - )"); - }; +template t_JSON_3164 { + template('$(format-json --scope rfc3164 + --pair PRI="<$PRI>" + --key LEGACY_MSGHDR + --exclude FACILITY + --exclude PRIORITY + )'); + }; # =============================================================================================== # JSON; for JSON pretty-printing (for RFC5424) # =============================================================================================== template t_JSON_5424 { - template("$(format-json --scope all-nv-pairs - --exclude fields.* - --exclude .splunk.* - --exclude HOST - --exclude HOST_FROM - --exclude RAWMSG - --exclude SOURCE - )"); - }; + template('$(format-json --scope rfc5424 + --pair PRI="<$PRI>" + --key ISODATE + --exclude DATE + --exclude FACILITY + --exclude PRIORITY + )'); + }; diff --git a/package/etc/conf.d/filters/checkpoint/splunk.conf b/package/etc/conf.d/filters/checkpoint/splunk.conf index 701ebc2..87643d1 100644 --- a/package/etc/conf.d/filters/checkpoint/splunk.conf +++ b/package/etc/conf.d/filters/checkpoint/splunk.conf @@ -1,5 +1,6 @@ filter f_checkpoint_splunk { - match('\|(?:origin_sic_name|originsicname)\=[cC][nN]|\|product\=SmartConsole\|' value("RAWMSG") type("pcre")); + match('\|(?:origin_sic_name|originsicname)\=[cC][nN]|\|product\=SmartConsole\|' value("MSG") type("pcre")) or + match('\|(?:origin_sic_name|originsicname)\=[cC][nN]|\|product\=SmartConsole\|' value("LEGACY_MSGHDR") type("pcre")); }; filter f_checkpoint_splunk_alerts { diff --git a/package/etc/conf.d/filters/paloalto/panos.conf b/package/etc/conf.d/filters/paloalto/panos.conf index 6b645b1..dedddf3 100644 --- a/package/etc/conf.d/filters/paloalto/panos.conf +++ b/package/etc/conf.d/filters/paloalto/panos.conf @@ -1,5 +1,5 @@ filter f_is_palalto_format{ - match("rfc5424_bsdtime" value("fields.sc4s_syslog_format")) + match("rfc3164_version" value("fields.sc4s_syslog_format")) or match("rfc3164" value("fields.sc4s_syslog_format")) }; diff --git a/package/etc/conf.d/log_paths/internal.conf.tmpl b/package/etc/conf.d/log_paths/internal.conf.tmpl index 4373f5c..15ea441 100644 --- a/package/etc/conf.d/log_paths/internal.conf.tmpl +++ b/package/etc/conf.d/log_paths/internal.conf.tmpl @@ -17,23 +17,25 @@ log { ); }; - {{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_INTERNAL_METRICS_HEC" "no") | conv.ToBool) }} + {{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_INTERNAL_METRICS_HEC" "no")) }} destination(d_hecmetrics); {{- end}} } else { - {{- if eq (getenv "SC4S_DEBUG_STDOUT" "yes") "yes"}} - destination(d_stdout); - {{- end}} rewrite { r_set_splunk_dest_default(sourcetype("sc4s:events"), index("main"))}; parser {p_add_context_splunk(key("sc4s_events")); }; - {{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_INTERNAL_EVENTS_HEC" "no") | conv.ToBool) }} + {{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_INTERNAL_EVENTS_HEC" "no")) }} destination(d_hec_internal); {{- end}} + {{- if eq (getenv "SC4S_DEBUG_STDOUT" "yes") "yes"}} + destination(d_stdout); + {{- end}} + }; + flags(flow-control,final); }; {{- end}} {{- tmpl.Exec "log_path" "yes" }} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl index da7ffeb..9c64b3b 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl @@ -1,3 +1,4 @@ +# Checkpoint # Generate the custom port if defined {{ $context := dict "port_id" "CHECKPOINT_SPLUNK" "parser" "common" }} {{ tmpl.Exec "t/source_network.t" $context }} @@ -22,12 +23,9 @@ log { set("${PROGRAM}", value(".PROGRAM")); subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM")); }; - rewrite { - r_set_splunk_dest_default(sourcetype("nix:syslog"), index("netops"), source("program:${.PROGRAM}") ) - }; + rewrite { r_set_splunk_dest_default(sourcetype("nix:syslog"), index("netops"), source("program:${.PROGRAM}")) }; parser { p_add_context_splunk(key("checkpoint_os")); }; - } else { parser { kv-parser(prefix(".kv.") pair-separator("|") template("${MSGHDR} ${MSG}")); @@ -74,23 +72,13 @@ log { }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".kv.*")); - }; -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CHECKPOINT_SPLUNK_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CHECKPOINT_SPLUNK_HEC" "no")) }} destination(d_hec); {{- end}} - -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CHECKPOINT_SPLUNK") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CHECKPOINT_SPLUNK" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-cisco_acs.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-cisco_acs.conf.tmpl index 0b9ccfd..15d0963 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-cisco_acs.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-cisco_acs.conf.tmpl @@ -68,29 +68,15 @@ log { parser {p_add_context_splunk(key("cisco_acs")); }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("PID")); - unset(value("LEGACY_MSGHDR")); - unset(value("EPOCH")); - unset(value("VERSION")); - unset(value("TIMESECFRAC")); - groupunset(values("ACS.*")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ACS_HEC" "no") | conv.ToBool) }} - destination(d_hec); +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_ACS_HEC" "no")) }} + destination(d_hec); {{- end}} - {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ACS") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_ACS" "no")) }} destination(d_archive); - {{- end}} +{{- end}} flags(flow-control,final); }; diff --git a/package/etc/conf.d/log_paths/p_rfc3164-cisco_asa.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-cisco_asa.conf.tmpl index 172445d..8b9ca9a 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-cisco_asa.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-cisco_asa.conf.tmpl @@ -19,24 +19,14 @@ log { r_set_splunk_dest_default(sourcetype("cisco:asa"), index("netfw")) }; parser {p_add_context_splunk(key("cisco_asa")); }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ASA_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_ASA_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ASA_LEGACY") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_ASA_LEGACY" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-cisco_ios.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-cisco_ios.conf.tmpl index cb90f70..015f86b 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-cisco_ios.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-cisco_ios.conf.tmpl @@ -18,28 +18,15 @@ log { guess-time-zone(); r_set_splunk_dest_default(sourcetype("cisco:ios"), index("netops")) }; - parser { - p_add_context_splunk(key("cisco_ios")); - }; - + parser { p_add_context_splunk(key("cisco_ios")); }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".cisco.*")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_IOS_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_IOS_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_IOS") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_IOS" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-cisco_ise.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-cisco_ise.conf.tmpl index 544c0a3..7157f5e 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-cisco_ise.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-cisco_ise.conf.tmpl @@ -68,29 +68,15 @@ log { parser {p_add_context_splunk(key("cisco_ise")); }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("PID")); - unset(value("LEGACY_MSGHDR")); - unset(value("EPOCH")); - unset(value("VERSION")); - unset(value("TIMESECFRAC")); - groupunset(values("ISE.*")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ISE_HEC" "no") | conv.ToBool) }} - destination(d_hec); +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_ISE_HEC" "no")) }} + destination(d_hec); {{- end}} - {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ISE") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_ISE" "no")) }} destination(d_archive); - {{- end}} +{{- end}} flags(flow-control,final); }; diff --git a/package/etc/conf.d/log_paths/p_rfc3164-cisco_nxos.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-cisco_nxos.conf.tmpl index 59db061..6c40bc0 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-cisco_nxos.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-cisco_nxos.conf.tmpl @@ -19,27 +19,15 @@ log { r_set_splunk_dest_default(sourcetype("cisco:ios"), index("netops"), template("t_hdr_msg")) }; - parser { - p_add_context_splunk(key("cisco_nx_os")); - }; + parser { p_add_context_splunk(key("cisco_nx_os")); }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_NXOS_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_NXOS_HEC" "no")) }} destination(d_hec); {{- end}} - -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_NXOS") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_NXOS" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-forcepoint_webprotect.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-forcepoint_webprotect.conf.tmpl index 72d4de7..dc65e02 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-forcepoint_webprotect.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-forcepoint_webprotect.conf.tmpl @@ -17,27 +17,17 @@ log { rewrite { subst(" [^ =]+\=\-", "", value("MESSAGE"), flags("global")); set("forcepoint_webprotect", value("fields.sc4s_vendor_product")); - r_set_splunk_dest_default(sourcetype("websense:cg:kv"), index("netproxy"), template("t_hdr_msg")) + r_set_splunk_dest_default(sourcetype("websense:cg:kv"), index("netproxy")) }; parser {p_add_context_splunk(key("forcepoint_webprotect")); }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_FORCEPOINT_WEBPROTECT_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_FORCEPOINT_WEBPROTECT_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_FORCEPOINT_WEBPROTECT") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_FORCEPOINT_WEBPROTECT" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-fortinet_fortios.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-fortinet_fortios.conf.tmpl index 29f17fc..a3bfc99 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-fortinet_fortios.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-fortinet_fortios.conf.tmpl @@ -39,24 +39,14 @@ log { }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".kv.*")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_FORTINET_FORTIOS_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_FORTINET_FORTIOS_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_FORTINET_FORTIOS") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_FORTINET_FORTIOS" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-infoblox.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-infoblox.conf.tmpl index ecfad6e..1c047af 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-infoblox.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-infoblox.conf.tmpl @@ -1,4 +1,4 @@ -# Juniper IDP +# Infoblox {{ $context := dict "port_id" "INFOBLOX" "parser" "common" }} {{ tmpl.Exec "t/source_network.t" $context }} @@ -22,11 +22,8 @@ log { set("${PROGRAM}", value(".PROGRAM")); subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM")); r_set_splunk_dest_default(sourcetype("infoblox:dns"), index("netdns"), source("program:${.PROGRAM}")) - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - }; - parser { - p_add_context_splunk(key("infoblox_dns")); }; + parser { p_add_context_splunk(key("infoblox_dns")); }; } elif { filter{program("dhcpd")}; rewrite { @@ -34,11 +31,8 @@ log { set("${PROGRAM}", value(".PROGRAM")); subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM")); r_set_splunk_dest_default(sourcetype("infoblox:dhcp"), index("netipam"), source("program:${.PROGRAM}")) - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - }; - parser { - p_add_context_splunk(key("infoblox_dhcp")); }; + parser { p_add_context_splunk(key("infoblox_dhcp")); }; } elif { filter{program("threat-protect-log")}; rewrite { @@ -46,48 +40,27 @@ log { set("${PROGRAM}", value(".PROGRAM")); subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM")); r_set_splunk_dest_default(sourcetype("infoblox:threat"), index("netids"), source("program:${.PROGRAM}")) - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - }; - parser { - p_add_context_splunk(key("infoblox_threat")); }; + parser { p_add_context_splunk(key("infoblox_threat")); }; } else { rewrite { set("nix_syslog", value("fields.sc4s_vendor_product")); subst("^[^\t]+\t", "", value("MESSAGE"), flags("global")); set("${PROGRAM}", value(".PROGRAM")); subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM")); + r_set_splunk_dest_default(sourcetype("nix:syslog"), index("osnix"), source("program:${.PROGRAM}")) }; - - rewrite { - r_set_splunk_dest_default(sourcetype("nix:syslog"), index("osnix"), source("program:${.PROGRAM}") ) - set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); - - }; - parser { p_add_context_splunk(key("nix_syslog")); }; }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - unset(value("PID")); - groupunset(values(".kv.*")); - }; - - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_INFOBLOX_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_INFOBLOX_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_INFOBLOX") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_INFOBLOX" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_idp.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_idp.conf.tmpl index 16f2c0c..5aecfc0 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_idp.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_idp.conf.tmpl @@ -18,30 +18,15 @@ log { set("juniper_idp", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("juniper:idp"), index("netids")) }; - parser { - p_add_context_splunk(key("juniper_idp")); - }; - + parser { p_add_context_splunk(key("juniper_idp")); }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_sdata_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_sdata_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - unset(value("PID")); - groupunset(values(".kv.*")); - }; - - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_IDP_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_JUNIPER_IDP_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_IDP") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_JUNIPER_IDP" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_junos.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_junos.conf.tmpl index f0c8a1d..004e8c8 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_junos.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_junos.conf.tmpl @@ -39,22 +39,13 @@ log { }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_JUNOS_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_JUNIPER_JUNOS_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_JUNOS") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_JUNIPER_JUNOS" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_netscreen.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_netscreen.conf.tmpl index dde7d54..ece524e 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_netscreen.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_netscreen.conf.tmpl @@ -17,28 +17,15 @@ log { set("juniper_netscreen", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("netscreen:firewall"), index("netfw")) }; - - parser { - p_add_context_splunk(key("juniper_netscreen")); - }; - + parser { p_add_context_splunk(key("juniper_netscreen")); }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_NETSCREEN_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_JUNIPER_NETSCREEN_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_NETSCREEN") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_JUNIPER_NETSCREEN" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm.conf.tmpl index 8d8d999..4cac2a7 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm.conf.tmpl @@ -16,30 +16,17 @@ log { rewrite { set("juniper_nsm", value("fields.sc4s_vendor_product")); - r_set_splunk_dest_default(sourcetype("juniper:nsm"), index("netfw"))}; - - parser { - p_add_context_splunk(key("juniper_nsm")); + r_set_splunk_dest_default(sourcetype("juniper:nsm"), index("netfw")) }; - + parser { p_add_context_splunk(key("juniper_nsm")); }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; - - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - - {{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_NSM_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_JUNIPER_NSM_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_NSM") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_JUNIPER_NSM" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm_idp.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm_idp.conf.tmpl index c1196c5..f33f3f6 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm_idp.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm_idp.conf.tmpl @@ -16,27 +16,16 @@ log { rewrite { set("juniper_nsm_idp", value("fields.sc4s_vendor_product")); - r_set_splunk_dest_default(sourcetype("juniper:nsm:idp"), index("netids"))}; - parser { - p_add_context_splunk(key("juniper_nsm_idp")); + r_set_splunk_dest_default(sourcetype("juniper:nsm:idp"), index("netids")) }; + parser { p_add_context_splunk(key("juniper_nsm_idp")); }; + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_standard))" value("MSG")); }; - - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_standard))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_NSM_IDP_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_JUNIPER_NSM_IDP_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_NSM_IDP") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_JUNIPER_NSM_IDP" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-microfocus_arcsight.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-microfocus_arcsight.conf.tmpl index 8d26d45..25ddce5 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-microfocus_arcsight.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-microfocus_arcsight.conf.tmpl @@ -49,9 +49,7 @@ log { r_set_splunk_dest_default(sourcetype("cef"), index("main")) }; - parser { - p_add_context_splunk(key("cef_{fields.cef_device_vendor}_${fields.cef_device_product}")); - }; + parser { p_add_context_splunk(key("cef_{fields.cef_device_vendor}_${fields.cef_device_product}")); }; parser (p_microfocus_arcsight_header); @@ -72,19 +70,13 @@ log { #We want to unset the fields we won't need, as this is copied into the #disk queue for network destinations. This can be very disk expensive #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".cef.*")); - }; + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-paloalto_panos.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-paloalto_panos.conf.tmpl index ee4caaf..8c6f97f 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-paloalto_panos.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-paloalto_panos.conf.tmpl @@ -16,13 +16,11 @@ log { # The palo message does not include a program value in the header, unfortunately. # The use of colon in the message tricks the RFC3164 parser; We will re-write the message # so the parser will not incorrectly break it apart. - # While we are at it we will save the mesage type into the program field so parser can find it. + # While we are at it we will save the message type into the program field so parser can find it. rewrite { set("paloalto_panos", value("fields.sc4s_vendor_product")); set("${LEGACY_MSGHDR}${MESSAGE}" value("MESSAGE")); - unset(value("LEGACY_MSGHDR")); - unset(value("PROGRAM")); }; parser { #basic parsing @@ -77,23 +75,13 @@ log { }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".pan.*")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_PALOALTO_PANOS_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_PALOALTO_PANOS_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_PALOALTO_PANOS") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_PALOALTO_PANOS" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps.conf.tmpl index 0c805e7..46c5267 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps.conf.tmpl @@ -16,37 +16,23 @@ log { rewrite { set("proofpoint_pps_filter", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("pps_filter_log"), index("email"))}; - parser { - p_add_context_splunk(key("proofpoint_pps_filter")); - }; + parser { p_add_context_splunk(key("proofpoint_pps_filter")); }; } else { rewrite { set("proofpoint_pps_sendmail", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("pps_mail_log"), index("email"))}; - parser { - p_add_context_splunk(key("proofpoint_pps_sendmail")); - }; + parser { p_add_context_splunk(key("proofpoint_pps_sendmail")); }; }; parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".kv.*")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_PROOFPOINT_PPS_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_PROOFPOINT_PPS_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_PROOFPOINT_PPS") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_PROOFPOINT_PPS" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3165-symantec_brightmail.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-symantec_brightmail.conf.tmpl similarity index 59% rename from package/etc/conf.d/log_paths/p_rfc3165-symantec_brightmail.conf.tmpl rename to package/etc/conf.d/log_paths/p_rfc3164-symantec_brightmail.conf.tmpl index 70c1298..a2b4c57 100644 --- a/package/etc/conf.d/log_paths/p_rfc3165-symantec_brightmail.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-symantec_brightmail.conf.tmpl @@ -1,3 +1,4 @@ +#Symantec Brightmail {{- if ((getenv "SC4S_SOURCE_FF_SYMANTEC_BRIGHTMAIL_GROUPMSG" "yes") | conv.ToBool) }} filter f_symantec_brightmail_complete{ match("yes", value("SMG.COMPLETE") type(glob)); @@ -46,34 +47,19 @@ log { rewrite { set("symantec_brightmail", value("fields.sc4s_vendor_product")); + r_set_splunk_dest_default(sourcetype("symantec:smg:mail"), index("email"), source("program:${.PROGRAM}")) }; - - rewrite { - r_set_splunk_dest_default(sourcetype("symantec:smg:mail"), index("email"), source("program:${.PROGRAM}") ) - - }; - parser { p_add_context_splunk(key("symantec_brightmail")); }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - - {{- if ((getenv "SC4S_SYMANTEC_BRIGHTMAIL_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_SYMANTEC_BRIGHTMAIL_HEC" "no") | conv.ToBool) }} - destination(d_hec); - {{- end}} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_SYMANTEC_BRIGHTMAIL_HEC" "no")) }} + destination(d_hec); +{{- end}} - {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_SYMANTEC_BRIGHTMAIL") }} - destination(d_archive); - {{- end}} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_SYMANTEC_BRIGHTMAIL" "no")) }} + destination(d_archive); +{{- end}} }; } else { @@ -81,39 +67,23 @@ log { rewrite { set("symantec_brightmail", value("fields.sc4s_vendor_product")); + r_set_splunk_dest_default(sourcetype("symantec:smg"), index("email"), source("program:${.PROGRAM}")) }; - - rewrite { - r_set_splunk_dest_default(sourcetype("symantec:smg"), index("email"), source("program:${.PROGRAM}") ) - - }; - parser { p_add_context_splunk(key("symantec_brightmail")); }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - - {{- if ((getenv "SC4S_SYMANTEC_BRIGHTMAIL_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_SYMANTEC_BRIGHTMAIL_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_SYMANTEC_BRIGHTMAIL_HEC" "no")) }} destination(d_hec); - {{- end}} +{{- end}} - {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_SYMANTEC_BRIGHTMAIL") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_SYMANTEC_BRIGHTMAIL" "no")) }} destination(d_archive); - {{- end}} +{{- end}} {{- if ((getenv "SC4S_SOURCE_FF_SYMANTEC_BRIGHTMAIL_GROUPMSG" "yes") | conv.ToBool) }} }; {{- end}} - flags(flow-control,final); }; {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-ubiquiti_unifi.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-ubiquiti_unifi.conf.tmpl index 7ca3176..a6ab503 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-ubiquiti_unifi.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-ubiquiti_unifi.conf.tmpl @@ -17,28 +17,26 @@ log { parser {p_add_context_splunk(key("ubiquiti_unifi")); }; - #Fiewall + #Firewall if { filter { match("ubiquiti_unifi_fw", value("fields.sc4s_vendor_product") type(glob) ); }; - if (match("[^)]\s\S+\skernel:\s[^ll\sheader][^\[\d+.\d+\]]\S+\s\w+:" value("RAWMSG"))) { - rewrite { r_set_splunk_dest_default(sourcetype("ubnt:threat"), index("netids")) - set("${LEGACY_MSGHDR}${MSG}" value("MSG"));}; + rewrite { set("${LEGACY_MSGHDR}${MSG}" value("MSG")); }; + + if (match("[^)]\s\S+\skernel:\s[^ll\sheader][^\[\d+.\d+\]]\S+\s\w+:" value("MSG"))) { + rewrite { r_set_splunk_dest_default(sourcetype("ubnt:threat"), index("netids")) }; parser {p_add_context_splunk(key("ubiquiti_unifi_threat")); }; - } elif (match("\S+\slinkcheck:" value("RAWMSG"))) { - rewrite { r_set_splunk_dest_default(sourcetype("ubnt:link"), index("netops")) - set("${LEGACY_MSGHDR}${MSG}" value("MSG"));}; + } elif (match("\S+\slinkcheck:" value("MSG"))) { + rewrite { r_set_splunk_dest_default(sourcetype("ubnt:link"), index("netops")) }; parser {p_add_context_splunk(key("ubiquiti_unifi_link")); }; - } elif (match("\d+:\d+:\d+\s\S+\ssudo:" value("RAWMSG"))) { - rewrite { r_set_splunk_dest_default(sourcetype("ubnt:sudo"), index("netops")) - set("${LEGACY_MSGHDR}${MSG}" value("MSG"));}; + } elif (match("\d+:\d+:\d+\s\S+\ssudo:" value("MSG"))) { + rewrite { r_set_splunk_dest_default(sourcetype("ubnt:sudo"), index("netops")) }; parser {p_add_context_splunk(key("ubiquiti_unifi_sudo")); }; } else { rewrite { r_set_splunk_dest_default(sourcetype("ubnt:fw"), index("netfw")); - set("${LEGACY_MSGHDR}${MSG}" value("MSG")); }; parser {p_add_context_splunk(key("ubiquiti_unifi_fw")); }; }; @@ -47,25 +45,28 @@ log { filter { host('^(?US[^,]{1,10}),(?[a-z0-9]{9,16}),(?v\d{1,2}\.\d{1,2}\.\d{1,2}\.\d{1,6})', flags("store-matches")); }; - if (match("hostapd:\s+ath" value("RAWMSG"))) { + + rewrite { set("${LEGACY_MSGHDR}${MSG}" value("MSG")); }; + + if (match('hostapd:\s+ath' value("MSG"))) { rewrite { + set("ubiquiti_unifi_wireless", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("ubnt:hostapd"), index("netops")); - set("${LEGACY_MSGHDR}${MSG}" value("MSG")); - set("${HOST_FROM}", value("HOST")); + set("${FULLHOST_FROM}", value("HOST")); }; parser {p_add_context_splunk(key("ubiquiti_unifi_wireless")); }; - } elif (match("\d+:\d+:\d+\s\S+\smcad:" value("RAWMSG"))) { + } elif (match('\d+:\d+:\d+\s\S+\smcad:' value("MSG"))) { rewrite { + set("ubiquiti_unifi_wireless", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("ubnt:mcad"), index("netops")); - set("${LEGACY_MSGHDR}${MSG}" value("MSG")); - set("${HOST_FROM}", value("HOST")); + set("${FULLHOST_FROM}", value("HOST")); }; parser {p_add_context_splunk(key("ubiquiti_unifi_wireless")); }; } else { rewrite { + set("ubiquiti_unifi_switch", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("ubnt:switch"), index("netops")); - set("${LEGACY_MSGHDR}${MSG}" value("MSG")); - set("${FROM_HOST}",value("HOST")); + set("${FULLHOST_FROM}",value("HOST")); set("${model}", value("fields.model")); set("${serial}", value("fields.serial")); set("${firmware}", value("fields.firmware")); @@ -79,50 +80,41 @@ log { program('^(?U\d[^,]{1,10}),(?[a-z0-9]{9,16}),(?v\d{1,2}\.\d{1,2}\.\d{1,2}\.\d{1,6})', flags("store-matches")); }; rewrite { + set("ubiquiti_unifi_wireless", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("ubnt:wireless"), index("netops")); - set("${FROM_HOST}",value("HOST")); + set("${FULLHOST_FROM}",value("HOST")); set("${model}", value("fields.model")); set("${serial}", value("fields.serial")); set("${firmware}", value("fields.firmware")); }; parser {p_add_context_splunk(key("ubiquiti_unifi_wireless")); }; - } elif (match("traputil.c\(696\) " value("RAWMSG"))) { + } elif (match("traputil.c\(696\) " value("MSG"))) { rewrite { + set("ubiquiti_unifi_edge_switch", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("ubnt:edgeswitch"), index("netops")); - set("${HOST_FROM}", value("HOST")); - set("${LEGACY_MSGHDR}${MSG}" value("MSG")); + set("${FULLHOST_FROM}", value("HOST")); }; parser {p_add_context_splunk(key("ubiquiti_unifi_edge_switch")); }; } else { rewrite { + set("ubiquiti_unifi", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("ubnt"), index("netops")); - set("${HOST_FROM}", value("HOST")); - set("${LEGACY_MSGHDR}${MSG}" value("MSG")); + set("${FULLHOST_FROM}", value("HOST")); }; parser {p_add_context_splunk(key("ubiquiti_unifi")); }; }; - parser (compliance_meta_by_source); - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_PROOFPOINT_UBIQUITI_UNIFI_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_PROOFPOINT_UBIQUITI_UNIFI_HEC" "no")) }} destination(d_hec); {{- end}} - -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_UBIQUITI_UNIFI") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_UBIQUITI_UNIFI" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-zscaler_nss.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-zscaler_nss.conf.tmpl index 0bcdf3d..d70f139 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-zscaler_nss.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-zscaler_nss.conf.tmpl @@ -1,4 +1,4 @@ -# Proofpoint +# Zscaler {{ $context := dict "port_id" "ZSCALER_NSS" "parser" "common" }} {{ tmpl.Exec "t/source_network.t" $context }} # The following is an inline template; we will use this to generate the actual log path @@ -58,26 +58,14 @@ log { }; }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".kv.*")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_ZSCALER_NSS_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_ZSCALER_NSS_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_ZSCALER_NSS") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_ZSCALER_NSS" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc5424-noversion_cisco_asa.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc5424-noversion_cisco_asa.conf.tmpl index 065c824..1b93b48 100644 --- a/package/etc/conf.d/log_paths/p_rfc5424-noversion_cisco_asa.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc5424-noversion_cisco_asa.conf.tmpl @@ -15,27 +15,17 @@ log { rewrite { set("cisco_asa", value("fields.sc4s_vendor_product")); - r_set_splunk_dest_default(sourcetype("cisco:asa"), index("netfw"))}; + r_set_splunk_dest_default(sourcetype("cisco:asa"), index("netfw")) + }; parser {p_add_context_splunk(key("cisco_asa")); }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".kv.*")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ASA_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_ASA_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ASA") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_ASA" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc5424-noversion_symantec_proxy.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc5424-noversion_symantec_proxy.conf.tmpl index c455a9d..7a40ebf 100644 --- a/package/etc/conf.d/log_paths/p_rfc5424-noversion_symantec_proxy.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc5424-noversion_symantec_proxy.conf.tmpl @@ -19,26 +19,14 @@ log { }; parser {p_add_context_splunk(key("bluecoat_proxy")); }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); }; - - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_msg_only))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".kv.*")); - }; - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_SYMANTEC_PROXY_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_SYMANTEC_PROXY_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_SYMANTEC_PROXY") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_SYMANTEC_PROXY" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc5424-strict_juniper_junos.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc5424-strict_juniper_junos.conf.tmpl index 0d103a8..e808f85 100644 --- a/package/etc/conf.d/log_paths/p_rfc5424-strict_juniper_junos.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc5424-strict_juniper_junos.conf.tmpl @@ -46,22 +46,16 @@ log { #We want to unset the fields we won't need, as this is copied into the #disk queue for network destinations. This can be very disk expensive #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_JSON_5424))" value("MSG")); - unset(value("RAWMSG")); - groupunset(values(".kv.*")); - }; + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_JSON_5424))" value("MSG")); }; -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNOS_STRUCTURED_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_JUNOS_STRUCTURED_HEC" "no")) }} destination(d_hec); {{- end}} - -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNOS_STRUCTURED") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_JUNOS_STRUCTURED" "no")) }} destination(d_archive); {{- end}} - flags(flow-control,final); }; {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc5424_epoch-cisco_meraki.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc5424_epoch-cisco_meraki.conf.tmpl index 793b356..f080c1c 100644 --- a/package/etc/conf.d/log_paths/p_rfc5424_epoch-cisco_meraki.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc5424_epoch-cisco_meraki.conf.tmpl @@ -1,4 +1,4 @@ -# Checkpoint Splunk format +# Cisco Meraki {{ $context := dict "port_id" "CISCO_MERAKI" "parser" "common" }} {{ tmpl.Exec "t/source_network.t" $context }} # The following is an inline template; we will use this to generate the actual log path @@ -17,25 +17,15 @@ log { r_set_splunk_dest_default(sourcetype("meraki"), index("netfw")) }; parser {p_add_context_splunk(key("cisco_meraki")); }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".kv.*")); - }; -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_MERAKI_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_CISCO_MERAKI_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_MERAKI") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_CISCO_MERAKI" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_vmware_vsphere.conf.tmpl b/package/etc/conf.d/log_paths/p_vmware_vsphere.conf.tmpl index 5396355..5d31966 100644 --- a/package/etc/conf.d/log_paths/p_vmware_vsphere.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_vmware_vsphere.conf.tmpl @@ -1,3 +1,4 @@ +#VMware ESXi and NSX # Generate the custom port if defined {{ $context := dict "port_id" "VMWARE" "parser" "common" }} {{ tmpl.Exec "t/source_network.t" $context }} @@ -7,64 +8,66 @@ log { {{- if eq (.) "yes"}} source(s_DEFAULT); - filter(f_vmware_all); {{- end}} {{- if eq (.) "no"}} source (s_VMWARE); {{- end}} - #NSX first because its the cheapest check if { filter(f_is_rfc5424_strict); filter(f_vmware_nsx); rewrite { + set("vmware_nsx", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("vmware:vsphere:nsx"), index("main"), source("program:${PROGRAM}")); - set("$(template ${.splunk.sc4s_template} $(template t_JSON_5424))" value("MSG")); - }; - parser { - p_add_context_splunk(key("vmware_nsx")); }; + parser { p_add_context_splunk(key("vmware_nsx")); }; + parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_JSON_5424))" value("MSG")); }; + } elif { filter(f_vmware_nsx); rewrite { + set("vmware_nsx", value("fields.sc4s_vendor_product")); set("${PROGRAM}", value(".PROGRAM")); subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM")); r_set_splunk_dest_default(sourcetype("vmware:vsphere:nsx"), index("main"), source("program:${.PROGRAM}")); - set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); - }; - parser { - p_add_context_splunk(key("vmware_nsx")); }; + parser { p_add_context_splunk(key("vmware_nsx")); }; + parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); }; + #esx things } elif { filter(f_is_rfc5424_strict); filter(f_vmware_vsphere); rewrite { + set("vmware_vsphere_esx", value("fields.sc4s_vendor_product")); r_set_splunk_dest_default(sourcetype("vmware:vsphere:esx"), index("main"), source("program:${PROGRAM}")); - set("$(template ${.splunk.sc4s_template} $(template t_JSON_5424))" value("MSG")); - }; - parser { - p_add_context_splunk(key("vmware_esx")); }; + parser { p_add_context_splunk(key("vmware_esx")); }; + parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_JSON_5424))" value("MSG")); }; + } elif { filter(f_vmware_vsphere); rewrite { + set("vmware_vsphere_esx", value("fields.sc4s_vendor_product")); set("${PROGRAM}", value(".PROGRAM")); subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM")); r_set_splunk_dest_default(sourcetype("vmware:vsphere:esx"), index("main"), source("program:${.PROGRAM}")); - set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); - }; - parser { - p_add_context_splunk(key("vmware_esx")); }; + parser { p_add_context_splunk(key("vmware_esx")); }; + parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); }; + } else { rewrite { @@ -74,34 +77,17 @@ log { subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM")); }; - rewrite { - r_set_splunk_dest_default(sourcetype("nix:syslog"), index("osnix"), source("program:${.PROGRAM}") ) - - }; - + rewrite { r_set_splunk_dest_default(sourcetype("nix:syslog"), index("osnix"), source("program:${.PROGRAM}")) }; parser { p_add_context_splunk(key("nix_syslog")); }; - parser (compliance_meta_by_source); - - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); }; }; - parser (compliance_meta_by_source); - -{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_VMWARE_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_VMWARE_HEC" "no")) }} destination(d_hec); {{- end}} - -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_VMWARE") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_VMWARE" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_za_nix_syslog.conf.tmpl b/package/etc/conf.d/log_paths/p_za_nix_syslog.conf.tmpl index c8f4e2a..91fa349 100644 --- a/package/etc/conf.d/log_paths/p_za_nix_syslog.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_za_nix_syslog.conf.tmpl @@ -1,4 +1,4 @@ -# Proofpoint +# Linux/Unix OS system logs {{ $context := dict "port_id" "NIX_SYSLOG" "parser" "common" }} {{ tmpl.Exec "t/source_network.t" $context }} # The following is an inline template; we will use this to generate the actual log path @@ -19,30 +19,16 @@ log { subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM")); }; - rewrite { - r_set_splunk_dest_default(sourcetype("nix:syslog"), index("osnix"), source("program:${.PROGRAM}") ) - - }; - + rewrite { r_set_splunk_dest_default(sourcetype("nix:syslog"), index("osnix"), source("program:${.PROGRAM}")) }; parser { p_add_context_splunk(key("nix_syslog")); }; - parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); }; - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${.splunk.sc4s_template} $(template t_legacy_hdr_msg))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - }; - -{{- if ((getenv "SC4S_NIX_SYSLOG_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_NIX_SYSLOG_HEC" "no") | conv.ToBool) }} +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_NIX_SYSLOG_HEC" "no")) }} destination(d_hec); {{- end}} -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_NIX_SYSLOG") }} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_NIX_SYSLOG" "no")) }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_zz_fallback.conf.tmpl b/package/etc/conf.d/log_paths/p_zz_fallback.conf.tmpl index cb9ed87..44be960 100644 --- a/package/etc/conf.d/log_paths/p_zz_fallback.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_zz_fallback.conf.tmpl @@ -1,59 +1,32 @@ +# Fallback for un-parsed sources log { source(s_DEFAULT); + rewrite { set("SC4S_fallback", value("fields.sc4s_vendor_product")); }; + if { filter(f_is_rfc5424_strict); - rewrite { - r_set_splunk_dest_default(sourcetype("sc4s:fallback"), index("main")); - set("$(template ${.splunk.sc4s_template} $(template t_JSON))" value("MSG")); - }; - parser { - p_add_context_splunk(key("sc4s_fallback")); - }; - {{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_ARCHIVE_HEC" "no") | conv.ToBool) }} - destination(d_hec); - {{- end}} - - - #in fallback archive only write rawmsg as msg - rewrite { - unset(value("RAWMSG")); - groupunset(values(".kv.*")); - }; - - {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_FALLBACK") }} - destination(d_archive); - {{- end}} + rewrite { r_set_splunk_dest_default(sourcetype("sc4s:fallback"), index("main")); }; + parser { p_add_context_splunk(key("sc4s_fallback")); }; + parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_JSON_5424))" value("MSG")); }; } else { - rewrite { - r_set_splunk_dest_default(sourcetype("sc4s:fallback"), index("main") ); - set("$(template ${.splunk.sc4s_template} $(template t_JSON))" value("MSG")); - }; - parser { - p_add_context_splunk(key("sc4s_fallback")); - }; - - {{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_ARCHIVE_HEC" "no") | conv.ToBool) }} - destination(d_hec); - {{- end}} - - - #in fallback archive only write rawmsg as msg - rewrite { - set("$RAWMSG" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".kv.*")); - }; - - {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_FALLBACK") }} - destination(d_archive); - {{- end}} + + rewrite { r_set_splunk_dest_default(sourcetype("sc4s:fallback"), index("main")); }; + parser { p_add_context_splunk(key("sc4s_fallback")); }; + parser (compliance_meta_by_source); + rewrite { set("$(template ${.splunk.sc4s_template} $(template t_JSON_3164))" value("MSG")); }; }; +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_FALLBACK_HEC" "no")) }} + destination(d_hec); +{{- end}} +{{- if or (conv.ToBool (getenv "SC4S_ARCHIVE_GLOBAL" "no")) (conv.ToBool (getenv "SC4S_ARCHIVE_FALLBACK" "no")) }} + destination(d_archive); +{{- end}} flags(flow-control,fallback); -}; + +}; \ No newline at end of file diff --git a/package/etc/conf.d/log_paths/startup.conf.tmpl b/package/etc/conf.d/log_paths/startup.conf.tmpl new file mode 100644 index 0000000..3c5df61 --- /dev/null +++ b/package/etc/conf.d/log_paths/startup.conf.tmpl @@ -0,0 +1,37 @@ +{{- define "log_path"}} +log { + source(s_startup_out); + + + rewrite { r_set_splunk_dest_default(sourcetype("sc4s:events"), index("main"))}; + parser {p_add_context_splunk(key("sc4s_events:startup:out")); }; + + {{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_INTERNAL_EVENTS_HEC" "no") | conv.ToBool) }} + destination(d_hec_internal); + {{- end}} + + {{- if eq (getenv "SC4S_DEBUG_STDOUT" "yes") "yes"}} + destination(d_stdout); + {{- end}} + + flags(flow-control,final); + }; +log { + source(s_startup_err); + + + rewrite { r_set_splunk_dest_default(sourcetype("sc4s:events:startup:err"), index("main"))}; + parser {p_add_context_splunk(key("sc4s_events")); }; + +{{- if or (conv.ToBool (getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes")) (conv.ToBool (getenv "SC4S_DEST_INTERNAL_EVENTS_HEC" "no")) }} + destination(d_hec_internal); +{{- end}} + +{{- if eq (getenv "SC4S_DEBUG_STDOUT" "yes") "yes"}} + destination(d_stdout); +{{- end}} + + flags(flow-control,final); + }; +{{- end}} +{{- tmpl.Exec "log_path" "yes" }} diff --git a/package/etc/conf.d/sources/startup.conf b/package/etc/conf.d/sources/startup.conf new file mode 100644 index 0000000..00fbce7 --- /dev/null +++ b/package/etc/conf.d/sources/startup.conf @@ -0,0 +1,13 @@ +source s_startup_out { + file("/var/log/syslog-ng.out" + program-override("syslog-ng-config") + flags(no-hostname,no-parse,assume-utf8) + ); + +}; +source s_startup_err { + file("/var/log/syslog-ng.err" + program-override("syslog-ng-config") + flags(no-hostname,no-parse,assume-utf8) + ); +}; \ No newline at end of file diff --git a/package/etc/context_templates/compliance_meta_by_source.conf b/package/etc/context_templates/compliance_meta_by_source.conf index 90b67f4..e74da82 100644 --- a/package/etc/context_templates/compliance_meta_by_source.conf +++ b/package/etc/context_templates/compliance_meta_by_source.conf @@ -1,4 +1,3 @@ -@version: 3.24 filter f_test_test { # host("something-*" type(glob)) or # netmask(169.254.100.0/24) diff --git a/package/etc/context_templates/vendor_product_by_source.conf b/package/etc/context_templates/vendor_product_by_source.conf index ac87c2e..1f5f13f 100644 --- a/package/etc/context_templates/vendor_product_by_source.conf +++ b/package/etc/context_templates/vendor_product_by_source.conf @@ -1,5 +1,3 @@ -@version: 3.24 - filter f_test_test { host("testvp-*" type(glob)) #or netmask(xxx.xxx.xxx.xxx/xx) diff --git a/package/etc/go_templates/source_network.t b/package/etc/go_templates/source_network.t index 8a71d29..8b43665 100644 --- a/package/etc/go_templates/source_network.t +++ b/package/etc/go_templates/source_network.t @@ -62,13 +62,13 @@ source s_{{ .port_id}} { {{ if eq .parser "rfc5424_strict" }} filter(f_rfc5424_strict); parser { - syslog-parser(flags(syslog-protocol store-raw-message)); + syslog-parser(flags(syslog-protocol)); }; rewrite(set_rfc5424_strict); {{- else if eq .parser "rfc5424_noversion" }} filter(f_rfc5424_noversion); parser { - syslog-parser(flags(syslog-protocol store-raw-message)); + syslog-parser(flags(syslog-protocol)); }; rewrite(set_rfc5424_noversion); {{- else if eq .parser "cisco_parser" }} @@ -79,7 +79,7 @@ source s_{{ .port_id}} { rewrite(set_rfc5424_epochtime); {{- else if eq .parser "rfc3164" }} parser { - syslog-parser(time-zone({{getenv "SC4S_DEFAULT_TIMEZONE" "GMT"}}) flags(store-raw-message)); + syslog-parser(time-zone({{getenv "SC4S_DEFAULT_TIMEZONE" "GMT"}}) ); }; rewrite(set_rfc3164); {{- else if eq .parser "no_parse" }} @@ -87,31 +87,31 @@ source s_{{ .port_id}} { {{- else }} if {filter(f_rfc5424_strict); parser { - syslog-parser(flags(syslog-protocol store-raw-message)); + syslog-parser(flags(syslog-protocol)); }; rewrite(set_rfc5424_strict); } elif { filter(f_rfc5424_noversion); parser { - syslog-parser(flags(syslog-protocol store-raw-message)); + syslog-parser(flags(syslog-protocol)); }; rewrite(set_rfc5424_noversion); } elif { parser {cisco-parser()}; rewrite(set_cisco_ios); } elif { - filter(f_rfc5424_bsdtime); - rewrite(set_rfc3894_bsdtime); + filter(f_rfc3164_version); + rewrite(set_rfc3164_no_version_string); parser { - syslog-parser(time-zone({{- getenv "SC4S_DEFAULT_TIMEZONE" "GMT"}}) flags(store-raw-message, guess-timezone)); + syslog-parser(time-zone({{- getenv "SC4S_DEFAULT_TIMEZONE" "GMT"}}) flags(guess-timezone)); }; - rewrite(set_rfc5424_bsdtime); + rewrite(set_rfc3164_version); } elif { parser (p_cisco_meraki); rewrite(set_rfc5424_epochtime); } else { parser { - syslog-parser(time-zone({{- getenv "SC4S_DEFAULT_TIMEZONE" "GMT"}}) flags(store-raw-message, guess-timezone)); + syslog-parser(time-zone({{- getenv "SC4S_DEFAULT_TIMEZONE" "GMT"}}) flags(guess-timezone)); }; rewrite(set_rfc3164); }; diff --git a/package/etc/syslog-ng.conf b/package/etc/syslog-ng.conf index 84c177b..fd96543 100644 --- a/package/etc/syslog-ng.conf +++ b/package/etc/syslog-ng.conf @@ -1,4 +1,4 @@ -@version:3.24 +@version:3.25 # syslog-ng configuration file. # diff --git a/package/sbin/entrypoint.sh b/package/sbin/entrypoint.sh index 9abad88..aa01301 100755 --- a/package/sbin/entrypoint.sh +++ b/package/sbin/entrypoint.sh @@ -14,5 +14,8 @@ mkdir -p /opt/syslog-ng/etc/conf.d/local/config/ cp --verbose -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context/ cp --verbose -R /opt/syslog-ng/etc/local_config/* /opt/syslog-ng/etc/conf.d/local/config/ +echo syslog-ng checking config +/opt/syslog-ng/sbin/syslog-ng -s >/var/log/syslog-ng.out 2>/var/log/syslog-ng.err + echo syslog-ng starting exec /opt/syslog-ng/sbin/syslog-ng $@ \ No newline at end of file diff --git a/tests/test_common.py b/tests/test_common.py index 9855cc9..eecec2d 100644 --- a/tests/test_common.py +++ b/tests/test_common.py @@ -22,7 +22,7 @@ def test_defaultroute(record_property, setup_wordlist, setup_splunk): sendsingle(message) - st = env.from_string("search index=main host=\"{{ host }}\" sourcetype=\"sc4s:fallback\" | head 2") + st = env.from_string("search index=main host=\"{{ host }}\" sourcetype=\"sc4s:fallback\" PROGRAM=\"test\" | head 2") search = st.render(host=host) resultCount, eventCount = splunk_single(setup_splunk, search) @@ -52,7 +52,7 @@ def test_internal(record_property, setup_wordlist, setup_splunk): assert resultCount == 1 -def test_tag(record_property, setup_wordlist, setup_splunk): +def test_fallback(record_property, setup_wordlist, setup_splunk): host = "{}-{}".format(random.choice(setup_wordlist), random.choice(setup_wordlist)) mt = env.from_string("{{ mark }} {% now 'utc', '%b %d %H:%M:%S' %} testvp-{{ host }} test\n") @@ -60,7 +60,7 @@ def test_tag(record_property, setup_wordlist, setup_splunk): sendsingle(message) - st = env.from_string("search index=main host=\"testvp-{{ host }}\" sourcetype=\"sc4s:fallback\" sc4s_vendor_product=test_test | head 2") + st = env.from_string("search index=main host=\"testvp-{{ host }}\" sourcetype=\"sc4s:fallback\" | head 2") search = st.render(host=host) resultCount, eventCount = splunk_single(setup_splunk, search) @@ -149,3 +149,26 @@ def test_tz_fix_ny(record_property, setup_wordlist, setup_splunk): record_property("message", message) assert resultCount == 1 + + +def test_check_config_version(record_property, setup_wordlist, setup_splunk): + + st = env.from_string("search index=main sourcetype=\"sc4s:events:startup:err\" \"Configuration file format is too old\" ") + search = st.render() + + resultCount, eventCount = splunk_single(setup_splunk, search) + + record_property("resultCount", resultCount) + + assert resultCount == 0 + +def test_check_config_version_multiple(record_property, setup_wordlist, setup_splunk): + + st = env.from_string("search index=main sourcetype=\"sc4s:events:startup:err\" \"you have multiple @version directives\" ") + search = st.render() + + resultCount, eventCount = splunk_single(setup_splunk, search) + + record_property("resultCount", resultCount) + + assert resultCount == 0