diff --git a/.readthedocs.yml b/.readthedocs.yml index 2880d22..aae9256 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -9,4 +9,9 @@ mkdocs: formats: all submodules: - exclude: all \ No newline at end of file + exclude: all + +python: + version: 3.7 + install: + - requirements: mkdocs-requirements.txt \ No newline at end of file diff --git a/docker-compose-ci.yml b/docker-compose-ci.yml index a699498..7566f3d 100644 --- a/docker-compose-ci.yml +++ b/docker-compose-ci.yml @@ -6,10 +6,16 @@ # #You should have received a copy of the CC0 legalcode along with this #work. If not, see . -version: "3.2" +version: "3.7" services: test: build: ./tests + entrypoint: + - /entrypoint.sh + - --workers + - auto + - --tests-per-worker + - auto links: - splunk - sc4s diff --git a/docker-compose.yml b/docker-compose.yml index 082040f..d1e1558 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -6,7 +6,7 @@ # #You should have received a copy of the CC0 legalcode along with this #work. If not, see . -version: "3.2" +version: "3.7" services: test: build: ./tests diff --git a/docs/configuration.md b/docs/configuration.md index 5926646..f7ab084 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -15,17 +15,18 @@ and variables needed to properly configure SC4S for your environment. | Variable | Values | Description | |----------|---------------|-------------| -| SC4S_DEST_SPLUNK_HEC_WORKERS | numeric | Number of destination workers (threads). Set this to the number of HEC endpoints up to a max of 32. | -| SC4S_DEST_SPLUNK_HEC_TLS_VERIFY | yes(default) or no | verify HTTP(s) certificate | +| SC4S_DEST_SPLUNK_HEC_GLOBAL | yes | Send events to Splunk using HEC | | SC4S_DEST_SPLUNK_HEC_CIPHER_SUITE | comma separated list | Open SSL cipher suite list | | SC4S_DEST_SPLUNK_HEC_SSL_VERSION | comma separated list | Open SSL version list | | SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE | path | Custom trusted cert file | +| SC4S_DEST_SPLUNK_HEC_TLS_VERIFY | yes(default) or no | verify HTTP(s) certificate | +| SC4S_DEST_SPLUNK_HEC_WORKERS | numeric | Number of destination workers (threads). Set this to the number of HEC endpoints up to a max of 32. | ## SC4S Disk Buffer Configuration Disk buffers in SC4S are allocated _per destination_. In the future as more destinations are supported, a separate list of variables will be used for each. This is why you see the `DEST_SPLUNK_HEC` in the variable names below. -* NOTE: "Reliable" disk buffering offeres little advantage over "normal" disk buffering, at a significant performance penalty. +* NOTE: "Reliable" disk buffering offers little advantage over "normal" disk buffering, at a significant performance penalty. For this reason, normal disk buffering is recommended. * NOTE: If you add destinations locally in your configuration, pay attention to the _cumulative_ buffer requirements when allocating local disk. @@ -44,11 +45,10 @@ may hide this nuance. ## Archive File Configuration -This feature is designed to support "compliance" archival of all messages. To enable this feature update the Unit file -or docker compose to mount an appropriate host folder to the container folder ``/opt/syslog-ng/var/archive``. -The files will be stored in a folder structure using the naming pattern +This feature is designed to support "compliance" archival of all messages. Instructions for enabling this feature are included +in each "getting started" runtime document. The files will be stored in a folder structure using the naming pattern ``${YEAR}/${MONTH}/${DAY}/${fields.sc4s_vendor_product}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}.log"``. -This pattern will create one file per "vendor_product" per minute with records formatted using syslog-ng's EWMM template. +This pattern will create one file per "vendor_product" per minute with records formatted using syslog-ng's EWMM template. **WARNING POTENTIAL OUTAGE CAUSING CONSEQUENCE** @@ -59,14 +59,14 @@ and/or move them to an archival system to avoid disk space failures. |----------|---------------|-------------| | SC4S_ARCHIVE_GLOBAL | yes or undefined | Enable archive of all vendor_products | | SC4S_ARCHIVE_LISTEN_ | yes(default) or undefined | See sources section of documentation enables selective archival | - + ## Syslog Source Configuration | Variable | Values/Default | Description | |----------|----------------|-------------| | SC4S_LISTEN_DEFAULT_TLS_PORT | undefined or 6514 | Enable a TLS listener on port 6514 | -| SC4S_SOURCE_TLS_OPTIONS | See openssl | List of SSl/TLS protocol versions to support | +| SC4S_SOURCE_TLS_OPTIONS | See openssl | List of SSl/TLS protocol versions to support | | SC4S_SOURCE_TLS_CIPHER_SUITE | See openssl | List of Ciphers to support | | SC4S_SOURCE_TCP_MAX_CONNECTIONS | 2000 | Max number of TCP Connections | | SC4S_SOURCE_TCP_IW_SIZE | 20000000 | Initial Window size | @@ -76,7 +76,8 @@ and/or move them to an archival system to avoid disk space failures. ## Syslog Source TLS Certificate Configuration -* Create a folder ``/opt/sc4s/tls`` +* Create a folder ``/opt/sc4s/tls`` if not already done as part of the "getting started" process. +* Uncomment the appropriate mount line in the unit or yaml file (again, documented in the "getting started" runtime documents). * Save the server private key in PEM format with NO PASSWORD to ``/opt/sc4s/tls/server.key`` * Save the server certificate in PEM format to ``/opt/sc4s/tls/server.pem`` * Add the following line to ``/opt/sc4s/env_file`` @@ -85,12 +86,27 @@ and/or move them to an archival system to avoid disk space failures. SC4S_SOURCE_TLS_ENABLE=yes ``` -## Override index or metadata based on host, ip, or subnet +## Log Path overrides of index or metadata + +In some cases it is appropriate to override the default SC4S index or other Splunk metadata (such as an +source, host, or sourcetype) for a given data source. This is accomplished by the use of a lookup file that identifies these +source exceptions based on the log path used by the incoming message. These log path overrides are documented in the associated +"sources" document. For each data source, you will see a table of the form + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| foo_bar | fb_log | netfw | none | + +In this case the key, `foo_bar`, will be an entry into the `splunk_indexes.csv` file that is populated in `/opt/sc4s/local/context` when SC4S +is run for the first time. The other columns show the default sourcetype and index when not overriden. This file contruct +is best shown with an example. Here is the table for Juniper Netscreen devices, from the "sources" document: + +| key | sourcetype | index | notes | +|------------------------|---------------------|----------------|---------------| +| juniper_netscreen | netscreen:firewall | netfw | none | +| juniper_idp | juniper:idp | netfw | none | -In some cases it is appropriate to re-direct events to an alternate index or append metadata (such as an -indexed field) based on PCI scope, geography, or other criterion. This is accomplished by the use -of a file that uniquely identifies these source exceptions via syslog-ng filters, -which maps to an associated lookup of alternate indexes, sources, or other metadata. +Here is a snippet from the `splunk_indexes.csv` file: * Get the filter and lookup files ```bash @@ -98,38 +114,99 @@ cd /opt/sc4s/default sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context_templates/compliance_meta_by_source.conf sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context_templates/compliance_meta_by_source.csv ``` +#juniper_sslvpn,index,netfw +juniper_netscreen,index,ns_index +#juniper_nsm,index,netfw +``` + +The columns in this file are `key`, `metadata`, and `value`. The `key` entries are +by default "commmented out", which is really a half-truth because CSV files don't allow comments. Therefore, to ensure there +is a match from the log path that references this file, be sure to remove the leading `#`. Once this is done, the following changes can be +made by adding one or more rows to the table and specifying one or more of the following `metadata`/`value` pairs for a given `key`: + + * `index` to specify an alternate `value` for index + * `source` to specify an alternate `value` for source + * `sourcetype` to specify an alternate `value` for sourcetype (be _very_ careful when changing this; only do so if a downstream + TA is _not_ being used, or a custom TA (built by you) is being used.) + +In this case, the `juniper_netscreen` key is "uncommented" (thereby enabling it), and the new index used for that data source will be +`ns_index`. + +In general, for most deployments the index should be the only change needed; the defaults for the others should almost +never be overridden (particularly for "Out of the Box" data sources). Even then, care should be taken when considering an alternate +index, as the defaults for SC4S were chosen with best practices in mind. + +This `csv` file can also be appended when building custom SC4S log paths (filters). Care should be taken during filter design to choose +appropriate index and sourctype defaults, so that admins are not compelled to override them. + + +## Override index or metadata based on host, ip, or subnet + +In other cases it is appropriate to provide the same overrides but based on PCI scope, geography, or other criterion rather than globally. +This is accomplished by the use of a file that uniquely identifies these source exceptions via syslog-ng filters, +which maps to an associated lookup of alternate indexes, sources, or other metadata. In addition, (indexed) fields can also be +added to futher classify the data. + +* The `conf` and `csv` files referenced below will be populated into the `/opt/sc4s/local/context` directory when SC4S is run for the first +time after being set up according to the "getting started" runtime documents. * Edit the file ``compliance_meta_by_source.conf`` to supply uniquely named filters to identify events subject to override. -* Edit the file ``compliance_meta_by_source.csv`` to supply appropriate the field(s) and values. -The three columns in the table are `filter name`, `field name`, and `value`. `field name` obeys the following convention: - * ``fields.fieldname`` where `fieldname` will become the name of an indexed field with the supplied value - * ``.splunk.index`` to specify an alternate value for index - * ``.splunk.source`` to specify an alternate value for source - -* For the Docker/Podman runtimes, update the docker/podman run command in the systemd unit file or the docker-compose to -include volumes mapping the files above. -* In the Unit file, add the following lines to the `ExecStart` command prior to `$SC4SIMAGE` then restart using the command -``sudo systemctl daemon-reload; sudo systemctl restart sc4s`` - -`` -SC4S_UNIT_VP_CSV=-v /opt/sc4s/default/compliance_meta_by_source.csv:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.csv \ -SC4S_UNIT_VP_CONF=-v /opt/sc4s/default/compliance_meta_by_source.conf:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.conf \ -`` - -* For the Docker Swarm runtime, update the docker compose yml to add the following volume mounts to thee sc4s service and -redeploy the updated service using the command: -``docker stack deploy --compose-file docker-compose.yml sc4s`` - -`` - - /opt/sc4s/default/compliance_meta_by_source.csv:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.csv - - /opt/sc4s/default/compliance_meta_by_source.conf:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.conf -`` +* Edit the file ``compliance_meta_by_source.csv`` to supply appropriate field(s) and values. + +The three columns in the `csv` file are `filter name`, `field name`, and `value`. Filter names in the `conf` file must match one or more +corresonding `filter name` rows in the `csv` file. The `field name` column obeys the following convention: + + * `.splunk.index` to specify an alternate `value` for index + * `.splunk.source` to specify an alternate `value` for source + * `.splunk.sourcetype` to specify an alternate `value` for sourcetype (be _very_ careful when changing this; only do so if a downstream + TA is _not_ being used, or a custom TA (built by you) is being used.) + * `fields.fieldname` where `fieldname` will become the name of an indexed field sent to Splunk with the supplied `value` + +This file construct is best shown by an example. Here is a sample ``compliance_meta_by_source.conf`` file: + +``` +@version: 3.24 +filter f_test_test { + host("something-*" type(glob)) or + netmask(192.168.100.1/24) +}; +``` +and the corresponding ``compliance_meta_by_source.csv`` file: + +``` +f_test_test,.splunk.index,"pciindex" +f_test_test,fields.compliance,"pci" +``` + +First off, ensure that the proper version string exists at the top of the `conf` file, and that the filter name(s) in the `conf` file match +one or more rows in the `csv` file. In this case, any incoming message with a hostname starting with `something-` or arriving from a netmask +of `192.168.100.1/24` will match the `f_test_test` filter, and the corresponding entries in the `csv` file will be checked for overrides. +In this case, the new index is `pciindex`, and an indexed field named `compliance` will be sent to Splunk, with it's value set to `pci`. +To add additional overrides, simply add another `filter foo_bar {};` stanza to the `conf` file, and add appropriate entries to the `csv` file +that match the filter name(s) to the overrides you deisre. + +* IMPORTANT: The files above are actual syslog-ng config file snippets that get parsed directly by the underlying syslog-ng +process. Take care that your syntax is correct; for more information on proper syslog-ng syntax, see the syslog-ng +[documentation](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.24/administration-guide/57#TOPIC-1298086). +A syntax error will cause the runtime process to abort in the "preflight" phase at startup. + +Finally, to update your changes for the systemd-based runtimes, restart SC4S using the commands: +``` +sudo systemctl daemon-reload +sudo systemctl restart sc4s +``` + +For the Docker Swarm runtime, redeploy the updated service using the command: +``` +docker stack deploy --compose-file docker-compose.yml sc4s +``` + ## Data Durability - Local Disk Buffer Configuration -SC4S provides capability to minimize the number of lost events if the connection to all the Splunk Indexers goes down. This capability utilizes the disk buffering feature of Syslog-ng. SC4S receives a response from the Splunk HTTP Event Collector (HEC) when a message is received successfully. If a confirmation message from the HEC endpoint is not received (or a “server busy” reply, such as a “503” is sent), the load balancer will try the next HEC endpoint in the pool. If all pool members are exhausted (such as would occur if there were a full network outage to the HEC endpoints), events will queue to the local disk buffer on the SC4S Linux host. SC4S will continue attempting to send the failed events while it buffers all new incoming events to disk. If the disk space allocated to disk buffering fills up then SC4S will stop accepting new events and subsequent events will be lost. Once SC4S gets confirmation that events are again being received by one or more indexers, events will then stream from the buffer using FIFO queueing. The number of events in the disk buffer will reduce as long as the incoming event volume is less than the maximum SC4S (with the disk buffer in the path) can handle. When all events have been emptied from the disk buffer, SC4S will resume streaming events directly to Splunk. +SC4S provides capability to minimize the number of lost events if the connection to all the Splunk Indexers goes down. This capability utilizes the disk buffering feature of Syslog-ng. SC4S receives a response from the Splunk HTTP Event Collector (HEC) when a message is received successfully. If a confirmation message from the HEC endpoint is not received (or a “server busy” reply, such as a “503” is sent), the load balancer will try the next HEC endpoint in the pool. If all pool members are exhausted (such as would occur if there were a full network outage to the HEC endpoints), events will queue to the local disk buffer on the SC4S Linux host. SC4S will continue attempting to send the failed events while it buffers all new incoming events to disk. If the disk space allocated to disk buffering fills up then SC4S will stop accepting new events and subsequent events will be lost. Once SC4S gets confirmation that events are again being received by one or more indexers, events will then stream from the buffer using FIFO queueing. The number of events in the disk buffer will reduce as long as the incoming event volume is less than the maximum SC4S (with the disk buffer in the path) can handle. When all events have been emptied from the disk buffer, SC4S will resume streaming events directly to Splunk. For more detail on the Syslog-ng behavior the documentation can be found here: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/55#TOPIC-1209280 -SC4S has disk buffering enabled by default and it is strongly recommended that you keep it on, however this feature does have a performance cost. +SC4S has disk buffering enabled by default and it is strongly recommended that you keep it on, however this feature does have a performance cost. Without disk buffering enabled SC4S can handle up to 345K EPS (800 bytes/event avg) With “Normal” disk buffering enabled SC4S can handle up to 60K EPS (800 bytes/event avg) -- This is still a lot of data! diff --git a/docs/gettingstarted/byoe-rhel7.md b/docs/gettingstarted/byoe-rhel7.md index 881f949..59baff5 100644 --- a/docs/gettingstarted/byoe-rhel7.md +++ b/docs/gettingstarted/byoe-rhel7.md @@ -71,7 +71,7 @@ sudo cp -R etc/* /opt/syslog-ng/etc/ ```bash sudo curl -o /usr/local/bin/gomplate -sSL https://github.com/hairyhenderson/gomplate/releases/download/v3.5.0/gomplate_linux-amd64 sudo chmod 755 /usr/local/bin/gomplate -gomplate --help +gomplate --version ``` * Install the latest python @@ -127,6 +127,8 @@ mkdir -p /opt/syslog-ng/etc/conf.d/local/config/ cp --verbose -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context/ cp --verbose -R -n /opt/syslog-ng/etc/local_config/* /opt/syslog-ng/etc/conf.d/local/config/ mkdir -p /opt/syslog-ng/var/data/disk-buffer/ +mkdir -p /opt/syslog-ng/var/archive/ +mkdir -p /opt/syslog-ng/tls/ ``` * (Optional) Execute the preconfiguration shell script created above. You may also optionally execute it as part of the unit diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index dfa8b4d..44d2255 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -51,8 +51,10 @@ services: volumes: - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z +# Uncomment the following line if local disk archiving is desired +# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z # Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls +# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. @@ -76,7 +78,14 @@ of events in the event of network failure to the Splunk infrastructure. If you are sure, after stopping SC4S, that all data has been sent, these files can be removed. They will be created again upon restart. -* IMPORTANT: When creating the two directories above, ensure the directories created match the volume mounts specified in the +* Create the subdirectory ``/opt/sc4s/archive``. This will be used as a mount point for local storage of syslog events +(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration" +document for details on the directory structure the archive uses. + +* Create the subdirectory ``/opt/sc4s/tls``. This will be used as a mount point for custom TLS certificates +(if the optional mount is uncommented above). + +* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the `docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup. ## Configure the SC4S environment @@ -107,12 +116,15 @@ Log paths are preconfigured to utilize a convention of index destinations that a * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. ## Configure source filtering by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. @@ -122,8 +134,8 @@ apply to support such sources. To identify sources that require this step, refer In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in -the "Configuration" section. +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. ## Start/Restart SC4S diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index 7ed5afe..0af8b6b 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -59,8 +59,10 @@ services: volumes: - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z +# Uncomment the following line if local disk archiving is desired +# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z # Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls +# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. @@ -82,9 +84,16 @@ of events in the event of network failure to the Splunk infrastructure. * This directory will populate with the disk buffer files upon SC4S startup. If SC4S restarts for any reason, a new set of files will be created in addition to the original ones. _The original ones will not be removed_. If you are sure, after stopping SC4S, that all data has been sent, these files can be removed. They will be created -again upon restart. +again upon restart + +* Create the subdirectory ``/opt/sc4s/archive``. This will be used as a mount point for local storage of syslog events +(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration" +document for details on the directory structure the archive uses. + +* Create the subdirectory ``/opt/sc4s/tls``. This will be used as a mount point for custom TLS certificates +(if the optional mount is uncommented above). -* IMPORTANT: When creating the two directories above, ensure the directories created match the volume mounts specified in the +* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the `docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup. ## Configure the SC4S environment @@ -116,6 +125,8 @@ Log paths are preconfigured to utilize a convention of index destinations that a * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. ## Configure source filtering by source IP or host name @@ -123,6 +134,7 @@ this table that pertain to the individual data source filters that are included Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. @@ -132,8 +144,8 @@ apply to support such sources. To identify sources that require this step, refer In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in -the "Configuration" section. +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. ## Start/Restart SC4S diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 838cdaa..7eae5b0 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -40,8 +40,10 @@ Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf. # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" +# Uncomment the following line if local disk archiving is desired +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" # Uncomment the following line if custom TLS certs are provided -# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" +# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z" TimeoutStartSec=0 Restart=always @@ -56,6 +58,8 @@ ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ + "$SC4S_LOCAL_ARCHIVE_MOUNT" \ + "$SC4S_TLS_DIR" \ --name SC4S --rm \ $SC4S_IMAGE ``` @@ -79,9 +83,16 @@ of events in the event of network failure to the Splunk infrastructure. * This directory will populate with the disk buffer files upon SC4S startup. If SC4S restarts for any reason, a new set of files will be created in addition to the original ones. _The original ones will not be removed_. If you are sure, after stopping SC4S, that all data has been sent, these files can be removed. They will be created -again upon restart. +again upon restart + +* Create the subdirectory ``/opt/sc4s/archive``. This will be used as a mount point for local storage of syslog events +(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration" +document for details on the directory structure the archive uses. + +* Create the subdirectory ``/opt/sc4s/tls``. This will be used as a mount point for custom TLS certificates +(if the optional mount is uncommented above). -* IMPORTANT: When creating the two directories above, ensure the directories created match the volume mounts specified in the +* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the unit file above. Failure to do this will cause SC4S to abort at startup. ## Configure the SC4S environment @@ -112,12 +123,15 @@ Log paths are preconfigured to utilize a convention of index destinations that a * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. ## Configure source filtering by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. @@ -127,8 +141,8 @@ apply to support such sources. To identify sources that require this step, refer In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in -the "Configuration" section. +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. ## Configure SC4S for systemd and start SC4S diff --git a/docs/gettingstarted.md b/docs/gettingstarted/index.md similarity index 94% rename from docs/gettingstarted.md rename to docs/gettingstarted/index.md index e113881..c3822dc 100644 --- a/docs/gettingstarted.md +++ b/docs/gettingstarted/index.md @@ -1,4 +1,6 @@ -# Getting Started +# Before you start + +## Getting Started Splunk Connect for Syslog is a containerized distribution of syslog-ng with a configuration framework designed to simplify getting syslog data into Splunk Enterprise and Splunk Cloud. Our approach is @@ -6,7 +8,7 @@ to provide a runtime-agnostic solution allowing customers to deploy using the co environment of choice. -# Planning Deployment +## Planning Deployment Syslog is an overloaded term that refers to multiple message formats AND optionally a wire protocol for transmission of events between computer systems over UDP, TCP, or TLS. The protocol is designed to minimize @@ -24,13 +26,13 @@ environment. * Plan for appropriately sized hardware (see)[performance.md] -# Implementation +## Implementation -## Splunk Setup +### Splunk Setup -### Create Indexes +#### Create Indexes -SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create them in Splunk when +SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create them in Splunk when using the SC4S defaults. SC4S can be easily customized to use different indexes if desired. * email @@ -42,7 +44,7 @@ using the SC4S defaults. SC4S can be easily customized to use different indexes * netipam * em_metrics (ensure this is created as a metrics index) -### Install Related Splunk Apps +#### Install Related Splunk Apps Install the following: @@ -50,7 +52,7 @@ Install the following: * [Splunk Add-on for Infrastructure](https://splunkbase.splunk.com/app/4217/) * [Splunk Metrics Workspace](https://splunkbase.splunk.com/app/4192/) *NOTE Included in Splunk 7.3.0 and above* -### Configure the Splunk HTTP Event Collector +#### Configure the Splunk HTTP Event Collector - Set up the Splunk HTTP Event Collector with the HEC endpoints behind a load balancer (VIP) configured for https round robin *WITHOUT* sticky session. Alternatively, a list of HEC endpoint URLs can be configured in SC4S (native Syslog-ng load balancing) if no load balancer is in place. In either case, it is @@ -61,14 +63,14 @@ event destinations. or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your Splunk type. -## Implement a Container Runtime and SC4S +### Implement a Container Runtime and SC4S -### Prerequisites +#### Prerequisites * Linux host with Docker (CE 19.x or greater with Docker Swarm) or Podman enabled, depending on runtime choice (below). * A network load balancer (NLB) configured for round robin. Note: Special consideration may be required when more advanced products are used. The optimal configuration of the load balancer will round robin each http POST request (not each connection). -### Select a Container Runtime and SC4S Configuration +#### Select a Container Runtime and SC4S Configuration | Container and Orchestration | Notes | |-----------------------------|-------| @@ -76,9 +78,9 @@ Splunk type. | [Docker CE + systemd single node](gettingstarted/docker-systemd-general.md) | First choice for Debian and Ubuntu; second choice for CentOS for those with limited existing Docker experience | | [Docker CE + Swarm single node](gettingstarted/docker-swarm-general.md) | Option for Debian, Ubuntu, CentOS, and Desktop Docker desiring Docker Compose or Swarm orchestration | | [Docker CE + Swarm single node RHEL 7.7](gettingstarted/docker-swarm-rhel7.md) | Option for RedHat 7.7 desiring Docker Compose or Swarm orchestration | -| [Bring your own Envionment](gettingstarted/byoe-rhel7.md) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers | +| [Bring your own Envionment](gettingstarted/byoe-rhel7.md) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers | -## Offline Container Installation +### Offline Container Installation Follow these instructions to "stage" SC4S by downloading the container so that it can be loaded "out of band" on a host machine, such as an airgapped system, without internet connectivity. @@ -116,7 +118,7 @@ attempt to obtain the container image via the internet. Environment="SC4S_IMAGE=sc4slocal:latest" ``` -# Scale out +## Scale out Additional hosts can be deployed for syslog collection from additional network zones and locations: diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 1d0ddc3..18f8f82 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -10,8 +10,8 @@ Refer to [Installation](https://podman.io/getting-started/installation) ```ini [Unit] Description=SC4S Container -After=network.service -Requires=network.service +Wants=network.target network-online.target +After=network.target network-online.target [Service] Environment="SC4S_IMAGE=splunk/scs:latest" @@ -22,8 +22,10 @@ Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf. # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" +# Uncomment the following line if local disk archiving is desired +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" # Uncomment the following line if custom TLS certs are provided -# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" +# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z" TimeoutStartSec=0 Restart=always @@ -38,6 +40,8 @@ ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ + "$SC4S_LOCAL_ARCHIVE_MOUNT" \ + "$SC4S_TLS_DIR" \ --name SC4S --rm \ $SC4S_IMAGE ``` @@ -61,9 +65,16 @@ of events in the event of network failure to the Splunk infrastructure. * This directory will populate with the disk buffer files upon SC4S startup. If SC4S restarts for any reason, a new set of files will be created in addition to the original ones. _The original ones will not be removed_. If you are sure, after stopping SC4S, that all data has been sent, these files can be removed. They will be created -again upon restart. +again upon restart + +* Create the subdirectory ``/opt/sc4s/archive``. This will be used as a mount point for local storage of syslog events +(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration" +document for details on the directory structure the archive uses. + +* Create the subdirectory ``/opt/sc4s/tls``. This will be used as a mount point for custom TLS certificates +(if the optional mount is uncommented above). -* IMPORTANT: When creating the two directories above, ensure the directories created match the volume mounts specified in the +* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the unit file above. Failure to do this will cause SC4S to abort at startup. ## Configure the sc4s environment @@ -94,12 +105,15 @@ Log paths are preconfigured to utilize a convention of index destinations that a * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. ## Configure source filtering by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. @@ -109,8 +123,8 @@ apply to support such sources. To identify sources that require this step, refer In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in -the "Configuration" section. +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. ## Configure SC4S for systemd and start SC4S diff --git a/docs/logo.png b/docs/logo.png new file mode 100644 index 0000000..fe6f098 Binary files /dev/null and b/docs/logo.png differ diff --git a/docs/sources.md b/docs/sources.md deleted file mode 100644 index 22c3aab..0000000 --- a/docs/sources.md +++ /dev/null @@ -1,1090 +0,0 @@ -# Introduction -When using Splunk Connect for Syslog to onboard a data source, the SC4S filter performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source. - -SC4S "unique" filters are based either on the port upon which events arrive or the hostname/CIDR block from which they are sent. The "soup" filters run for events that arrive on port 514 (default for syslog), and contain regex and other syslog-specific parsers to identify events from a specific source, apply the correct sourcetype, and set other metadata. Data sources which generate events that are not unique enough to accurately identify with soup filters _must_ employ the "unique" filters (port/hostname/CIDR block) instead -- the soup filters are unavailable for these sources. - -If SC4S receives an event on port 514 which has no soup filter, that event will be given a "fallback" sourcetype. If you see events in Splunk with the fallback sourcetype, then you should figure out what source the events are from and determine why these events are not being sourcetyped correctly. The most common reason for events categorized as "fallback" is the lack of a SC4S filter for that source, and in some cases a misconfigured relay which alters the integrity of the message format. In most cases this means a new SC4S filter must be developed. In this situation you can either build a filter or file an issue with the community to request help. - -# Vendor - Checkpoint - -## Product - Log Exporter (Splunk) - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/4293/ | -| Product Manual | https://sc1.checkpoint.com/documents/App_for_Splunk/html_frameset.htm | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| cp_log | None | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| checkpoint_splunk | cp_log | netfw | none | - -### Source and Index Configuration - -Checkpoint Software blades with CIM mapping have been sub-grouped into sources -to allow routing to appropriate indexes. All other source meta data is left at default - -| key | source | index | notes | -|----------------|----------------|----------------|----------------| -| checkpoint_splunk_dlp | dlp | netdlp | none | -| checkpoint_splunk_email | email | email | none | -| checkpoint_splunk_firewall | firewall | netfw | none | -| checkpoint_splunk_sessions | sessions | netops | none | -| checkpoint_splunk_web | web | netproxy | none | - -### Filter type - -MSG Parse: This filter parses message content - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Follow vendor configuration steps per Product Manual above - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_CHECKPOINT_SPLUNK_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the port number defined | - -### Verification - -Use the following search to validate events are present - -``` -index= sourcetype=cp_log -``` - -Verify timestamp, and host values match as expected -# Vendor - Cisco - -## Product - ASA (Pre Firepower) - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/1620/ | -| Product Manual | https://www.cisco.com/c/en/us/td/docs/security/asa/asa82/configuration/guide/config/monitor_syslog.html | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| cisco:asa | None | -| cisco:pix | Not supported | -| cisco:fwsm | Not supported | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| cisco_asa | cisco:asa | netfw | none | - - -### Filter type - -MSG Parse: This filter parses message content - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Follow vendor configuration steps per Product Manual above ensure: - * Log Level is 6 "Informational" - * Protocol is TCP/IP - * permit-hostdown is on - * device-id is hostname and included - * timestamp is included - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_CISCO_ASA_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format | -| SC4S_LISTEN_CISCO_ASA_LEGACY_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC3164 format | - -### Verification - -Use the following search to validate events are present - -``` -index= sourcetype=cisco:asa -``` - -Verify timestamp, and host values match as expected - -## Product - IOS and NX-OS based equipment - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/1467/ | -| IOS Manual | https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst2960/software/release/12-2_55_se/configuration/guide/scg_2960/swlog.html | -| NX-OS Manual | https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide/sm_5syslog.html| -| Cisco ACI | https://community.cisco.com/legacyfs/online/attachments/document/technote-aci-syslog_external-v1.pdf | -| Cisco WLC & AP | https://www.cisco.com/c/en/us/support/docs/wireless/4100-series-wireless-lan-controllers/107252-WLC-Syslog-Server.html#anc8 | - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| cisco:ios | This source type is also used for NX-OS, ACI and WLC product lines | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| cisco_ios | cisco:ios | netops | none | -| cisco_nx_os | cisco:ios | netops | none | - -### Filter type - -* Cisco IOS products can be identified by message parsing alone -* Cisco NX OS, WLC, and ACI products must be identified by host or ip assignment update the filter `f_cisco_nx_os` as required - - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* IOS Follow vendor configuration steps per Product Manual above ensure: - * Ensure a reliable NTP server is set and synced - * Log Level is 6 "Informational" - * Protocol is TCP/IP - * permit-hostdown is on - * device-id is hostname and included - * timestamp is included -* NX-OS Follow vendor configuration steps per Product Manual above ensure: - * Ensure a reliable NTP server is set and synced - * Log Level is 6 "Informational" user may select alternate levels by module based on use cases - * Protocol is TCP/IP - * device-id is hostname and included - * timestamp is included and milisecond accuracy selected -* ACI Logging configuration of the ACI product often varies by use case. - * Ensure NTP sync is configured and active - * Ensure proper host names are configured -* WLC - * Ensure NTP sync is configured and active - * Ensure proper host names are configured - * For security use cases per AP logging is required - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_CISCO_IOS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | -| SC4S_LISTEN_CISCO_NX_OS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -Use the following search to validate events are present, for NX-OS, WLC and ACI products ensure each host filter condition is verified - -``` -index= sourcetype=cisco:ios | stats count by host -``` - -## Product - ISE - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/1915/ | -| Product Manual | https://www.cisco.com/c/en/us/td/docs/security/ise/2-6/Cisco_ISE_Syslogs/Cisco_ISE_Syslogs/Cisco_ISE_Syslogs_chapter_00.html | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| cisco:ise:syslog | Aggregation used | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| cisco_ise | cisco:ise:syslog | netauth | None | - - -### Filter type - -PATTERN MATCH - -### Setup and Configuration - -* No special steps required - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_CISCO_ISE_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format | -| SC4S_LISTEN_CISCO_ISE_UDP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format | - -### Verification - -Use the following search to validate events are present - -``` -index= sourcetype=cisco:ise:syslog -``` - -Verify timestamp, and host values match as expected - -## Product - Meraki Product Line MR, MS, MX, MV - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/3018/ | -| Product Manual | https://documentation.meraki.com/zGeneral_Administration/Monitoring_and_Reporting/Syslog_Server_Overview_and_Configuration | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| merkai | None | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| cisco_meraki | meraki | netfw | The current TA does not sub sourcetype or utilize source preventing segmenation into more appropriate indexes | - - -### Filter type - -IP, Netmask, Host or Port - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Follow vendor configuration steps per Product Manual above - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_CISCO_MERAKI_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format | -| SC4S_LISTEN_CISCO_MERAKI_UDP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format | - -### Verification - -Use the following search to validate events are present - -``` -index= sourcetype=merkai -``` - -Verify timestamp, and host values match as expected - -# Vendor - Forcepoint - -## Product - Webprotect (Websense) - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/2966/ | -| Product Manual | http://www.websense.com/content/support/library/web/v85/siem/siem.pdf | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| websense:cg:kv | None | - - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| forcepoint_webprotect | websense:cg:kv | netproxy | none | - -### Filter type - -MSG Parse: This filter parses message content - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Refer to the admin manual for specific details of configuration to send Reliable syslog using RFC 3195 format, a typical logging configuration will include the following features. - - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_FORCEPOINT_WEBPROTECT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -An active proxy will generate frequent events, in addition WebProtect has the ability to test logging functionality using a built in command - - -``` -index= sourcetype=websense:cg:kv -``` - -# Vendor - Fortinet - -## Product - Fortigate - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/2846/ | -| Product Manual | https://docs.fortinet.com/product/fortigate/6.2 | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| fgt_log | The catch all sourcetype is not used | -| fgt_traffic | None | -| fgt_utm | None | -| fgt_event | None - - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| fortinet_fortios_traffic | fgt_traffic | netops | none | -| fortinet_fortios_utm | fgt_utm | netids | none | -| fortinet_fortios_event | fgt_event | netops | none | -| fortinet_fortios_log | fgt_log | netops | none | - - -### Filter type - -MSG Parse: This filter parses message content - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Refer to the admin manual for specific details of configuration to send Reliable syslog using RFC 3195 format, a typical logging configuration will include the following features. - -``` -config log memory filter - -set forward-traffic enable - -set local-traffic enable - -set sniffer-traffic disable - -set anomaly enable - -set voip disable - -set multicast-traffic enable - -set dns enable - -end - -config system global - -set cli-audit-log enable - -end - -config log setting - -set neighbor-event enable - -end - -``` - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_FORTINET_FORTIOS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -An active firewall will generate frequent events, in addition fortigate has the ability to test logging functionality using a built in command - -``` -diag log test -``` - -Verify timestamp, and host values match as expected - -``` -index= (sourcetype=fgt_log OR sourcetype=fgt_traffic OR sourcetype=fgt_utm) -``` - -### UTM Message type - -![FortiGate UTM message](FortiGate_utm.png) - -### Traffic Message Type - -![FortiGate Traffic message](FortiGate_traffic.png) - -###Event Message Type -![FortiGate Event message](FortiGate_event.png) - -Verify timestamp, and host values match as expected - -# Vendor - Imperva - -## Product - Incapsula - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on CEF | https://bitbucket.org/SPLServices/ta-cef-for-splunk/downloads/ | -| Splunk Add-on Source Specific | https://bitbucket.org/SPLServices/ta-cef-imperva-incapsula/downloads/ | -| Product Manual | https://docs.imperva.com/bundle/cloud-application-security/page/more/log-configuration.htm | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| cef | Common sourcetype | - -### Source - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| Imperva:Incapsula | Common sourcetype | - -### Index Configuration - -| key | source | index | notes | -|----------------|----------------|----------------|----------------| -| cef_Incapsula_SIEMintegration | Imperva:Incapsula | netwaf | none | - -### Filter type - -MSG Parse: This filter parses message content - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -An active site will generate frequent events use the following search to check for new events - -Verify timestamp, and host values match as expected - -``` -index= (sourcetype=cef source="Imperva:Incapsula") -``` - -# Vendor - Juniper - -## Product - Juniper JunOS - -| Ref | Link | -|-------------------|-------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/2847/ | -| JunOS TechLibrary | https://www.juniper.net/documentation/en_US/junos/topics/example/syslog-messages-configuring-qfx-series.html | - -### Sourcetypes - -| sourcetype | notes | -|--------------------------|------------------------------------------------------------------| -| juniper:junos:firewall | None | -| juniper:junos:idp | None | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------------------|------------------------|----------------|---------------| -| juniper_junos_flow | juniper:junos:firewall | netfw | none | -| juniper_junos_idp | juniper:junos:idp | netids | none | -| juniper_junos_utm | juniper:junos:firewall | netfw | none | - -### Filter type - -* MSG Parse: This filter parses message content - - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index as required. -* Follow vendor configuration steps per referenced Product Manual - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_JUNIPER_JUNOS_LEGACY_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined using legacy 3164 format| -| SC4S_LISTEN_JUNIPER_JUNOS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined using 5424 format | - -### Verification - -Use the following search to validate events are present; for Juniper JunOS ensure each host filter condition is verified - -``` -index= sourcetype=juniper:junos:firewall | stats count by host -index= sourcetype=juniper:junos:idp | stats count by host -``` - -Verify timestamp, and host values match as expected - -## Product - Juniper NSM - -| Ref | Link | -|----------------|-------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/2847/ | -| NSM syslog KB | http://kb.juniper.net/InfoCenter/index?page=content&id=KB11810 | - -### Sourcetypes - -| sourcetype | notes | -|------------------|-----------------------------------------------------------------------| -| juniper:nsm | None | -| juniper:nsm:idp | None | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|------------------------|---------------------|----------------|---------------| -| juniper_nsm | juniper:nsm | netfw | none | -| juniper_nsm_idp | juniper:nsm:idp | netids | none | - -### Filter type - -* Juniper NSM products must be identified by host or ip assignment. Update the filter `f_juniper_nsm` or `f_juniper_nsm_idp` as required - - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index as required. -* Follow vendor configuration steps per Product Manual - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_JUNIPER_NSM_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | -| SC4S_LISTEN_JUNIPER_NSM_UDP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -Use the following search to validate events are present; for Juniper NSM ensure each host filter condition is verified - -``` -index= sourcetype=juniper:nsm | stats count by host -index= sourcetype=juniper:nsm:idp | stats count by host -``` - -Verify timestamp, and host values match as expected - -## Product - Juniper Netscreen - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/2847/ | -| Netscreen Manual | http://kb.juniper.net/InfoCenter/index?page=content&id=KB4759 | - -### Sourcetypes - -| sourcetype | notes | -|-------------------------|------------------------------------------------------------------------------------------------| -| netscreen:firewall | None | -| juniper:idp | None | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|------------------------|---------------------|----------------|---------------| -| juniper_netscreen | netscreen:firewall | netfw | none | -| juniper_idp | juniper:idp | netfw | none | - -### Filter type - -* Juniper Netscreen products must be identified by host or ip assignment. Update the filter `f_juniper_netscreen` or `f_juniper_idp` as required - - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index as required. -* Follow vendor configuration steps per Product Manual - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | -| SC4S_LISTEN_JUNIPER_NETSCREEN_UDP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -Use the following search to validate events are present; for Juniper Netscreen products ensure each host filter condition is verified - -``` -index= sourcetype=netscreen:firewall | stats count by host -index= sourcetype=juniper:idp | stats count by host -``` - -Verify timestamp, and host values match as expected - -## Product - Juniper SSLVPN - -| Ref | Link | -|------------------|-------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/2847/ | -| Pulse Secure KB | https://kb.pulsesecure.net/articles/Pulse_Secure_Article/KB22227 | - -### Sourcetypes - -| sourcetype | notes | -|------------------|-----------------------------------------------------------------------| -| juniper:sslvpn | None | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|------------------------|---------------------|----------------|---------------| -| juniper_sslvpn | juniper:sslvpn | netfw | none | - -### Filter type - -* MSG Parse: This filter parses message content - - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index as required. -* Follow vendor configuration steps per referenced Product Manual - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_JUNIPER_JUNOS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -Use the following search to validate events are present; for Juniper SSL VPN ensure each host filter condition is verified - -``` -index= sourcetype=juniper:sslvpn | stats count by host -``` - -Verify timestamp, and host values match as expected - -# Vendor - Microfocus ArcSight - -## Product - Internal Agent Events - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on CEF | https://bitbucket.org/SPLServices/ta-cef-for-splunk/downloads/ | -| Product Manual | https://docs.imperva.com/bundle/cloud-application-security/page/more/log-configuration.htm | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| cef | Common sourcetype | - -### Source - -| source | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| ArcSight:ArcSight | Internal logs | - -### Index Configuration - -| key | source | index | notes | -|----------------|----------------|----------------|----------------| -| cef_ArcSight_ArcSight | ArcSight:ArcSight | main | none | - -### Filter type - -MSG Parse: This filter parses message content - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -An active site will generate frequent events use the following search to check for new events - -Verify timestamp, and host values match as expected - -``` -index= (sourcetype=cef source="ArcSight:ArcSight") -``` - -## Product - Microsoft Windows - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on CEF | https://bitbucket.org/SPLServices/ta-cef-for-splunk/downloads/ | -| Splunk Add-on CEF | https://bitbucket.org/SPLServices/ta-cef-microsoft-windows-for-splunk/downloads/ | -| Product Manual | https://docs.imperva.com/bundle/cloud-application-security/page/more/log-configuration.htm | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| cef | Common sourcetype | - -### Source - -| source | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| CEFEventLog:System or Application Event | Windows Application and System Event Logs | -| CEFEventLog:Microsoft Windows | Windows Security Event Logs | - -### Index Configuration - -| key | source | index | notes | -|----------------|----------------|----------------|----------------| -| cef_Microsoft_System or Application Event | CEFEventLog:System or Application Event | oswin | none | -| cef_Microsoft_Microsoft Windows | CEFEventLog:Microsoft Windows | oswinsec | none | - -### Filter type - -MSG Parse: This filter parses message content - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -An active site will generate frequent events use the following search to check for new events - -Verify timestamp, and host values match as expected - -``` -index= (sourcetype=cef (source="CEFEventLog:Microsoft Windows" OR source="CEFEventLog:System or Application Event")) -``` - -# Vendor - PaloAlto - -## Product - NGFW - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/2757/ | -| Product Manual | https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/configure-syslog-monitoring.html | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| pan:log | None | -| pan:traffic | None | -| pan:threat | None | -| pan:system | None | -| pan:config | None | -| pan:hipwatch | None | -| pan:correlation | None | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| pan_log | pan:log | netops | none | -| pan_traffic | pan:traffic | netfw | none | -| pan_threat | pan:threat | netproxy | none | -| pan_system | pan:system | netops | none | -| pan_config | pan:config | netops | none | -| pan_hipwatch | pan:hipwatch | netops | none | -| pan_correlation | pan:correlation | netops | none | - -### Filter type - -MSG Parse: This filter parses message content - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Refer to the admin manual for specific details of configuration - * Select TCP or SSL transport option - * Select IETF Format - * Ensure the format of the event is not customized - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_PALOALTO_PANOS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -An active firewall will generate frequent events. Use the following search to validate events are present per source device - -``` -index= sourcetype=pan:*| stats count by host -``` - -# Vendor - Proofpoint - -## Product - Proofpoint Protection Server - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/3080/ | -| Product Manual | https://proofpointcommunities.force.com/community/s/article/Remote-Syslog-Forwarding | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| pps_filter_log | | -| pps_mail_log | This sourcetype will conflict with sendmail itself, so will require that the PPS send syslog on a dedicated port or be uniquely identifiable with a hostname glob or CIDR block if this sourcetype is desired for PPS. | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| proofpoint_pps_filter | pps_filter_log | email | none | -| proofpoint_pps_sendmail | pps_mail_log | email | none | - - -### Filter type - -MSG Parse: This filter parses message content -* NOTE: This filter will simply parse the syslog message itself, and will _not_ perform the (required) re-assembly of related -messages to create meaningful final output. This will require follow-on processing in Splunk. - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Follow vendor configuration steps per referenced Product Manual - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_PROOFPOINT_PPS_FILTER_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined. If this option is used to ensure PPS sendmail sourcetype uniqueness (see above), set the same port number for this and the SC4S_PROOFPOINT_PPS_MAIL_TCP_PORT variable immediately below.| -| SC4S_PROOFPOINT_PPS_MAIL_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined. If this option is used to ensure PPS sendmail sourcetype uniqueness (see above), set the same port number for this and the SC4S_PROOFPOINT_PPS_FILTER_TCP_PORT variable immediately above. | - -### Verification - -One or two sourcetypes are included in Proofpoint PPS logs. The search below will surface both of them: - -``` -index= sourcetype=pps_*_log | stats count by host -``` - -# Vendor - Symantec - -## Product - ProxySG/ASG (Bluecoat) - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/2758/ | -| Product Manual | https://support.symantec.com/us/en/article.tech242216.html | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| bluecoat:proxysg:access:kv | Requires version TA 3.6 | - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| bluecoat_proxy | bluecoat:proxysg:access:kv | netops | none | - - -### Filter type - -MSG Parse: This filter parses message content - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration - * Select TCP or SSL transport option - * Ensure the format of the event is customized per Splunk documentation - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_SYMANTEC_PROXY_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -An active proxy will generate frequent events. Use the following search to validate events are present per source device - -``` -index= sourcetype=bluecoat:proxysg:access:kv | stats count by host -``` - - -# Vendor - Ubiquiti - Unifi - -All Ubiquity Unfi firewalls, switches, and access points share a common syslog configuration via the NMS. - - -* Login to NMS -* Navigate to settings -* Navigate to Site -* Enable Remote syslog server -* Enter hostname and port -* Update ``vi /opt/sc4s/local/context/vendor_product_by_source.conf `` update the host or ip mask for ``f_ubiquiti_unifi_fw`` to identify USG firewalls - -## Product - Unifi Switch and Access Points - -Unifi devices are managed using the Network Management Controller - - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/4107/ | -| Product Manual | https://https://help.ubnt.com/ | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| ubnt | Used when no sub source type is required by add on | -| ubnt:fw | USG events | -| ubnt:threat | USG IDS events | -| ubnt:switch | Unifi Switches | -| ubnt:wireless | Access Point logs | - - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| ubiquiti_unifi | ubnt | netops | none | -| ubiquiti_unifi_fw | ubnt:fw | netfw | none | -| ubiquiti_unifi_link | ubnt:link | netops | none | -| ubiquiti_unifi_sudo | ubnt:sudo | netops | none | -| ubiquiti_unifi_switch | ubnt:switch | netops | none | -| ubiquiti_unifi_threat | ubnt:threat | netids | none | -| ubiquiti_unifi_wireless | ubnt:wireless | netops | none | - - -### Filter type - -MSG Parse: This filter parses message content - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration - * Select TCP or SSL transport option - * Ensure the format of the event is customized per Splunk documentation - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_ZSCALER_NSS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -An active proxy will generate frequent events. Use the following search to validate events are present per source device - -``` -index= sourcetype=zscalernss-* | stats count by host -``` - - -# Vendor - Zscaler - -## Product - All Products - -The ZScaler product manual includes and extensive section of configuration for multiple Splunk TCP input ports around page -26. When using SC4S these ports are not required and should not be used. Simply configure all outputs from the NSS to utilize -the IP or host name of the SC4S instance and port 514 - - -| Ref | Link | -|----------------|---------------------------------------------------------------------------------------------------------| -| Splunk Add-on | https://splunkbase.splunk.com/app/3865/ | -| Product Manual | https://community.zscaler.com/t/zscaler-splunk-app-design-and-installation-documentation/4728 | - - -### Sourcetypes - -| sourcetype | notes | -|----------------|---------------------------------------------------------------------------------------------------------| -| zscalernss-alerts | Requires format customization add ``\tvendor=Zscaler\tproduct=alerts`` immediately prior to the ``\n`` in the NSS Alert Web format. See Zscaler manual for more info. | -| zscalernss-dns | Requires format customization add ``\tvendor=Zscaler\tproduct=dns`` immediately prior to the ``\n`` in the NSS DNS format. See Zscaler manual for more info. | -| zscalernss-web | None | -| zscalernss-zpa-app | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. | -| zscalernss-zpa-auth | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa_auth`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. | -| zscalernss-zpa-connector | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa_auth_connector`` immediately prior to the ``\n`` in the LSS Connector format. See Zscaler manual for more info. | -| zscalernss-fw | Requires format customization add ``\tvendor=Zscaler\tproduct=fw`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. | - - -### Sourcetype and Index Configuration - -| key | sourcetype | index | notes | -|----------------|----------------|----------------|----------------| -| zscalernss_alerts | zscalernss-alerts | main | none | -| zscalernss_dns | zscalernss-dns | netdns | none | -| zscalernss_fw | zscalernss-fw | netfw | none | -| zscalernss_web | zscalernss-web | netproxy | none | -| zscalernss-zpa-app | zscalernss_zpa-app | netids | none | -| zscalernss-zpa-auth | zscalernss_zpa_auth | netauth | none | -| zscalernss-zpa-connector | zscalernss_zpa_connector | netops | none | - - -### Filter type - -MSG Parse: This filter parses message content - -### Setup and Configuration - -* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. -* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. -* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration - * Select TCP or SSL transport option - * Ensure the format of the event is customized per Splunk documentation - -### Options - -| Variable | default | description | -|----------------|----------------|----------------| -| SC4S_LISTEN_ZSCALER_NSS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | - -### Verification - -An active proxy will generate frequent events. Use the following search to validate events are present per source device - -``` -index= sourcetype=zscalernss-* | stats count by host -``` diff --git a/docs/sources/Checkpoint/index.md b/docs/sources/Checkpoint/index.md new file mode 100644 index 0000000..ee2c590 --- /dev/null +++ b/docs/sources/Checkpoint/index.md @@ -0,0 +1,63 @@ +# Vendor - Checkpoint + +## Product - Log Exporter (Splunk) + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/4293/ | +| Product Manual | https://sc1.checkpoint.com/documents/App_for_Splunk/html_frameset.htm | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| cp_log | None | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| checkpoint_splunk | cp_log | netfw | none | + +### Source and Index Configuration + +Checkpoint Software blades with CIM mapping have been sub-grouped into sources +to allow routing to appropriate indexes. All other source meta data is left at default + +| key | source | index | notes | +|----------------|----------------|----------------|----------------| +| checkpoint_splunk_dlp | dlp | netdlp | none | +| checkpoint_splunk_email | email | email | none | +| checkpoint_splunk_firewall | firewall | netfw | none | +| checkpoint_splunk_sessions | sessions | netops | none | +| checkpoint_splunk_web | web | netproxy | none | + +### Filter type + +MSG Parse: This filter parses message content + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Follow vendor configuration steps per Product Manual above + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_CHECKPOINT_SPLUNK_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the port number defined | +| SC4S_LISTEN_CHECKPOINT_SPLUNK_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the port number defined | +| SC4S_ARCHIVE_CHECKPOINT_SPLUNK | no | Enable archive to disk for this specific source | +| SC4S_DEST_CHECKPOINT_SPLUNK_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +Use the following search to validate events are present + +``` +index= sourcetype=cp_log +``` + +Verify timestamp, and host values match as expected diff --git a/docs/sources/Cisco/index.md b/docs/sources/Cisco/index.md new file mode 100644 index 0000000..9d6cad4 --- /dev/null +++ b/docs/sources/Cisco/index.md @@ -0,0 +1,236 @@ +# Vendor - Cisco + +## Product - ASA (Pre Firepower) + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/1620/ | +| Product Manual | https://www.cisco.com/c/en/us/td/docs/security/asa/asa82/configuration/guide/config/monitor_syslog.html | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| cisco:asa | None | +| cisco:pix | Not supported | +| cisco:fwsm | Not supported | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| cisco_asa | cisco:asa | netfw | none | + + +### Filter type + +MSG Parse: This filter parses message content + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Follow vendor configuration steps per Product Manual above ensure: + * Log Level is 6 "Informational" + * Protocol is TCP/IP + * permit-hostdown is on + * device-id is hostname and included + * timestamp is included + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_CISCO_ASA_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format | +| SC4S_LISTEN_CISCO_ASA_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined expecting RFC5424 format | +| SC4S_ARCHIVE_CISCO_ASA | no | Enable archive to disk for this specific source | +| SC4S_DEST_CISCO_ASA_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | +| SC4S_LISTEN_CISCO_ASA_LEGACY_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC3164 format | +| SC4S_LISTEN_CISCO_ASA_LEGACY_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined expecting RFC3164 format | +| SC4S_ARCHIVE_CISCO_ASA_LEGACY | no | Enable archive to disk for this specific source | +| SC4S_DEST_CISCO_ASA_LEGACY_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +Use the following search to validate events are present + +``` +index= sourcetype=cisco:asa +``` + +Verify timestamp, and host values match as expected + +## Product - IOS and NX-OS based equipment + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/1467/ | +| IOS Manual | https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst2960/software/release/12-2_55_se/configuration/guide/scg_2960/swlog.html | +| NX-OS Manual | https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide/sm_5syslog.html| +| Cisco ACI | https://community.cisco.com/legacyfs/online/attachments/document/technote-aci-syslog_external-v1.pdf | +| Cisco WLC & AP | https://www.cisco.com/c/en/us/support/docs/wireless/4100-series-wireless-lan-controllers/107252-WLC-Syslog-Server.html#anc8 | + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| cisco:ios | This source type is also used for NX-OS, ACI and WLC product lines | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| cisco_ios | cisco:ios | netops | none | +| cisco_nx_os | cisco:ios | netops | none | + +### Filter type + +* Cisco IOS products can be identified by message parsing alone +* Cisco NX OS, WLC, and ACI products must be identified by host or ip assignment update the filter `f_cisco_nx_os` as required + + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* IOS Follow vendor configuration steps per Product Manual above ensure: + * Ensure a reliable NTP server is set and synced + * Log Level is 6 "Informational" + * Protocol is TCP/IP + * permit-hostdown is on + * device-id is hostname and included + * timestamp is included +* NX-OS Follow vendor configuration steps per Product Manual above ensure: + * Ensure a reliable NTP server is set and synced + * Log Level is 6 "Informational" user may select alternate levels by module based on use cases + * Protocol is TCP/IP + * device-id is hostname and included + * timestamp is included and milisecond accuracy selected +* ACI Logging configuration of the ACI product often varies by use case. + * Ensure NTP sync is configured and active + * Ensure proper host names are configured +* WLC + * Ensure NTP sync is configured and active + * Ensure proper host names are configured + * For security use cases per AP logging is required + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_CISCO_IOS_UDP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_CISCO_IOS_TCP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_CISCO_IOS | no | Enable archive to disk for this specific source | +| SC4S_DEST_CISCO_IOS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | +| SC4S_LISTEN_CISCO_NX_OS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_CISCO_NX_OS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_CISCO_NXOS | no | Enable archive to disk for this specific source | +| SC4S_DEST_CISCO_NXOS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +Use the following search to validate events are present, for NX-OS, WLC and ACI products ensure each host filter condition is verified + +``` +index= sourcetype=cisco:ios | stats count by host +``` + +## Product - ISE + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/1915/ | +| Product Manual | https://www.cisco.com/c/en/us/td/docs/security/ise/2-6/Cisco_ISE_Syslogs/Cisco_ISE_Syslogs/Cisco_ISE_Syslogs_chapter_00.html | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| cisco:ise:syslog | Aggregation used | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| cisco_ise | cisco:ise:syslog | netauth | None | + + +### Filter type + +PATTERN MATCH + +### Setup and Configuration + +* No special steps required + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_CISCO_ISE_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format | +| SC4S_LISTEN_CISCO_ISE_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined expecting RFC5424 format | +| SC4S_ARCHIVE_CISCO_ISE | no | Enable archive to disk for this specific source | +| SC4S_DEST_CISCO_ISE_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +Use the following search to validate events are present + +``` +index= sourcetype=cisco:ise:syslog +``` + +Verify timestamp, and host values match as expected + +## Product - Meraki Product Line MR, MS, MX, MV + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/3018/ | +| Product Manual | https://documentation.meraki.com/zGeneral_Administration/Monitoring_and_Reporting/Syslog_Server_Overview_and_Configuration | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| merkai | None | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| cisco_meraki | meraki | netfw | The current TA does not sub sourcetype or utilize source preventing segmenation into more appropriate indexes | + + +### Filter type + +IP, Netmask, Host or Port + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Follow vendor configuration steps per Product Manual above + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_CISCO_MERAKI_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format | +| SC4S_LISTEN_CISCO_MERAKI_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined expecting RFC5424 format | +| SC4S_ARCHIVE_CISCO_MERAKI | no | Enable archive to disk for this specific source | +| SC4S_DEST_CISCO_MERAKI_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +Use the following search to validate events are present + +``` +index= sourcetype=merkai +``` + +Verify timestamp, and host values match as expected + diff --git a/docs/sources/Forcepoint/index.md b/docs/sources/Forcepoint/index.md new file mode 100644 index 0000000..e5fdeff --- /dev/null +++ b/docs/sources/Forcepoint/index.md @@ -0,0 +1,52 @@ +# Vendor - Forcepoint + +## Product - Webprotect (Websense) + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2966/ | +| Product Manual | http://www.websense.com/content/support/library/web/v85/siem/siem.pdf | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| websense:cg:kv | None | + + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| forcepoint_webprotect | websense:cg:kv | netproxy | none | + +### Filter type + +MSG Parse: This filter parses message content + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Refer to the admin manual for specific details of configuration to send Reliable syslog using RFC 3195 format, a typical logging configuration will include the following features. + + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_FORCEPOINT_WEBPROTECT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_FORCEPOINT_WEBPROTECT_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_FORCEPOINT_WEBPROTECT | no | Enable archive to disk for this specific source | +| SC4S_DEST_FORCEPOINT_WEBPROTECT_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +An active proxy will generate frequent events, in addition WebProtect has the ability to test logging functionality using a built in command + + +``` +index= sourcetype=websense:cg:kv +``` + diff --git a/docs/FortiGate_event.png b/docs/sources/Fortinet/FortiGate_event.png similarity index 100% rename from docs/FortiGate_event.png rename to docs/sources/Fortinet/FortiGate_event.png diff --git a/docs/FortiGate_traffic.png b/docs/sources/Fortinet/FortiGate_traffic.png similarity index 100% rename from docs/FortiGate_traffic.png rename to docs/sources/Fortinet/FortiGate_traffic.png diff --git a/docs/FortiGate_utm.png b/docs/sources/Fortinet/FortiGate_utm.png similarity index 100% rename from docs/FortiGate_utm.png rename to docs/sources/Fortinet/FortiGate_utm.png diff --git a/docs/sources/Fortinet/index.md b/docs/sources/Fortinet/index.md new file mode 100644 index 0000000..a13bc1d --- /dev/null +++ b/docs/sources/Fortinet/index.md @@ -0,0 +1,108 @@ +# Vendor - Fortinet + +## Product - Fortigate + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2846/ | +| Product Manual | https://docs.fortinet.com/product/fortigate/6.2 | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| fgt_log | The catch all sourcetype is not used | +| fgt_traffic | None | +| fgt_utm | None | +| fgt_event | None + + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| fortinet_fortios_traffic | fgt_traffic | netops | none | +| fortinet_fortios_utm | fgt_utm | netids | none | +| fortinet_fortios_event | fgt_event | netops | none | +| fortinet_fortios_log | fgt_log | netops | none | + + +### Filter type + +MSG Parse: This filter parses message content + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Refer to the admin manual for specific details of configuration to send Reliable syslog using RFC 3195 format, a typical logging configuration will include the following features. + +``` +config log memory filter + +set forward-traffic enable + +set local-traffic enable + +set sniffer-traffic disable + +set anomaly enable + +set voip disable + +set multicast-traffic enable + +set dns enable + +end + +config system global + +set cli-audit-log enable + +end + +config log setting + +set neighbor-event enable + +end + +``` + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_FORTINET_FORTIOS_TCP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_LISTEN_FORTINET_FORTIOS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_FORTINET_FORTIOS | no | Enable archive to disk for this specific source | +| SC4S_DEST_FORTINET_FORTIOS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +An active firewall will generate frequent events, in addition fortigate has the ability to test logging functionality using a built in command + +``` +diag log test +``` + +Verify timestamp, and host values match as expected + +``` +index= (sourcetype=fgt_log OR sourcetype=fgt_traffic OR sourcetype=fgt_utm) +``` + +### UTM Message type + +![FortiGate UTM message](FortiGate_utm.png) + +### Traffic Message Type + +![FortiGate Traffic message](FortiGate_traffic.png) + +###Event Message Type +![FortiGate Event message](FortiGate_event.png) + +Verify timestamp, and host values match as expected \ No newline at end of file diff --git a/docs/sources/Imperva/index.md b/docs/sources/Imperva/index.md new file mode 100644 index 0000000..2ae9eea --- /dev/null +++ b/docs/sources/Imperva/index.md @@ -0,0 +1,53 @@ +# Vendor - Imperva + +## Product - Incapsula + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on CEF | https://bitbucket.org/SPLServices/ta-cef-for-splunk/downloads/ | +| Splunk Add-on Source Specific | https://bitbucket.org/SPLServices/ta-cef-imperva-incapsula/downloads/ | +| Product Manual | https://docs.imperva.com/bundle/cloud-application-security/page/more/log-configuration.htm | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| cef | Common sourcetype | + +### Source + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| Imperva:Incapsula | Common sourcetype | + +### Index Configuration + +| key | source | index | notes | +|----------------|----------------|----------------|----------------| +| cef_Incapsula_SIEMintegration | Imperva:Incapsula | netwaf | none | + +### Filter type + +MSG Parse: This filter parses message content + +### Options + +Note listed for reference processing utilizes the Microsoft ArcSight log path as this format is a subtype of CEF + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT | no | Enable archive to disk for this specific source | +| SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +An active site will generate frequent events use the following search to check for new events + +Verify timestamp, and host values match as expected + +``` +index= (sourcetype=cef source="Imperva:Incapsula") +``` \ No newline at end of file diff --git a/docs/sources/Juniper/index.md b/docs/sources/Juniper/index.md new file mode 100644 index 0000000..98cd701 --- /dev/null +++ b/docs/sources/Juniper/index.md @@ -0,0 +1,207 @@ +# Vendor - Juniper + +## Product - Juniper JunOS + +| Ref | Link | +|-------------------|-------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2847/ | +| JunOS TechLibrary | https://www.juniper.net/documentation/en_US/junos/topics/example/syslog-messages-configuring-qfx-series.html | + +### Sourcetypes + +| sourcetype | notes | +|--------------------------|------------------------------------------------------------------| +| juniper:junos:firewall | None | +| juniper:junos:idp | None | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------------------|------------------------|----------------|---------------| +| juniper_junos_flow | juniper:junos:firewall | netfw | none | +| juniper_junos_idp | juniper:junos:idp | netids | none | +| juniper_junos_utm | juniper:junos:firewall | netfw | none | + +### Filter type + +* MSG Parse: This filter parses message content + + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index as required. +* Follow vendor configuration steps per referenced Product Manual + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_JUNIPER_JUNOS_LEGACY_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined using legacy 3164 format| +| SC4S_LISTEN_JUNIPER_JUNOS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined using 5424 format | +| SC4S_ARCHIVE_JUNIPER_JUNOS | no | Enable archive to disk for this specific source | +| SC4S_DEST_JUNIPER_JUNOS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +Use the following search to validate events are present; for Juniper JunOS ensure each host filter condition is verified + +``` +index= sourcetype=juniper:junos:firewall | stats count by host +index= sourcetype=juniper:junos:idp | stats count by host +``` + +Verify timestamp, and host values match as expected + +## Product - Juniper NSM + +| Ref | Link | +|----------------|-------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2847/ | +| NSM syslog KB | http://kb.juniper.net/InfoCenter/index?page=content&id=KB11810 | + +### Sourcetypes + +| sourcetype | notes | +|------------------|-----------------------------------------------------------------------| +| juniper:nsm | None | +| juniper:nsm:idp | None | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|------------------------|---------------------|----------------|---------------| +| juniper_nsm | juniper:nsm | netfw | none | +| juniper_nsm_idp | juniper:nsm:idp | netids | none | + +### Filter type + +* Juniper NSM products must be identified by host or ip assignment. Update the filter `f_juniper_nsm` or `f_juniper_nsm_idp` as required + + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index as required. +* Follow vendor configuration steps per Product Manual + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_JUNIPER_NSM_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_JUNIPER_NSM_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_JUNIPER_NSM | no | Enable archive to disk for this specific source | +| SC4S_DEST_JUNIPER_NSM_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +Use the following search to validate events are present; for Juniper NSM ensure each host filter condition is verified + +``` +index= sourcetype=juniper:nsm | stats count by host +index= sourcetype=juniper:nsm:idp | stats count by host +``` + +Verify timestamp, and host values match as expected + +## Product - Juniper Netscreen + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2847/ | +| Netscreen Manual | http://kb.juniper.net/InfoCenter/index?page=content&id=KB4759 | + +### Sourcetypes + +| sourcetype | notes | +|-------------------------|------------------------------------------------------------------------------------------------| +| netscreen:firewall | None | +| juniper:idp | None | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|------------------------|---------------------|----------------|---------------| +| juniper_netscreen | netscreen:firewall | netfw | none | +| juniper_idp | juniper:idp | netfw | none | + +### Filter type + +* Juniper Netscreen products must be identified by host or ip assignment. Update the filter `f_juniper_netscreen` or `f_juniper_idp` as required + + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index as required. +* Follow vendor configuration steps per Product Manual + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_JUNIPER_NETSCREEN_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_JUNIPER_NETSCREEN | no | Enable archive to disk for this specific source | +| SC4S_DEST_JUNIPER_NETSCREEN_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +Use the following search to validate events are present; for Juniper Netscreen products ensure each host filter condition is verified + +``` +index= sourcetype=netscreen:firewall | stats count by host +index= sourcetype=juniper:idp | stats count by host +``` + +Verify timestamp, and host values match as expected + +## Product - Juniper SSLVPN + +| Ref | Link | +|------------------|-------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2847/ | +| Pulse Secure KB | https://kb.pulsesecure.net/articles/Pulse_Secure_Article/KB22227 | + +### Sourcetypes + +| sourcetype | notes | +|------------------|-----------------------------------------------------------------------| +| juniper:sslvpn | None | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|------------------------|---------------------|----------------|---------------| +| juniper_sslvpn | juniper:sslvpn | netfw | none | + +### Filter type + +* MSG Parse: This filter parses message content + + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index as required. +* Follow vendor configuration steps per referenced Product Manual + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_JUNIPER_JUNOS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_JUNIPER_JUNOS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_JUNIPER_JUNOS | no | Enable archive to disk for this specific source | +| SC4S_DEST_JUNIPER_JUNOS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +Use the following search to validate events are present; for Juniper SSL VPN ensure each host filter condition is verified + +``` +index= sourcetype=juniper:sslvpn | stats count by host +``` + +Verify timestamp, and host values match as expected \ No newline at end of file diff --git a/docs/sources/Microfocus/index.md b/docs/sources/Microfocus/index.md new file mode 100644 index 0000000..5909324 --- /dev/null +++ b/docs/sources/Microfocus/index.md @@ -0,0 +1,99 @@ +# Vendor - Microfocus ArcSight + +## Product - Internal Agent Events + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on CEF | https://bitbucket.org/SPLServices/ta-cef-for-splunk/downloads/ | +| Product Manual | https://docs.imperva.com/bundle/cloud-application-security/page/more/log-configuration.htm | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| cef | Common sourcetype | + +### Source + +| source | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| ArcSight:ArcSight | Internal logs | + +### Index Configuration + +| key | source | index | notes | +|----------------|----------------|----------------|----------------| +| cef_ArcSight_ArcSight | ArcSight:ArcSight | main | none | + +### Filter type + +MSG Parse: This filter parses message content + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | + +### Verification + +An active site will generate frequent events use the following search to check for new events + +Verify timestamp, and host values match as expected + +``` +index= (sourcetype=cef source="ArcSight:ArcSight") +``` + +## Product - Microsoft Windows + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on CEF | https://bitbucket.org/SPLServices/ta-cef-for-splunk/downloads/ | +| Splunk Add-on CEF | https://bitbucket.org/SPLServices/ta-cef-microsoft-windows-for-splunk/downloads/ | +| Product Manual | https://docs.imperva.com/bundle/cloud-application-security/page/more/log-configuration.htm | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| cef | Common sourcetype | + +### Source + +| source | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| CEFEventLog:System or Application Event | Windows Application and System Event Logs | +| CEFEventLog:Microsoft Windows | Windows Security Event Logs | + +### Index Configuration + +| key | source | index | notes | +|----------------|----------------|----------------|----------------| +| cef_Microsoft_System or Application Event | CEFEventLog:System or Application Event | oswin | none | +| cef_Microsoft_Microsoft Windows | CEFEventLog:Microsoft Windows | oswinsec | none | + +### Filter type + +MSG Parse: This filter parses message content + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_MICROFOCUS_ARCSIGHT_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT | no | Enable archive to disk for this specific source | +| SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +An active site will generate frequent events use the following search to check for new events + +Verify timestamp, and host values match as expected + +``` +index= (sourcetype=cef (source="CEFEventLog:Microsoft Windows" OR source="CEFEventLog:System or Application Event")) +``` \ No newline at end of file diff --git a/docs/sources/PaloaltoNetworks/index.md b/docs/sources/PaloaltoNetworks/index.md new file mode 100644 index 0000000..bc27602 --- /dev/null +++ b/docs/sources/PaloaltoNetworks/index.md @@ -0,0 +1,63 @@ +# Vendor - PaloAlto + +## Product - NGFW + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2757/ | +| Product Manual | https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/configure-syslog-monitoring.html | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| pan:log | None | +| pan:traffic | None | +| pan:threat | None | +| pan:system | None | +| pan:config | None | +| pan:hipwatch | None | +| pan:correlation | None | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| pan_log | pan:log | netops | none | +| pan_traffic | pan:traffic | netfw | none | +| pan_threat | pan:threat | netproxy | none | +| pan_system | pan:system | netops | none | +| pan_config | pan:config | netops | none | +| pan_hipwatch | pan:hipwatch | netops | none | +| pan_correlation | pan:correlation | netops | none | + +### Filter type + +MSG Parse: This filter parses message content + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Refer to the admin manual for specific details of configuration + * Select TCP or SSL transport option + * Select IETF Format + * Ensure the format of the event is not customized + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_PALOALTO_PANOS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_PALOALTO_PANOS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_PALOALTO_PANOS | no | Enable archive to disk for this specific source | +| SC4S_DEST_PALOALTO_PANOS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +An active firewall will generate frequent events. Use the following search to validate events are present per source device + +``` +index= sourcetype=pan:*| stats count by host +``` diff --git a/docs/sources/Proofpoint/index.md b/docs/sources/Proofpoint/index.md new file mode 100644 index 0000000..1fac35c --- /dev/null +++ b/docs/sources/Proofpoint/index.md @@ -0,0 +1,53 @@ +# Vendor - Proofpoint + +## Product - Proofpoint Protection Server + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/3080/ | +| Product Manual | https://proofpointcommunities.force.com/community/s/article/Remote-Syslog-Forwarding | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| pps_filter_log | | +| pps_mail_log | This sourcetype will conflict with sendmail itself, so will require that the PPS send syslog on a dedicated port or be uniquely identifiable with a hostname glob or CIDR block if this sourcetype is desired for PPS. | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| proofpoint_pps_filter | pps_filter_log | email | none | +| proofpoint_pps_sendmail | pps_mail_log | email | none | + + +### Filter type + +MSG Parse: This filter parses message content +* NOTE: This filter will simply parse the syslog message itself, and will _not_ perform the (required) re-assembly of related +messages to create meaningful final output. This will require follow-on processing in Splunk. + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Follow vendor configuration steps per referenced Product Manual + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_PROOFPOINT_PPS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined. | +| SC4S_PROOFPOINT_PPS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined. | +| SC4S_ARCHIVE_PROOFPOINT_PPS | no | Enable archive to disk for this specific source | +| SC4S_DEST_PROOFPOINT_PPS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +One or two sourcetypes are included in Proofpoint PPS logs. The search below will surface both of them: + +``` +index= sourcetype=pps_*_log | stats count by host +``` \ No newline at end of file diff --git a/docs/sources/Symantec/index.md b/docs/sources/Symantec/index.md new file mode 100644 index 0000000..e169c78 --- /dev/null +++ b/docs/sources/Symantec/index.md @@ -0,0 +1,51 @@ +# Vendor - Symantec + +## Product - ProxySG/ASG (Bluecoat) + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2758/ | +| Product Manual | https://support.symantec.com/us/en/article.tech242216.html | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| bluecoat:proxysg:access:kv | Requires version TA 3.6 | + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| bluecoat_proxy | bluecoat:proxysg:access:kv | netops | none | + + +### Filter type + +MSG Parse: This filter parses message content + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration + * Select TCP or SSL transport option + * Ensure the format of the event is customized per Splunk documentation + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_SYMANTEC_PROXY_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_SYMANTEC_PROXY_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_SYMANTEC_PROXY | no | Enable archive to disk for this specific source | +| SC4S_DEST_SYMANTEC_PROXY_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +An active proxy will generate frequent events. Use the following search to validate events are present per source device + +``` +index= sourcetype=bluecoat:proxysg:access:kv | stats count by host +``` diff --git a/docs/sources/Ubiquiti/index.md b/docs/sources/Ubiquiti/index.md new file mode 100644 index 0000000..1769377 --- /dev/null +++ b/docs/sources/Ubiquiti/index.md @@ -0,0 +1,75 @@ +# Vendor - Ubiquiti - Unifi + +All Ubiquity Unfi firewalls, switches, and access points share a common syslog configuration via the NMS. + + +* Login to NMS +* Navigate to settings +* Navigate to Site +* Enable Remote syslog server +* Enter hostname and port +* Update ``vi /opt/sc4s/local/context/vendor_product_by_source.conf `` update the host or ip mask for ``f_ubiquiti_unifi_fw`` to identify USG firewalls + +## Product - Unifi Switch and Access Points + +Unifi devices are managed using the Network Management Controller + + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/4107/ | +| Product Manual | https://https://help.ubnt.com/ | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| ubnt | Used when no sub source type is required by add on | +| ubnt:fw | USG events | +| ubnt:threat | USG IDS events | +| ubnt:switch | Unifi Switches | +| ubnt:wireless | Access Point logs | + + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| ubiquiti_unifi | ubnt | netops | none | +| ubiquiti_unifi_fw | ubnt:fw | netfw | none | +| ubiquiti_unifi_link | ubnt:link | netops | none | +| ubiquiti_unifi_sudo | ubnt:sudo | netops | none | +| ubiquiti_unifi_switch | ubnt:switch | netops | none | +| ubiquiti_unifi_threat | ubnt:threat | netids | none | +| ubiquiti_unifi_wireless | ubnt:wireless | netops | none | + + +### Filter type + +MSG Parse: This filter parses message content + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration + * Select TCP or SSL transport option + * Ensure the format of the event is customized per Splunk documentation + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_UBIQUITI_UNIFI_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_UBIQUITI_UNIFI_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_UBIQUITI_UNIFI | no | Enable archive to disk for this specific source | +| SC4S_DEST_UBIQUITI_UNIFI_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +An active proxy will generate frequent events. Use the following search to validate events are present per source device + +``` +index= sourcetype=zscalernss-* | stats count by host +``` diff --git a/docs/sources/Zscaler/index.md b/docs/sources/Zscaler/index.md new file mode 100644 index 0000000..67b70e1 --- /dev/null +++ b/docs/sources/Zscaler/index.md @@ -0,0 +1,69 @@ +# Vendor - Zscaler + +## Product - All Products + +The ZScaler product manual includes and extensive section of configuration for multiple Splunk TCP input ports around page +26. When using SC4S these ports are not required and should not be used. Simply configure all outputs from the NSS to utilize +the IP or host name of the SC4S instance and port 514 + + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/3865/ | +| Product Manual | https://community.zscaler.com/t/zscaler-splunk-app-design-and-installation-documentation/4728 | + + +### Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| zscalernss-alerts | Requires format customization add ``\tvendor=Zscaler\tproduct=alerts`` immediately prior to the ``\n`` in the NSS Alert Web format. See Zscaler manual for more info. | +| zscalernss-dns | Requires format customization add ``\tvendor=Zscaler\tproduct=dns`` immediately prior to the ``\n`` in the NSS DNS format. See Zscaler manual for more info. | +| zscalernss-web | None | +| zscalernss-zpa-app | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. | +| zscalernss-zpa-auth | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa_auth`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. | +| zscalernss-zpa-connector | Requires format customization add ``\tvendor=Zscaler\tproduct=zpa_auth_connector`` immediately prior to the ``\n`` in the LSS Connector format. See Zscaler manual for more info. | +| zscalernss-fw | Requires format customization add ``\tvendor=Zscaler\tproduct=fw`` immediately prior to the ``\n`` in the Firewall format. See Zscaler manual for more info. | + + +### Sourcetype and Index Configuration + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| zscalernss_alerts | zscalernss-alerts | main | none | +| zscalernss_dns | zscalernss-dns | netdns | none | +| zscalernss_fw | zscalernss-fw | netfw | none | +| zscalernss_web | zscalernss-web | netproxy | none | +| zscalernss-zpa-app | zscalernss_zpa-app | netids | none | +| zscalernss-zpa-auth | zscalernss_zpa_auth | netauth | none | +| zscalernss-zpa-connector | zscalernss_zpa_connector | netops | none | + + +### Filter type + +MSG Parse: This filter parses message content + +### Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source. +* Refer to the Splunk TA documentation for the specific customer format required for proxy configuration + * Select TCP or SSL transport option + * Ensure the format of the event is customized per Splunk documentation + +### Options + +| Variable | default | description | +|----------------|----------------|----------------| +| SC4S_LISTEN_ZSCALER_NSS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined | +| SC4S_LISTEN_ZSCALER_NSS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined | +| SC4S_ARCHIVE_ZSCALER_NSS | no | Enable archive to disk for this specific source | +| SC4S_DEST_ZSCALER_NSS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source | + +### Verification + +An active proxy will generate frequent events. Use the following search to validate events are present per source device + +``` +index= sourcetype=zscalernss-* | stats count by host +``` diff --git a/docs/sources/index.md b/docs/sources/index.md new file mode 100644 index 0000000..c32c773 --- /dev/null +++ b/docs/sources/index.md @@ -0,0 +1,7 @@ +# Introduction +When using Splunk Connect for Syslog to onboard a data source, the SC4S filter performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source. + +SC4S "unique" filters are based either on the port upon which events arrive or the hostname/CIDR block from which they are sent. The "soup" filters run for events that arrive on port 514 (default for syslog), and contain regex and other syslog-specific parsers to identify events from a specific source, apply the correct sourcetype, and set other metadata. Data sources which generate events that are not unique enough to accurately identify with soup filters _must_ employ the "unique" filters (port/hostname/CIDR block) instead -- the soup filters are unavailable for these sources. + +If SC4S receives an event on port 514 which has no soup filter, that event will be given a "fallback" sourcetype. If you see events in Splunk with the fallback sourcetype, then you should figure out what source the events are from and determine why these events are not being sourcetyped correctly. The most common reason for events categorized as "fallback" is the lack of a SC4S filter for that source, and in some cases a misconfigured relay which alters the integrity of the message format. In most cases this means a new SC4S filter must be developed. In this situation you can either build a filter or file an issue with the community to request help. + diff --git a/mkdocs-requirements.txt b/mkdocs-requirements.txt new file mode 100644 index 0000000..4c8f017 --- /dev/null +++ b/mkdocs-requirements.txt @@ -0,0 +1 @@ +mkdocs-material diff --git a/mkdocs.yml b/mkdocs.yml index cc7f4e8..6b82a92 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,10 +1,44 @@ site_name: Splunk Connect for Syslog -theme: readthedocs + nav: - Home: 'index.md' - - Performance: 'performance.md' - - Getting Started: 'gettingstarted.md' - - Demo Lab: 'demo.md' + - Getting Started: + - 'Read First': 'gettingstarted/index.md' + - 'Podman + systemd single node': 'gettingstarted/podman-systemd-general.md' + - 'Docker CE + systemd single node': 'gettingstarted/docker-systemd-general.md' + - 'Docker CE + Swarm single node': 'gettingstarted/docker-swarm-rhel7.md' + - 'Bring your own Envionment': 'gettingstarted/byoe-rhel7.md' - Configuration: 'configuration.md' - - Sources: 'sources.md' + - Sources: + - About: sources/index.md + - Checkpoint: sources/Checkpoint/index.md + - Cisco: sources/Cisco/index.md + - Forcepoint: sources/Forcepoint/index.md + - Fortinet: sources/Fortinet/index.md + - Imperva: sources/Imperva/index.md + - Juniper: sources/Juniper/index.md + - Microfocus: sources/Microfocus/index.md + - 'Paloalto Networks': sources/PaloaltoNetworks/index.md + - Proofpoint: sources/Proofpoint/index.md + - Symantec: sources/Symantec/index.md + - Ubiquiti: sources/Ubiquiti/index.md + - Zscaler: sources/Zscaler/index.md + - 'Demo Lab': 'demo.md' + - Performance: 'performance.md' - Troubleshooting: 'troubleshooting.md' + +markdown_extensions: + - toc: + permalink: True + - smarty + - fenced_code + - sane_lists + - codehilite + +theme: + name: 'material' + palette: + primary: 'black' + accent: 'orange' + favicon: 'logo.png' + logo: 'logo.png' \ No newline at end of file diff --git a/package/etc/conf.d/local/config/log_paths/example.conf.tmpl b/package/etc/conf.d/local/config/log_paths/example.conf.tmpl index 6a68b45..6eae6a3 100644 --- a/package/etc/conf.d/local/config/log_paths/example.conf.tmpl +++ b/package/etc/conf.d/local/config/log_paths/example.conf.tmpl @@ -55,7 +55,7 @@ source (s_LOCAL_EXAMPLE); # Send it to Splunk - destination(d_hec); #--HEC-- + destination(d_hec); # Note: We normally do not use the "final" flag; this will allow another plugin to be created that will # forward events to another system diff --git a/package/etc/conf.d/log_paths/internal.conf.tmpl b/package/etc/conf.d/log_paths/internal.conf.tmpl index c751b9e..4373f5c 100644 --- a/package/etc/conf.d/log_paths/internal.conf.tmpl +++ b/package/etc/conf.d/log_paths/internal.conf.tmpl @@ -16,7 +16,11 @@ log { value("MESSAGE") flags("utf8" "global") ); }; - destination(d_hecmetrics); #--HEC-- + + {{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_INTERNAL_METRICS_HEC" "no") | conv.ToBool) }} + destination(d_hecmetrics); + {{- end}} + } else { {{- if eq (getenv "SC4S_DEBUG_STDOUT" "yes") "yes"}} @@ -24,7 +28,11 @@ log { {{- end}} rewrite { r_set_splunk_dest_default(sourcetype("sc4s:events"), index("main"))}; parser {p_add_context_splunk(key("sc4s_events")); }; + + {{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_INTERNAL_EVENTS_HEC" "no") | conv.ToBool) }} destination(d_hec_internal); + {{- end}} + }; }; {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl index 2fa996f..1f5c620 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl @@ -69,8 +69,10 @@ log { unset(value("LEGACY_MSGHDR")); groupunset(values(".kv.*")); }; - +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CHECKPOINT_SPLUNK_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} + {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CHECKPOINT_SPLUNK") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-cisco_asa.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-cisco_asa.conf.tmpl index 2506ca5..a07d6a1 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-cisco_asa.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-cisco_asa.conf.tmpl @@ -32,7 +32,9 @@ log { unset(value("LEGACY_MSGHDR")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ASA_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ASA_LEGACY") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-cisco_ios.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-cisco_ios.conf.tmpl index c47fcd9..554277b 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-cisco_ios.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-cisco_ios.conf.tmpl @@ -35,7 +35,9 @@ log { groupunset(values(".cisco.*")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_IOS_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_IOS") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-cisco_ise.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-cisco_ise.conf.tmpl index 5f1bfb9..cca9afd 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-cisco_ise.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-cisco_ise.conf.tmpl @@ -84,7 +84,9 @@ log { groupunset(values("ISE.*")); }; - destination(d_hec); +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ISE_HEC" "no") | conv.ToBool) }} + destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ISE") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-cisco_nxos.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-cisco_nxos.conf.tmpl index 458ada4..fd7a8b4 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-cisco_nxos.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-cisco_nxos.conf.tmpl @@ -34,7 +34,10 @@ log { unset(value("LEGACY_MSGHDR")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_NXOS_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} + {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_NXOS") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-forcepoint_webprotect.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-forcepoint_webprotect.conf.tmpl index 5c000ba..b35e857 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-forcepoint_webprotect.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-forcepoint_webprotect.conf.tmpl @@ -33,7 +33,9 @@ log { unset(value("LEGACY_MSGHDR")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_FORCEPOINT_WEBPROTECT_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_FORCEPOINT_WEBPROTECT") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-fortinet_fortios.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-fortinet_fortios.conf.tmpl index 6fe1189..1be63f1 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-fortinet_fortios.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-fortinet_fortios.conf.tmpl @@ -51,7 +51,10 @@ log { groupunset(values(".kv.*")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_FORTINET_FORTIOS_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} + {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_FORTINET_FORTIOS") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_idp.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_idp.conf.tmpl index c1dc820..761a8c0 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_idp.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_idp.conf.tmpl @@ -36,7 +36,10 @@ log { groupunset(values(".kv.*")); }; + +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_IDP_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_IDP") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_junos.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_junos.conf.tmpl index 6185a34..d461e5e 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_junos.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_junos.conf.tmpl @@ -50,7 +50,9 @@ log { unset(value("LEGACY_MSGHDR")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_JUNOS_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_JUNOS") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_netscreen.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_netscreen.conf.tmpl index 9d6116f..b735a73 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_netscreen.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_netscreen.conf.tmpl @@ -34,7 +34,9 @@ log { unset(value("LEGACY_MSGHDR")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_NETSCREEN_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_NETSCREEN") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm.conf.tmpl index 404781d..5571ba0 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm.conf.tmpl @@ -35,7 +35,9 @@ log { unset(value("LEGACY_MSGHDR")); }; + {{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_NSM_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_NSM") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm_idp.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm_idp.conf.tmpl index 0383731..bb717e0 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm_idp.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-juniper_nsm_idp.conf.tmpl @@ -32,7 +32,9 @@ log { unset(value("LEGACY_MSGHDR")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNIPER_NSM_IDP_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNIPER_NSM_IDP") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-microfocus_arcsight.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-microfocus_arcsight.conf.tmpl index 549b939..db6cd6a 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-microfocus_arcsight.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-microfocus_arcsight.conf.tmpl @@ -80,7 +80,9 @@ log { groupunset(values(".cef.*")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_MICROFOCUS_ARCSIGHT_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_MICROFOCUS_ARCSIGHT") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-paloalto_panos.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-paloalto_panos.conf.tmpl index 71335e2..05c45ad 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-paloalto_panos.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-paloalto_panos.conf.tmpl @@ -84,7 +84,9 @@ log { groupunset(values(".pan.*")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_PALOALTO_PANOS_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_PALOALTO_PANOS") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps.conf.tmpl new file mode 100644 index 0000000..78f234c --- /dev/null +++ b/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps.conf.tmpl @@ -0,0 +1,63 @@ +# Proofpoint Protection Server +{{ $context := dict "port_id" "PROOFPOINT_PPS" "parser" "common" }} +{{ tmpl.Exec "t/source_network.t" $context }} +# The following is an inline template; we will use this to generate the actual log path +{{ define "log_path" }} +log { +{{- if eq (.) "yes" }} + source(s_DEFAULT); + filter { filter(f_proofpoint_pps_filter) or filter(f_proofpoint_pps_sendmail) }; +{{- end }} +{{- if eq (.) "no" }} + source (s_PROOFPOINT_PPS); +{{- end }} + + if (filter(f_proofpoint_pps_filter)) { + rewrite { + set("proofpoint_pps_filter", value("fields.sc4s_vendor_product")); + r_set_splunk_dest_default(sourcetype("pps_filter_log"), index("email"))}; + parser { + p_add_context_splunk(key("proofpoint_pps_filter")); + }; + } else { + rewrite { + set("proofpoint_pps_sendmail", value("fields.sc4s_vendor_product")); + r_set_splunk_dest_default(sourcetype("pps_mail_log"), index("email"))}; + parser { + p_add_context_splunk(key("proofpoint_pps_sendmail")); + }; + }; + + parser (compliance_meta_by_source); + + + #We want to unset the fields we won't need, as this is copied into the + #disk queue for network destinations. This can be very disk expensive + #if we don't + rewrite { + set("$(template ${fields.sc4s_template} $(template t_msg_only))" value("MSG")); + unset(value("RAWMSG")); + unset(value("PROGRAM")); + unset(value("LEGACY_MSGHDR")); + groupunset(values(".kv.*")); + }; + +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_PROOFPOINT_PPS_HEC" "no") | conv.ToBool) }} + destination(d_hec); +{{- end}} + +{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_PROOFPOINT_PPS") }} + destination(d_archive); +{{- end}} + + flags(flow-control); +}; +{{- end}} + +{{- if or (or (getenv (print "SC4S_LISTEN_PROOFPOINT_PPS_TCP_PORT")) (getenv (print "SC4S_LISTEN_PROOFPOINT_PPS_UDP_PORT"))) (getenv (print "SC4S_LISTEN_PROOFPOINT_PPS_TLS_PORT")) }} +# Listen on the specified dedicated port(s) for PROOFPOINT_PPS traffic + {{ tmpl.Exec "log_path" "no" }} +{{- end}} + +# Listen on the default port (typically 514) for PROOFPOINT_PPS traffic +{{ tmpl.Exec "log_path" "yes" }} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps_sendmail.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps_sendmail.conf.tmpl deleted file mode 100644 index 694f14e..0000000 --- a/package/etc/conf.d/log_paths/p_rfc3164-proofpoint_pps_sendmail.conf.tmpl +++ /dev/null @@ -1,51 +0,0 @@ -# Proofpoint -{{ $context := dict "port_id" "PROOFPOINT_PPS_SENDMAIL" "parser" "common" }} -{{ tmpl.Exec "t/source_network.t" $context }} -# The following is an inline template; we will use this to generate the actual log path -{{ define "log_path" }} -log { -{{- if eq (.) "yes" }} - source(s_DEFAULT); - filter(f_proofpoint_pps_sendmail); -{{- end }} -{{- if eq (.) "no" }} - source (s_PROOFPOINT_PPS_SENDMAIL); -{{- end }} - - rewrite { - set("proofpoint_pps_sendmail", value("fields.sc4s_vendor_product")); - r_set_splunk_dest_default(sourcetype("pps_mail_log"), index("email"))}; - parser { - p_add_context_splunk(key("proofpoint_pps_sendmail")); - }; - - parser (compliance_meta_by_source); - - #We want to unset the fields we won't need, as this is copied into the - #disk queue for network destinations. This can be very disk expensive - #if we don't - rewrite { - set("$(template ${fields.sc4s_template} $(template t_msg_only))" value("MSG")); - unset(value("RAWMSG")); - unset(value("PROGRAM")); - unset(value("LEGACY_MSGHDR")); - groupunset(values(".kv.*")); - }; - - destination(d_hec); - -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_PROOFPOINT_PPS_SENDMAIL") }} - destination(d_archive); -{{- end}} - - flags(flow-control); -}; -{{- end}} - -{{- if or (or (getenv (print "SC4S_LISTEN_PROOFPOINT_PPS_SENDMAIL_TCP_PORT")) (getenv (print "SC4S_LISTEN_PROOFPOINT_PPS_SENDMAIL_UDP_PORT"))) (getenv (print "SC4S_LISTEN_PROOFPOINT_PPS_SENDMAIL_TLS_PORT")) }} -# Listen on the specified dedicated port(s) for PROOFPOINT_PPS_SENDMAIL traffic - {{ tmpl.Exec "log_path" "no" }} -{{- end}} - -# Listen on the default port (typically 514) for PROOFPOINT_PPS_SENDMAIL traffic -{{ tmpl.Exec "log_path" "yes" }} diff --git a/package/etc/conf.d/log_paths/p_rfc3164-ubiquiti_unifi.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-ubiquiti_unifi.conf.tmpl index 226b310..55bd6a7 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-ubiquiti_unifi.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-ubiquiti_unifi.conf.tmpl @@ -117,7 +117,10 @@ log { unset(value("LEGACY_MSGHDR")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_PROOFPOINT_UBIQUITI_UNIFI_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} + {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_UBIQUITI_UNIFI") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc3164-zscaler_nss.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164-zscaler_nss.conf.tmpl index c9de545..47a77c1 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164-zscaler_nss.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164-zscaler_nss.conf.tmpl @@ -73,7 +73,9 @@ log { groupunset(values(".kv.*")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_ZSCALER_NSS_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_ZSCALER_NSS") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc5424-noversion_cisco_asa.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc5424-noversion_cisco_asa.conf.tmpl index 98e3b78..a7f069c 100644 --- a/package/etc/conf.d/log_paths/p_rfc5424-noversion_cisco_asa.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc5424-noversion_cisco_asa.conf.tmpl @@ -31,7 +31,9 @@ log { groupunset(values(".kv.*")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_ASA_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_ASA") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc5424-noversion_symantec_proxy.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc5424-noversion_symantec_proxy.conf.tmpl index a9f881e..855d390 100644 --- a/package/etc/conf.d/log_paths/p_rfc5424-noversion_symantec_proxy.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc5424-noversion_symantec_proxy.conf.tmpl @@ -34,7 +34,9 @@ log { groupunset(values(".kv.*")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_SYMANTEC_PROXY_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_SYMANTEC_PROXY") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/p_rfc5424-strict_juniper_junos.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc5424-strict_juniper_junos.conf.tmpl index cc2d052..40bce61 100644 --- a/package/etc/conf.d/log_paths/p_rfc5424-strict_juniper_junos.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc5424-strict_juniper_junos.conf.tmpl @@ -52,9 +52,12 @@ log { groupunset(values(".kv.*")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_JUNOS_STRUCTURED_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} + -{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CHECKPOINT") }} +{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_JUNOS_STRUCTURED") }} destination(d_archive); {{- end}} diff --git a/package/etc/conf.d/log_paths/p_rfc5424_epoch-cisco_meraki.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc5424_epoch-cisco_meraki.conf.tmpl index 149fb4b..18a37b4 100644 --- a/package/etc/conf.d/log_paths/p_rfc5424_epoch-cisco_meraki.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc5424_epoch-cisco_meraki.conf.tmpl @@ -31,7 +31,9 @@ log { groupunset(values(".kv.*")); }; +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_CISCO_MERAKI_HEC" "no") | conv.ToBool) }} destination(d_hec); +{{- end}} {{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_CISCO_MERAKI") }} destination(d_archive); diff --git a/package/etc/conf.d/log_paths/zfallback.conf b/package/etc/conf.d/log_paths/zfallback.conf deleted file mode 100644 index 478ff31..0000000 --- a/package/etc/conf.d/log_paths/zfallback.conf +++ /dev/null @@ -1,14 +0,0 @@ -log { - source(s_DEFAULT); - - rewrite { - r_set_splunk_dest_default(sourcetype("sc4s:fallback"), index("main"), template("t_JSON")); - set("$(template ${fields.sc4s_template} $(template t_JSON))" value("MSG")); - }; - parser { - p_add_context_splunk(key("sc4s_fallback")); - }; - - destination(d_hec); #--HEC-- - flags(flow-control,fallback); -}; diff --git a/package/etc/conf.d/log_paths/zfallback.conf.tmpl b/package/etc/conf.d/log_paths/zfallback.conf.tmpl new file mode 100644 index 0000000..0f36b1f --- /dev/null +++ b/package/etc/conf.d/log_paths/zfallback.conf.tmpl @@ -0,0 +1,31 @@ +log { + source(s_DEFAULT); + + rewrite { + r_set_splunk_dest_default(sourcetype("sc4s:fallback"), index("main"), template("t_JSON")); + set("$(template ${fields.sc4s_template} $(template t_JSON))" value("MSG")); + }; + parser { + p_add_context_splunk(key("sc4s_fallback")); + }; + +{{- if ((getenv "SC4S_DEST_SPLUNK_HEC_GLOBAL" "yes") | conv.ToBool) or (conv.ToBool (getenv "SC4S_DEST_ARCHIVE_HEC" "no") | conv.ToBool) }} + destination(d_hec); +{{- end}} + + + #in fallback archive only write rawmsg as msg + rewrite { + set("value(RAWMSG)" value("MSG")); + unset(value("RAWMSG")); + unset(value("PROGRAM")); + unset(value("LEGACY_MSGHDR")); + groupunset(values(".kv.*")); + }; + +{{- if (getenv "SC4S_ARCHIVE_GLOBAL") or (getenv "SC4S_ARCHIVE_FALLBACK") }} + destination(d_archive); +{{- end}} + + flags(flow-control,fallback); +}; diff --git a/package/etc/local_config/log_paths/example.conf.tmpl b/package/etc/local_config/log_paths/example.conf.tmpl index 0f12886..6f75c8e 100644 --- a/package/etc/local_config/log_paths/example.conf.tmpl +++ b/package/etc/local_config/log_paths/example.conf.tmpl @@ -53,7 +53,7 @@ log { # Send it to Splunk - destination(d_hec); #--HEC-- + destination(d_hec); # Note: We normally do not use the "final" flag; this will allow another plugin to be created that will # forward events to another system diff --git a/package/etc/syslog-ng.conf b/package/etc/syslog-ng.conf index 88897f1..4f19083 100644 --- a/package/etc/syslog-ng.conf +++ b/package/etc/syslog-ng.conf @@ -21,7 +21,7 @@ options { chain_hostnames (off); use_dns (no); use_fqdn (no); - dns-cache(yes); + dns-cache(no); create_dirs (no); keep-hostname (yes); create_dirs(yes); diff --git a/tests/Dockerfile b/tests/Dockerfile index 0bf3d12..285d52c 100644 --- a/tests/Dockerfile +++ b/tests/Dockerfile @@ -9,13 +9,14 @@ FROM python:3.7 COPY requirements.txt / - RUN pip3 install -r /requirements.txt RUN mkdir -p /work/tests RUN mkdir -p /work/test-results/functional COPY entrypoint.sh / COPY wait-for /bin/ COPY ./* /work/tests/ +COPY pytest.ini /work COPY ./data /work/tests/data #WORKDIR /work -CMD /entrypoint.sh \ No newline at end of file +CMD /entrypoint.sh + diff --git a/tests/entrypoint.sh b/tests/entrypoint.sh index 646dd56..6128096 100755 --- a/tests/entrypoint.sh +++ b/tests/entrypoint.sh @@ -20,4 +20,4 @@ echo check for splunk hec wait-for splunk:8088 -t 0 -- echo splunkhec is up -cd /work;python -m pytest --junitxml=/work/test-results/functional/functional.xml +cd /work;python -m pytest --junitxml=/work/test-results/functional/functional.xml $@ diff --git a/tests/pytest.ini b/tests/pytest.ini new file mode 100644 index 0000000..00b64d9 --- /dev/null +++ b/tests/pytest.ini @@ -0,0 +1,5 @@ +[pytest] +addopts = + --force-flaky --max-runs=3 --min-passes=1 +filterwarnings = + ignore::DeprecationWarning diff --git a/tests/requirements.txt b/tests/requirements.txt index 6d908dd..df09269 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -11,4 +11,7 @@ jinja2 jinja2-time splunk-sdk flake8 -pytz \ No newline at end of file +pytz +flaky +#pytest-randomly +pytest-parallel diff --git a/tests/test_ubiquiti_unifi.py b/tests/test_ubiquiti_unifi.py index 4eed2b2..0a69280 100644 --- a/tests/test_ubiquiti_unifi.py +++ b/tests/test_ubiquiti_unifi.py @@ -17,12 +17,12 @@ def test_ubiquiti_unifi_us8p60(record_property, setup_wordlist, setup_splunk): host = "{}-{}".format(random.choice(setup_wordlist), random.choice(setup_wordlist)) mt = env.from_string( - "{{mark}}{% now 'utc', '%b %d %H:%M:%S' %} US8P60,18e8294876c3,v4.0.66.10832 switch: DOT1S: dot1sBpduReceive(): Discarding the BPDU on port 0/7, since it is an invalid BPDU type") - message = mt.render(mark="<27>", host=host) + "{{mark}}{% now 'utc', '%b %d %H:%M:%S' %} US8P60,18e8294876c3,v4.0.66.10832 switch: DOT1S: dot1sBpduReceive(): Discarding the BPDU on port 0/7, since it is an invalid BPDU type {{key}}") + message = mt.render(mark="<27>", key=host) sendsingle(message) - st = env.from_string("search index=netops sourcetype=ubnt:switch earliest=-2m | head 2") - search = st.render(host=host) + st = env.from_string("search index=netops sourcetype=ubnt:switch \"{{key}}\" earliest=-2m | head 2") + search = st.render(key=host) resultCount, eventCount = splunk_single(setup_splunk, search) @@ -37,12 +37,12 @@ def test_ubiquiti_unifi_switch_us24p250(record_property, setup_wordlist, setup_s host = "{}-{}".format(random.choice(setup_wordlist), random.choice(setup_wordlist)) mt = env.from_string( - "{{mark}}{% now 'utc', '%b %d %H:%M:%S' %} US24P250,f09fc26f4419,v4.0.54.10625 switch: TRAPMGR: Cold Start: Unit: 0") - message = mt.render(mark="<27>", host=host) + "{{mark}}{% now 'utc', '%b %d %H:%M:%S' %} US24P250,f09fc26f4419,v4.0.54.10625 switch: TRAPMGR: Cold Start: Unit: {{key}}") + message = mt.render(mark="<27>", key=host) sendsingle(message) - st = env.from_string("search index=netops sourcetype=ubnt:switch earliest=-2m | head 2") - search = st.render(host=host) + st = env.from_string("search index=netops sourcetype=ubnt:switch \"{{key}}\" earliest=-2m | head 2") + search = st.render(key=host) resultCount, eventCount = splunk_single(setup_splunk, search)