merge

.readthedocs.yml

@@ -9,4 +9,9 @@ mkdocs:
 formats: all
 
 submodules:
-  exclude: all
+  exclude: all
+
+python:
+  version: 3.7
+  install:
+    - requirements: mkdocs-requirements.txt

docker-compose-ci.yml

@@ -6,10 +6,16 @@
 #
 #You should have received a copy of the CC0 legalcode along with this
 #work.  If not, see <http://creativecommons.org/publicdomain/zero/1.0/>.
-version: "3.2"
+version: "3.7"
 services:
   test:
     build: ./tests
+    entrypoint:
+      - /entrypoint.sh
+      - --workers
+      - auto
+      - --tests-per-worker
+      - auto
     links:
       - splunk
       - sc4s

docker-compose.yml

@@ -6,7 +6,7 @@
 #
 #You should have received a copy of the CC0 legalcode along with this
 #work.  If not, see <http://creativecommons.org/publicdomain/zero/1.0/>.
-version: "3.2"
+version: "3.7"
 services:
   test:
     build: ./tests

docs/gettingstarted/byoe-rhel7.md

@@ -71,7 +71,7 @@ sudo cp -R etc/* /opt/syslog-ng/etc/
 ```bash
 sudo curl -o /usr/local/bin/gomplate -sSL https://github.com/hairyhenderson/gomplate/releases/download/v3.5.0/gomplate_linux-amd64
 sudo chmod 755 /usr/local/bin/gomplate
-gomplate --help
+gomplate --version
 ```
 
 * Install the latest python
@@ -127,6 +127,8 @@ mkdir -p /opt/syslog-ng/etc/conf.d/local/config/
 cp --verbose -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context/
 cp --verbose -R -n /opt/syslog-ng/etc/local_config/* /opt/syslog-ng/etc/conf.d/local/config/
 mkdir -p /opt/syslog-ng/var/data/disk-buffer/
+mkdir -p /opt/syslog-ng/var/archive/
+mkdir -p /opt/syslog-ng/tls/
 ```
 
 * (Optional) Execute the preconfiguration shell script created above.  You may also optionally execute it as part of the unit

docs/gettingstarted/docker-swarm-general.md

@@ -51,8 +51,10 @@ services:
     volumes:
       - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z
       - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z
+# Uncomment the following line if local disk archiving is desired
+#     - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z
 # Uncomment the following line if custom TLS certs are provided
-#     - /opt/sc4s/tls:/opt/syslog-ng/tls
+#     - /opt/sc4s/tls:/opt/syslog-ng/tls:z
 ```
 
 * Create the subdirectory ``/opt/sc4s/local``.  This will be used as a mount point for local overrides and configurations.
@@ -76,7 +78,14 @@ of events in the event of network failure to the Splunk infrastructure.
     If you are sure, after stopping SC4S, that all data has been sent, these files can be removed.  They will be created
     again upon restart.
 
-* IMPORTANT:  When creating the two directories above, ensure the directories created match the volume mounts specified in the
+* Create the subdirectory ``/opt/sc4s/archive``.  This will be used as a mount point for local storage of syslog events
+(if the optional mount is uncommented above).  The events will be written in the syslog-ng EWMM format. See the "configuration"
+document for details on the directory structure the archive uses.
+
+* Create the subdirectory ``/opt/sc4s/tls``.  This will be used as a mount point for custom TLS certificates
+(if the optional mount is uncommented above). 
+
+* IMPORTANT:  When creating the directories above, ensure the directories created match the volume mounts specified in the
 `docker-compose.yml` file.  Failure to do this will cause SC4S to abort at startup.
 
 ## Configure the SC4S environment
@@ -107,12 +116,15 @@ Log paths are preconfigured to utilize a convention of index destinations that a
 * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your
 environment. Simply uncomment the relevant line and enter the desired index.  The "Sources" document details the specific entries in
 this table that pertain to the individual data source filters that are included with SC4S.
+* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well.  This is an advanced topic, and further
+information is covered in the "Log Path overrides" section of the Configuration document.
 
 ## Configure source filtering by source IP or host name
 
 Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps
 apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. 
 
+* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start.
 * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type.  
 * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax.  Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. 
 * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment.  In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`.
@@ -122,8 +134,8 @@ apply to support such sources. To identify sources that require this step, refer
 In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion.
 The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose.  These operate similarly to
 the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file
-lists one or more metadata items that can be overridden based on the filter name.  This is an advanced topic, and further information is in
-the "Configuration" section.
+lists one or more metadata items that can be overridden based on the filter name.  This is an advanced topic, and further information is
+covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document.
 
 ## Start/Restart SC4S
 

docs/gettingstarted/docker-swarm-rhel7.md

@@ -59,8 +59,10 @@ services:
     volumes:
       - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z
       - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z
+# Uncomment the following line if local disk archiving is desired
+#     - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z
 # Uncomment the following line if custom TLS certs are provided
-#     - /opt/sc4s/tls:/opt/syslog-ng/tls
+#     - /opt/sc4s/tls:/opt/syslog-ng/tls:z
 ```
 
 * Create the subdirectory ``/opt/sc4s/local``.  This will be used as a mount point for local overrides and configurations.
@@ -82,9 +84,16 @@ of events in the event of network failure to the Splunk infrastructure.
     * This directory will populate with the disk buffer files upon SC4S startup.  If SC4S restarts for any reason, a new
 set of files will be created in addition to the original ones.  _The original ones will not be removed_.
 If you are sure, after stopping SC4S, that all data has been sent, these files can be removed.  They will be created
-again upon restart.
+again upon restart
+
+* Create the subdirectory ``/opt/sc4s/archive``.  This will be used as a mount point for local storage of syslog events
+(if the optional mount is uncommented above).  The events will be written in the syslog-ng EWMM format. See the "configuration"
+document for details on the directory structure the archive uses.
+
+* Create the subdirectory ``/opt/sc4s/tls``.  This will be used as a mount point for custom TLS certificates
+(if the optional mount is uncommented above). 
 
-* IMPORTANT:  When creating the two directories above, ensure the directories created match the volume mounts specified in the
+* IMPORTANT:  When creating the directories above, ensure the directories created match the volume mounts specified in the
 `docker-compose.yml` file.  Failure to do this will cause SC4S to abort at startup.
 
 ## Configure the SC4S environment
@@ -116,13 +125,16 @@ Log paths are preconfigured to utilize a convention of index destinations that a
 * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your
 environment. Simply uncomment the relevant line and enter the desired index.  The "Sources" document details the specific entries in
 this table that pertain to the individual data source filters that are included with SC4S.
+* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well.  This is an advanced topic, and further
+information is covered in the "Log Path overrides" section of the Configuration document.
 
 
 ## Configure source filtering by source IP or host name
 
 Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps
 apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. 
 
+* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start.
 * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type.  
 * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax.  Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. 
 * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment.  In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`.
@@ -132,8 +144,8 @@ apply to support such sources. To identify sources that require this step, refer
 In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion.
 The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose.  These operate similarly to
 the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file
-lists one or more metadata items that can be overridden based on the filter name.  This is an advanced topic, and further information is in
-the "Configuration" section.
+lists one or more metadata items that can be overridden based on the filter name.  This is an advanced topic, and further information is
+covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document.
 
 ## Start/Restart SC4S
 

docs/gettingstarted/docker-systemd-general.md

@@ -40,8 +40,10 @@ Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.
 
 # Mount point for local disk buffer (required)
 Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z"
+# Uncomment the following line if local disk archiving is desired
+# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z"
 # Uncomment the following line if custom TLS certs are provided
-# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls"
+# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z"
 
 TimeoutStartSec=0
 Restart=always
@@ -56,6 +58,8 @@ ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp \
         --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \
+        "$SC4S_LOCAL_ARCHIVE_MOUNT" \
+        "$SC4S_TLS_DIR" \
         --name SC4S --rm \
 $SC4S_IMAGE
 ```
@@ -79,9 +83,16 @@ of events in the event of network failure to the Splunk infrastructure.
     * This directory will populate with the disk buffer files upon SC4S startup.  If SC4S restarts for any reason, a new
 set of files will be created in addition to the original ones.  _The original ones will not be removed_.
 If you are sure, after stopping SC4S, that all data has been sent, these files can be removed.  They will be created
-again upon restart.
+again upon restart
+
+* Create the subdirectory ``/opt/sc4s/archive``.  This will be used as a mount point for local storage of syslog events
+(if the optional mount is uncommented above).  The events will be written in the syslog-ng EWMM format. See the "configuration"
+document for details on the directory structure the archive uses.
+
+* Create the subdirectory ``/opt/sc4s/tls``.  This will be used as a mount point for custom TLS certificates
+(if the optional mount is uncommented above). 
 
-* IMPORTANT:  When creating the two directories above, ensure the directories created match the volume mounts specified in the
+* IMPORTANT:  When creating the directories above, ensure the directories created match the volume mounts specified in the
 unit file above.  Failure to do this will cause SC4S to abort at startup.
 
 ## Configure the SC4S environment
@@ -112,12 +123,15 @@ Log paths are preconfigured to utilize a convention of index destinations that a
 * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your
 environment. Simply uncomment the relevant line and enter the desired index.  The "Sources" document details the specific entries in
 this table that pertain to the individual data source filters that are included with SC4S.
+* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well.  This is an advanced topic, and further
+information is covered in the "Log Path overrides" section of the Configuration document.
 
 ## Configure source filtering by source IP or host name
 
 Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps
 apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. 
 
+* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start.
 * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type.  
 * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax.  Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. 
 * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment.  In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`.
@@ -127,8 +141,8 @@ apply to support such sources. To identify sources that require this step, refer
 In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion.
 The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose.  These operate similarly to
 the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file
-lists one or more metadata items that can be overridden based on the filter name.  This is an advanced topic, and further information is in
-the "Configuration" section.
+lists one or more metadata items that can be overridden based on the filter name.  This is an advanced topic, and further information is
+covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document.
 
 ## Configure SC4S for systemd and start SC4S
 

docs/gettingstarted.md → docs/gettingstarted/index.md

@@ -1,12 +1,14 @@
-# Getting Started
+# Before you start
+
+## Getting Started
 
 Splunk Connect for Syslog is a containerized distribution of syslog-ng with a configuration framework
 designed to simplify getting syslog data into Splunk Enterprise and Splunk Cloud. Our approach is
 to provide a runtime-agnostic solution allowing customers to deploy using the container runtime
 environment of choice.
 
 
-# Planning Deployment
+## Planning Deployment
 
 Syslog is an overloaded term that refers to multiple message formats AND optionally a wire protocol for
 transmission of events between computer systems over UDP, TCP, or TLS. The protocol is designed to minimize
@@ -24,13 +26,13 @@ environment.
 * Plan for appropriately sized hardware (see)[performance.md]
 
 
-# Implementation
+## Implementation
 
-## Splunk Setup
+### Splunk Setup
 
-### Create Indexes
+#### Create Indexes
 
-SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create them in Splunk when 
+SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create them in Splunk when
 using the SC4S defaults. SC4S can be easily customized to use different indexes if desired.
 
 * email
@@ -42,15 +44,15 @@ using the SC4S defaults. SC4S can be easily customized to use different indexes
 * netipam
 * em_metrics (ensure this is created as a metrics index)
 
-### Install Related Splunk Apps
+#### Install Related Splunk Apps
 
 Install the following:
 
 * [Splunk App for Infrastructure](https://splunkbase.splunk.com/app/3975/)
 * [Splunk Add-on for Infrastructure](https://splunkbase.splunk.com/app/4217/)
 * [Splunk Metrics Workspace](https://splunkbase.splunk.com/app/4192/) *NOTE Included in Splunk 7.3.0 and above*
 
-### Configure the Splunk HTTP Event Collector
+#### Configure the Splunk HTTP Event Collector
 
 - Set up the Splunk HTTP Event Collector with the HEC endpoints behind a load balancer (VIP) configured for https round robin *WITHOUT* sticky
 session.  Alternatively, a list of HEC endpoint URLs can be configured in SC4S (native Syslog-ng load balancing) if no load balancer is in place.  In either case, it is
@@ -61,24 +63,24 @@ event destinations.
 or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your
 Splunk type.
 
-## Implement a Container Runtime and SC4S
+### Implement a Container Runtime and SC4S
 
-### Prerequisites
+#### Prerequisites
 
 * Linux host with Docker (CE 19.x or greater with Docker Swarm) or Podman enabled, depending on runtime choice (below).
 * A network load balancer (NLB) configured for round robin. Note: Special consideration may be required when more advanced products are used. The optimal configuration of the load balancer will round robin each http POST request (not each connection).
 
-### Select a Container Runtime and SC4S Configuration
+#### Select a Container Runtime and SC4S Configuration
 
 | Container and Orchestration | Notes |
 |-----------------------------|-------|
 | [Podman + systemd single node](gettingstarted/podman-systemd-general.md) | First choice for RedHat 7.x/8.x and CentOS, second choice for Debian and Ubuntu (packages provided via PPA) |
 | [Docker CE + systemd single node](gettingstarted/docker-systemd-general.md) | First choice for Debian and Ubuntu; second choice for CentOS for those with limited existing Docker experience |
 | [Docker CE + Swarm single node](gettingstarted/docker-swarm-general.md) | Option for Debian, Ubuntu, CentOS, and Desktop Docker desiring Docker Compose or Swarm orchestration |
 | [Docker CE + Swarm single node RHEL 7.7](gettingstarted/docker-swarm-rhel7.md) | Option for RedHat 7.7 desiring Docker Compose or Swarm orchestration |
-| [Bring your own Envionment](gettingstarted/byoe-rhel7.md) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers | 
+| [Bring your own Envionment](gettingstarted/byoe-rhel7.md) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers |
 
-## Offline Container Installation
+### Offline Container Installation
 
 Follow these instructions to "stage" SC4S by downloading the container so that it can be loaded "out of band" on a
 host machine, such as an airgapped system, without internet connectivity.
@@ -116,7 +118,7 @@ attempt to obtain the container image via the internet.
 Environment="SC4S_IMAGE=sc4slocal:latest"
 ```
 
-# Scale out
+## Scale out
 
 Additional hosts can be deployed for syslog collection from additional network zones and locations: