Runtime docs cleanup

* Clean up `docker-compose.yml` and systemd `unit` files to include archive options * Clean up compose/unit file command layout * Clarify instructions for unique port setup
CSVD · Jan 12, 2020 · f52c5aa · f52c5aa
1 parent fd77a51
commit f52c5aa
Show file tree

Hide file tree

Showing 4 changed files with 104 additions and 78 deletions.
diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md
@@ -95,7 +95,8 @@ document for details on the directory structure the archive uses.
 
 # Configure the SC4S environment
 
-Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
+SC4S is almost entirely controlled through environment variables, which are read from a file at starteup.  Create a file named
+``/opt/sc4s/env_file`` and add the following environment variables and values:
 
 ```dotenv
 SPLUNK_HEC_URL=https://splunk.smg.aws:8088
@@ -107,8 +108,8 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment.
 
-* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints.  If the endpoint is a VIP,
-match this value to the total number of indexers behind the load balancer.
+* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32.
+If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer.
 
 * NOTE:  Splunk Connect for Syslog defaults to secure configurations.  If you are not using trusted SSL certificates, be sure to
 uncomment the last line in the example above.
@@ -135,18 +136,18 @@ No changes to the underlying SC4S default configuration (environment variables)
 
 For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in
 the data.  In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise.  
-For collection of such sources we provide a means of dedicating a unique listening port to a specific source.
-
-Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology
-in use.
+For collection of such sources, we provide a means of dedicating a unique listening port to a specific source.
 
 The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured
 by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the
-``target`` and ``published`` lines provide for 21 additional technology-specific ports. Follow these steps to configure unique ports:
+``target`` and ``published`` lines provide for 21 additional technology-specific UDP and TCP ports. 
+
+Follow these steps to configure unique ports:
 
-* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s).  See the "Sources" 
-section for more information on your specific device(s).
-* Modify the unit file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below:
+* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources"
+documentation to identify the specific environment variables that are mapped to each data source vendor/technology.
+* Modify the compose file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below.
+* Restart SC4S using the command in the "Start/Restart SC4S" section below.
 ```yaml
 version: "3.7"
 services:
@@ -183,8 +184,10 @@ services:
     volumes:
       - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z
       - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z
-#Uncomment the following line if custom TLS certs are provided
-#     - /opt/sc4s/tls:/opt/syslog-ng/tls
+# Uncomment the following line if local disk archiving is desired
+#     - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z
+# Uncomment the following line if custom TLS certs are provided
+#     - /opt/sc4s/tls:/opt/syslog-ng/tls:z
 ```
 
 ## Modify index destinations for Splunk 

diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md
@@ -103,7 +103,8 @@ document for details on the directory structure the archive uses.
 
 # Configure the SC4S environment
 
-Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
+SC4S is almost entirely controlled through environment variables, which are read from a file at starteup.  Create a file named
+``/opt/sc4s/env_file`` and add the following environment variables and values:
 
 ```dotenv
 SPLUNK_HEC_URL=https://splunk.smg.aws:8088
@@ -115,11 +116,11 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment.
 
-* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints.  If the endpoint is a VIP,
-match this value to the total number of indexers behind the load balancer.
+* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32.
+If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer.
 
 * NOTE:  Splunk Connect for Syslog defaults to secure configurations.  If you are not using trusted SSL certificates, be sure to
-uncomment the last line in the example below.
+uncomment the last line in the example above.
 
 ## Configure SC4S Listening Ports
 
@@ -143,18 +144,18 @@ No changes to the underlying SC4S default configuration (environment variables)
 
 For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in
 the data.  In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise.  
-For collection of such sources we provide a means of dedicating a unique listening port to a specific source.
-
-Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology
-in use.
+For collection of such sources, we provide a means of dedicating a unique listening port to a specific source.
 
 The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured
 by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the
-``target`` and ``published`` lines provide for 21 additional technology-specific ports. Follow these steps to configure unique ports:
+``target`` and ``published`` lines provide for 21 additional technology-specific UDP and TCP ports. 
+
+Follow these steps to configure unique ports:
 
-* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s).  See the "Sources" 
-section for more information on your specific device(s).
-* Modify the unit file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below:
+* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources"
+documentation to identify the specific environment variables that are mapped to each data source vendor/technology.
+* Modify the compose file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below.
+* Restart SC4S using the command in the "Start/Restart SC4S" section below.
 ```yaml
 version: "3.7"
 services:
@@ -191,8 +192,10 @@ services:
     volumes:
       - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z
       - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z
-#Uncomment the following line if custom TLS certs are provided
-#     - /opt/sc4s/tls:/opt/syslog-ng/tls
+# Uncomment the following line if local disk archiving is desired
+#     - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z
+# Uncomment the following line if custom TLS certs are provided
+#     - /opt/sc4s/tls:/opt/syslog-ng/tls:z
 ```
 
 ## Modify index destinations for Splunk 

diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md
@@ -38,10 +38,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest"
 
 Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z"
 
+# Optional mount point for local disk archive (EWMM output) files
+
+# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z"
+
 # Mount point for local disk buffer (required)
 Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z"
-# Uncomment the following line if local disk archiving is desired
-# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z"
+
 # Uncomment the following line if custom TLS certs are provided
 # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z"
 
@@ -52,16 +55,16 @@ ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE
 ExecStartPre=/usr/bin/docker run \
         --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
-        --name SC4S_preflight --rm \
-        $SC4S_IMAGE -s
-ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 \
+        --name SC4S_preflight \
+        --rm $SC4S_IMAGE -s
+ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 \
         --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \
         "$SC4S_LOCAL_ARCHIVE_MOUNT" \
         "$SC4S_TLS_DIR" \
-        --name SC4S --rm \
-$SC4S_IMAGE
+        --name SC4S \
+        --rm $SC4S_IMAGE
 ```
 
 * Create the subdirectory ``/opt/sc4s/local``.  This will be used as a mount point for local overrides and configurations.
@@ -95,9 +98,10 @@ document for details on the directory structure the archive uses.
 * IMPORTANT:  When creating the directories above, ensure the directories created match the volume mounts specified in the
 unit file above.  Failure to do this will cause SC4S to abort at startup.
 
-# Configure the SC4S environment
+# Configure the sc4s environment
 
-Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
+SC4S is almost entirely controlled through environment variables, which are read from a file at starteup.  Create a file named
+``/opt/sc4s/env_file`` and add the following environment variables and values:
 
 ```dotenv
 SPLUNK_HEC_URL=https://splunk.smg.aws:8088
@@ -109,18 +113,18 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment
 
-* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints.  If the endpoint is a VIP,
-match this value to the total number of indexers behind the load balancer.
+* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32.
+If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer.
 
-* NOTE:  Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to
-uncomment the last line in the example.
+* NOTE:  Splunk Connect for Syslog defaults to secure configurations.  If you are not using trusted SSL certificates, be sure to
+uncomment the last line in the example above.
 
 ## Configure SC4S Listening Ports
 
 Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS.
 The unit file and standard SC4S configurations reflect these defaults.  If it desired to change some or all of them, container port mapping
 can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the initial port in the
-`ExecStart` line for the main container (which represents the actual listening port on the host machine), like so:
+`ExecStart` line in the unit file for the main container (which represents the actual listening port on the host machine), like so:
 
 ```
 -p 614:514 -p 714:514/udp -p 8514:6514
@@ -132,18 +136,18 @@ on the _container_.  No changes to the underlying SC4S default configuration (en
 
 For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in
 the data.  In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise.  
-For collection of such sources we provide a means of dedicating a unique listening port to a specific source.
-
-Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology
-in use.
+For collection of such sources, we provide a means of dedicating a unique listening port to a specific source.
 
 The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the
-environment variable(s). In the following example, the `ExecStart` line for the main SC4S container is modified, where 
-``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Follow these steps to configure unique ports:
+environment variable(s). In the example below, the `ExecStart` line for the main SC4S container is modified, where 
+``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. 
 
-* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). See the "Sources" 
-section for more information on your specific device(s).
-* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below:
+Follow these steps to configure unique ports:
+
+* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources"
+documentation to identify the specific environment variables that are mapped to each data source vendor/technology.
+* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below.
+* Ensure that you reload the unit file as well as restarting SC4S. See the "Configure SC4S for systemd and start SC4S" section below.
 ```ini
 [Unit]
 Description=SC4S Container
@@ -157,8 +161,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest"
 
 Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z"
 
+# Optional mount point for local disk archive (EWMM output) files
+
+# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z"
+
 # Mount point for local disk buffer (required)
 Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z"
+
 # Uncomment the following line if custom TLS certs are provided
 # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls"
 
@@ -168,15 +177,16 @@ ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE
 ExecStartPre=/usr/bin/docker run \
         --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
-        --name SC4S_preflight --rm \
-        $SC4S_IMAGE -s
+        --name SC4S_preflight \
+        --rm $SC4S_IMAGE -s
 ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \
         --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \
+        "$SC4S_LOCAL_ARCHIVE_MOUNT" \
+        "$SC4S_TLS_DIR" \
         --name SC4S \
-        --rm \
-$SC4S_IMAGE
+        --rm $SC4S_IMAGE
 ```
 
 ## Modify index destinations for Splunk