Merge pull request #321 from splunk/docs/persist_volume

* Add detail on disk space needed for container volume

docs/configuration.md

@@ -10,6 +10,10 @@ and variables needed to properly configure SC4S for your environment.
 | SPLUNK_HEC_URL | url | URL(s) of the Splunk endpoint, can be a single URL space seperated list |
 | SPLUNK_HEC_TOKEN | string | Splunk HTTP Event Collector Token |
 
+* NOTE:  Do _not_ configure HEC Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http
+destination does not support this feature.  Moreover, HEC Ack would significantly degrade performance for streaming data such as
+syslog.
+
 
 ## Splunk HEC Destination Configuration
 
@@ -26,15 +30,24 @@ and variables needed to properly configure SC4S for your environment.
 
 Disk buffers in SC4S are allocated _per destination_.  In the future as more destinations are supported, a separate list of variables
 will be used for each.  This is why you see the `DEST_SPLUNK_HEC` in the variable names below.
+
 * NOTE:  "Reliable" disk buffering offers little advantage over "normal" disk buffering, at a significant performance penalty.
 For this reason, normal disk buffering is recommended.
+
 * NOTE:  If you add destinations locally in your configuration, pay attention to the _cumulative_ buffer requirements when allocating local
 disk.
-* Be sure to factor in the syslog-ng overhead (approx. 1.7x) when calculating the total buffer size needed.  See the "Data Resilience" section below for more information.
+
+* NOTE:  Disk buffer storage is configured via container volumes and is persistent between restarts of the conatiner.
+Be sure to account for disk space requirements on the local sc4s host when creating the container volumes in your respective
+runtime environment (outlined in the "getting started" runtime docs). These volumes can grow significantly if there is
+an extended outage to the SC4S destinations (HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation
+page for more info.
+
 * NOTE:  The values for the variables below represent the _total_ sizes of the buffers for the destination.  These sizes are divded by the
 number of workers (threads) when setting the actual syslog-ng buffer options, because the buffer options apply to each worker rather than the
 entire destination.  Pay careful attention to this when using the "BYOE" version of SC4S, where direct access to the syslog-ng config files
-may hide this nuance.  To determine the proper size of the disk buffer, consult the "Data Resilience" section below.
+may hide this nuance.  Lastly, be sure to factor in the syslog-ng data structure overhead (approx. 2x raw message size) when calculating the
+total buffer size needed. To determine the proper size of the disk buffer, consult the "Data Resilience" section below.
 
 | Variable | Values/Default   | Description |
 |----------|---------------|-------------|

docs/developing/index.md

@@ -2,17 +2,17 @@
 
 ## Get Docker
 
-Development requires Docker desktop available for windows + and mac or Docker CE available for Linux visit (Docker)[https://www.docker.com/get-started]
+Development requires Docker desktop available for windows + and mac or Docker CE available for Linux. Visit (Docker)[https://www.docker.com/get-started]
 for download instructions
 
 ## Setup VS Code IDE
 
-VS Code provides a free IDE experience that is effective for daily development with SC4S visit (Microsoft)[https://code.visualstudio.com/docs/introvideos/basics]
+VS Code provides a free IDE experience that is effective for daily development with SC4S. Visit (Microsoft)[https://code.visualstudio.com/docs/introvideos/basics]
 to download and install for your plaform (windows/mac/linux)
 
 ## Fork and Clone the github repository
 
-Visit our repository at (Github)[https://github.com/splunk/splunk-connect-for-syslog] and "fork" our repository this will allow you to make changes and submit pull requests.
+Visit our repository at (Github)[https://github.com/splunk/splunk-connect-for-syslog] and "fork" our repository. This will allow you to make changes and submit pull requests.
 
 ![How to Fork](gh_fork.png)
 
@@ -22,23 +22,24 @@ Click the clone icon and select the location
 
 ## Setup the project and install requirements
 
-The follow steps are only required on the first time run.
+The following steps are required _only_ on the first time run.
 
 * Install VS Code Extensions S
     * Python
     * Test Explorer
     * "Python Test Explorer"
-* From the terminal menu select Run Task
+* From the terminal menu select "Run Task"
 * Select "Setup step 1: python venv" then "go without scanning output"
-* From the terminal menu select Run Task
+* From the terminal menu select "Run Task"
 * Select "Setup step 2: python requirements" then "go without scanning output"
 
 ![VS Code setup](vsc_run.png)
 
 ## Click the test lab icon
 
-* Run all tests this will appear to do nothing for a period system fan may spin loud whiletests are run icons on each test will turn green
-or red to indicate pass fail however VS Code does not show the status of status until the tests complete
+* Run all tests. Icons on each test will turn green or red to indicate pass or fail.  Though VS Code does not show the status 
+of any given test until all tests complete in the test tree, you can select "Show test output" near the top of the test
+directory tree to see the terminal output of each test as it runs in the "Output" pane.
 
 ![VS Code Debug](vsc_debug.png)
 

docs/gettingstarted/docker-swarm-general.md

@@ -69,6 +69,10 @@ particular the state of the disk buffer.  This is a required step.
 sudo docker volume create splunk-sc4s-var
 ```
 
+* NOTE:  Be sure to account for disk space requirements for the docker volume created above. This volume is located in
+`/var/lib/docker/volumes/` and could grow significantly if there is an extended outage to the SC4S destinations
+(typically HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation page for more info.
+
 * Create the subdirectory ``/opt/sc4s/local``.  This will be used as a mount point for local overrides and configurations.
 
     * The empty ``local`` directory created above will populate with defaults and examples at the first invocation 
@@ -107,7 +111,9 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
 
-* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment.
+* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment.  Do _not_ configure HEC
+Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http destination does not support this
+feature.  Moreover, HEC Ack would significantly degrade performance for streaming data such as syslog.
 
 * Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32.
 If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer.

docs/gettingstarted/docker-swarm-rhel7.md

@@ -77,6 +77,10 @@ particular the state of the disk buffer.  This is a required step.
 sudo docker volume create splunk-sc4s-var
 ```
 
+* NOTE:  Be sure to account for disk space requirements for the docker volume created above. This volume is located in
+`/var/lib/docker/volumes/` and could grow significantly if there is an extended outage to the SC4S destinations
+(typically HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation page for more info.
+
 * Create the subdirectory ``/opt/sc4s/local``.  This will be used as a mount point for local overrides and configurations.
 
     * The empty ``local`` directory created above will populate with defaults and examples at the first invocation 
@@ -115,7 +119,9 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
 
-* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment.
+* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment.  Do _not_ configure HEC
+Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http destination does not support this
+feature.  Moreover, HEC Ack would significantly degrade performance for streaming data such as syslog.
 
 * Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32.
 If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer.

docs/gettingstarted/docker-systemd-general.md

@@ -75,6 +75,10 @@ particular the state of the disk buffer.  This is a required step.
 sudo docker volume create splunk-sc4s-var
 ```
 
+* NOTE:  Be sure to account for disk space requirements for the docker volume created above. This volume is located in
+`/var/lib/docker/volumes/` and could grow significantly if there is an extended outage to the SC4S destinations
+(typically HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation page for more info.
+
 * Create the subdirectory ``/opt/sc4s/local``.  This will be used as a mount point for local overrides and configurations.
 
     * The empty ``local`` directory created above will populate with defaults and examples at the first invocation 
@@ -113,7 +117,9 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
 
-* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment
+* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment.  Do _not_ configure HEC
+Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http destination does not support this
+feature.  Moreover, HEC Ack would significantly degrade performance for streaming data such as syslog.
 
 * Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32.
 If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer.

docs/gettingstarted/podman-systemd-general.md

@@ -57,6 +57,10 @@ particular the state of the disk buffer.  This is a required step.
 sudo podman volume create splunk-sc4s-var
 ```
 
+* NOTE:  Be sure to account for disk space requirements for the podman volume created above. This volume is located in
+`/var/lib/containers/storage/volumes/` and could grow significantly if there is an extended outage to the SC4S destinations
+(typically HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation page for more info.
+
 * Create the subdirectory ``/opt/sc4s/local``.  This will be used as a mount point for local overrides and configurations.
 
     * The empty ``local`` directory created above will populate with defaults and examples at the first invocation 
@@ -95,7 +99,9 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
 
-* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment
+* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment.  Do _not_ configure HEC
+Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http destination does not support this
+feature.  Moreover, HEC Ack would significantly degrade performance for streaming data such as syslog.
 
 * Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32.
 If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer.

docs/upgrade.md

@@ -13,3 +13,7 @@ files is incorporated into existing context files.
 
 * UNIT FILE CHANGES:  Make sure to update the unit file used to start the sc4s service with the changes included in this release. It
 includes updates for proper operation with RHEL 8, and is backward-compatible with RHEL 7.7.
+
+## Version 1.10.0
+
+* The "Development" section outlines new instructions for operation with the vscode IDE.

mkdocs.yml

@@ -10,7 +10,7 @@ nav:
       - "Docker CE + Swarm RHEL 7.7": "gettingstarted/docker-swarm-rhel7.md"
       - "Bring your own Envionment": "gettingstarted/byoe-rhel7.md"
   - Configuration: "configuration.md"
-  - Developing: "docs/developing/index.md"
+  - Development: "docs/developing/index.md"
   - Sources:
       - About: sources/index.md
       - Checkpoint: sources/Checkpoint/index.md
@@ -29,7 +29,7 @@ nav:
       - Proofpoint: sources/Proofpoint/index.md
       - Symantec: sources/Symantec/index.md
       - Ubiquiti: sources/Ubiquiti/index.md
-      - VMware: sources/VMWare/index.md
+      - VMware: sources/VMware/index.md
       - Zscaler: sources/Zscaler/index.md
   - "Demo Lab": "demo.md"
   - Performance: "performance.md"