From d3cc05c9633522da0d801bdc1b9429a1b10c90a1 Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Tue, 11 Feb 2020 16:36:21 -0800 Subject: [PATCH 1/3] * Add detail on disk space needed for container volume * Add description to configuration and runtime docs for properly accounting for container volume growth * Revert change to VMware nav to properly match that of referenced directory/file. * Change "Developing" to "Development" * Update "Upgrade" for 1.1.0 changes --- docs/configuration.md | 13 +++++++++++-- docs/developing/index.md | 17 +++++++++-------- docs/gettingstarted/docker-swarm-general.md | 4 ++++ docs/gettingstarted/docker-swarm-rhel7.md | 4 ++++ docs/gettingstarted/docker-systemd-general.md | 4 ++++ docs/gettingstarted/podman-systemd-general.md | 4 ++++ docs/upgrade.md | 4 ++++ mkdocs.yml | 4 ++-- 8 files changed, 42 insertions(+), 12 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 75b7e82..1343573 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -26,15 +26,24 @@ and variables needed to properly configure SC4S for your environment. Disk buffers in SC4S are allocated _per destination_. In the future as more destinations are supported, a separate list of variables will be used for each. This is why you see the `DEST_SPLUNK_HEC` in the variable names below. + * NOTE: "Reliable" disk buffering offers little advantage over "normal" disk buffering, at a significant performance penalty. For this reason, normal disk buffering is recommended. + * NOTE: If you add destinations locally in your configuration, pay attention to the _cumulative_ buffer requirements when allocating local disk. -* Be sure to factor in the syslog-ng overhead (approx. 1.7x) when calculating the total buffer size needed. See the "Data Resilience" section below for more information. + +* NOTE: Disk buffer storage is configured via container volumes and is persistent between restarts of the conatiner. +Be sure to account for disk space requirements on the local sc4s host when creating the container volumes in your respective +runtime environment (outlined in the "getting started" runtime docs). These volumes can grow significantly if there is +an extended outage to the SC4S destinations (HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation +page for more info. + * NOTE: The values for the variables below represent the _total_ sizes of the buffers for the destination. These sizes are divded by the number of workers (threads) when setting the actual syslog-ng buffer options, because the buffer options apply to each worker rather than the entire destination. Pay careful attention to this when using the "BYOE" version of SC4S, where direct access to the syslog-ng config files -may hide this nuance. To determine the proper size of the disk buffer, consult the "Data Resilience" section below. +may hide this nuance. Lastly, be sure to factor in the syslog-ng data structure overhead (approx. 2x raw message size) when calculating the +total buffer size needed. To determine the proper size of the disk buffer, consult the "Data Resilience" section below. | Variable | Values/Default | Description | |----------|---------------|-------------| diff --git a/docs/developing/index.md b/docs/developing/index.md index 5cf9063..d462d1d 100644 --- a/docs/developing/index.md +++ b/docs/developing/index.md @@ -2,17 +2,17 @@ ## Get Docker -Development requires Docker desktop available for windows + and mac or Docker CE available for Linux visit (Docker)[https://www.docker.com/get-started] +Development requires Docker desktop available for windows + and mac or Docker CE available for Linux. Visit (Docker)[https://www.docker.com/get-started] for download instructions ## Setup VS Code IDE -VS Code provides a free IDE experience that is effective for daily development with SC4S visit (Microsoft)[https://code.visualstudio.com/docs/introvideos/basics] +VS Code provides a free IDE experience that is effective for daily development with SC4S. Visit (Microsoft)[https://code.visualstudio.com/docs/introvideos/basics] to download and install for your plaform (windows/mac/linux) ## Fork and Clone the github repository -Visit our repository at (Github)[https://github.com/splunk/splunk-connect-for-syslog] and "fork" our repository this will allow you to make changes and submit pull requests. +Visit our repository at (Github)[https://github.com/splunk/splunk-connect-for-syslog] and "fork" our repository. This will allow you to make changes and submit pull requests. ![How to Fork](gh_fork.png) @@ -22,23 +22,24 @@ Click the clone icon and select the location ## Setup the project and install requirements -The follow steps are only required on the first time run. +The following steps are required _only_ on the first time run. * Install VS Code Extensions S * Python * Test Explorer * "Python Test Explorer" -* From the terminal menu select Run Task +* From the terminal menu select "Run Task" * Select "Setup step 1: python venv" then "go without scanning output" -* From the terminal menu select Run Task +* From the terminal menu select "Run Task" * Select "Setup step 2: python requirements" then "go without scanning output" ![VS Code setup](vsc_run.png) ## Click the test lab icon -* Run all tests this will appear to do nothing for a period system fan may spin loud whiletests are run icons on each test will turn green -or red to indicate pass fail however VS Code does not show the status of status until the tests complete +* Run all tests. Icons on each test will turn green or red to indicate pass or fail. Though VS Code does not show the status +of any given test until all tests complete in the test tree, you can select "Show test output" near the top of the test +directory tree to see the terminal output of each test as it runs in the "Output" pane. ![VS Code Debug](vsc_debug.png) diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index 0d7dc62..b9dc3a8 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -69,6 +69,10 @@ particular the state of the disk buffer. This is a required step. sudo docker volume create splunk-sc4s-var ``` +* NOTE: Be sure to account for disk space requirements for the docker volume created above. This volume is located in +`/var/lib/docker/volumes/` and could grow significantly if there is an extended outage to the SC4S destinations +(typically HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation page for more info. + * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. * The empty ``local`` directory created above will populate with defaults and examples at the first invocation diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index af6d519..58b4ffe 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -77,6 +77,10 @@ particular the state of the disk buffer. This is a required step. sudo docker volume create splunk-sc4s-var ``` +* NOTE: Be sure to account for disk space requirements for the docker volume created above. This volume is located in +`/var/lib/docker/volumes/` and could grow significantly if there is an extended outage to the SC4S destinations +(typically HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation page for more info. + * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. * The empty ``local`` directory created above will populate with defaults and examples at the first invocation diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 132e80b..2d3b0b6 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -75,6 +75,10 @@ particular the state of the disk buffer. This is a required step. sudo docker volume create splunk-sc4s-var ``` +* NOTE: Be sure to account for disk space requirements for the docker volume created above. This volume is located in +`/var/lib/docker/volumes/` and could grow significantly if there is an extended outage to the SC4S destinations +(typically HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation page for more info. + * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. * The empty ``local`` directory created above will populate with defaults and examples at the first invocation diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 8988b50..e4ab8c0 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -57,6 +57,10 @@ particular the state of the disk buffer. This is a required step. sudo podman volume create splunk-sc4s-var ``` +* NOTE: Be sure to account for disk space requirements for the podman volume created above. This volume is located in +`/var/lib/containers/storage/volumes/` and could grow significantly if there is an extended outage to the SC4S destinations +(typically HEC endpoints). See the "SC4S Disk Buffer Configuration" section on the Configruation page for more info. + * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. * The empty ``local`` directory created above will populate with defaults and examples at the first invocation diff --git a/docs/upgrade.md b/docs/upgrade.md index 2491f14..a0a3dde 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -13,3 +13,7 @@ files is incorporated into existing context files. * UNIT FILE CHANGES: Make sure to update the unit file used to start the sc4s service with the changes included in this release. It includes updates for proper operation with RHEL 8, and is backward-compatible with RHEL 7.7. + +## Version 1.10.0 + +* The "Development" section outlines new instructions for operation with the vscode IDE. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 6a73268..98c30ca 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -10,7 +10,7 @@ nav: - "Docker CE + Swarm RHEL 7.7": "gettingstarted/docker-swarm-rhel7.md" - "Bring your own Envionment": "gettingstarted/byoe-rhel7.md" - Configuration: "configuration.md" - - Developing: "docs/developing/index.md" + - Development: "docs/developing/index.md" - Sources: - About: sources/index.md - Checkpoint: sources/Checkpoint/index.md @@ -29,7 +29,7 @@ nav: - Proofpoint: sources/Proofpoint/index.md - Symantec: sources/Symantec/index.md - Ubiquiti: sources/Ubiquiti/index.md - - VMware: sources/VMWare/index.md + - VMware: sources/VMware/index.md - Zscaler: sources/Zscaler/index.md - "Demo Lab": "demo.md" - Performance: "performance.md" From e53ed1a37d78b2c37af35e4d596a3856d43cbc1b Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Wed, 12 Feb 2020 15:46:26 -0800 Subject: [PATCH 2/3] Update to clarify HEC Ack * Include note to warn not to set HEC Ack when configuring the HEC token on the Splunk side. --- docs/configuration.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 1343573..abdf261 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -10,6 +10,9 @@ and variables needed to properly configure SC4S for your environment. | SPLUNK_HEC_URL | url | URL(s) of the Splunk endpoint, can be a single URL space seperated list | | SPLUNK_HEC_TOKEN | string | Splunk HTTP Event Collector Token | +* NOTE: Do _not_ configure HEC Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http +destination does not support this feature. Moreover, HEC would significantly degrade performance for streaming data such as syslog. + ## Splunk HEC Destination Configuration From 121de7b309a6bdd3c8818d7ef4bb863422bb5265 Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Wed, 12 Feb 2020 15:52:53 -0800 Subject: [PATCH 3/3] Update to include HEC Ack warning in runtime docs * Update runtime docs to warn against setting HEC Ack --- docs/configuration.md | 3 ++- docs/gettingstarted/docker-swarm-general.md | 4 +++- docs/gettingstarted/docker-swarm-rhel7.md | 4 +++- docs/gettingstarted/docker-systemd-general.md | 4 +++- docs/gettingstarted/podman-systemd-general.md | 4 +++- 5 files changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index abdf261..9860a3c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -11,7 +11,8 @@ and variables needed to properly configure SC4S for your environment. | SPLUNK_HEC_TOKEN | string | Splunk HTTP Event Collector Token | * NOTE: Do _not_ configure HEC Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http -destination does not support this feature. Moreover, HEC would significantly degrade performance for streaming data such as syslog. +destination does not support this feature. Moreover, HEC Ack would significantly degrade performance for streaming data such as +syslog. ## Splunk HEC Destination Configuration diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index b9dc3a8..4d38ea2 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -111,7 +111,9 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. +* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. Do _not_ configure HEC +Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http destination does not support this +feature. Moreover, HEC Ack would significantly degrade performance for streaming data such as syslog. * Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index 58b4ffe..1c02bbb 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -119,7 +119,9 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. +* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. Do _not_ configure HEC +Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http destination does not support this +feature. Moreover, HEC Ack would significantly degrade performance for streaming data such as syslog. * Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 2d3b0b6..bdbb78d 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -117,7 +117,9 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. Do _not_ configure HEC +Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http destination does not support this +feature. Moreover, HEC Ack would significantly degrade performance for streaming data such as syslog. * Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index e4ab8c0..8c79df8 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -99,7 +99,9 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. Do _not_ configure HEC +Acknowledgement when deploying the HEC token on the Splunk side; the underlying syslog-ng http destination does not support this +feature. Moreover, HEC Ack would significantly degrade performance for streaming data such as syslog. * Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer.