Merge branch 'release/0.10.0'

.gitignore

@@ -379,4 +379,7 @@ fabric.properties
 !/package/scripts/
 .ecs
 /test-results/
-/.idea/
+/.idea/
+
+tests/test_plugin_*.py
+package/etc/conf.d/local/

.gitmodules

@@ -1,5 +1,6 @@
 [submodule "package/syslog-ng"]
 	path = package/syslog-ng
 	url = https://github.com/balabit/syslog-ng.git
-[submodule "Submod"]
 	branch = syslog-ng-3.23.1
+#
+

docs/configuration.md

@@ -1,21 +1,27 @@
-# Global Configuration
+# SC4S Configuration Variables
+
+Other than device filter creation, SC4S is almost entirely controlled by environment variables.  Here are the categories
+and variables needed to properly configure SC4S for your environment.
+
+## Global Configuration
 
 | Variable | Values        | Description |
 |----------|---------------|-------------|
 | SPLUNK_HEC_URL | url | URL(s) of the Splunk endpoint, can be a single URL space seperated list |
 | SPLUNK_HEC_TOKEN | string | Splunk HTTP Event Collector Token |
 
 
-# Splunk HEC destination Configuration
+## Splunk HEC Destination Configuration
 
 | Variable | Values        | Description |
 |----------|---------------|-------------|
+| SC4S_DEST_SPLUNK_HEC_WORKERS | numeric | Number of destination workers (threads).  Set this to the number of HEC endpoints up to a max of 32. |
 | SC4S_DEST_SPLUNK_HEC_TLS_VERIFY | yes(default) or no | verify HTTP(s) certificate |
 | SC4S_DEST_SPLUNK_HEC_CIPHER_SUITE | comma separated list | Open SSL cipher suite list |
 | SC4S_DEST_SPLUNK_HEC_SSL_VERSION |  comma separated list | Open SSL version list |
 | SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE | path | Custom trusted cert file |
 
-# Syslog Source Configuration
+## Syslog Source Configuration
 
 | Variable | Values/Default | Description |
 |----------|----------------|-------------|
@@ -28,7 +34,7 @@
 | SC4S_SOURCE_UDP_SO_RCVBUFF | 425984 | UDP server buffer size in bytes |
 
 
-# Syslog Source TLS Certificate Configuration
+## Syslog Source TLS Certificate Configuration
 
 * Create a folder ``/opt/sc4s/tls``
 * Save the server private key in PEM format with NO PASSWORD to ``/opt/sc4s/tls/server.key``

docs/gettingstarted.md

@@ -13,7 +13,7 @@ transmission of events between computer systems over UDP, TCP, or TLS. The proto
 overhead on the sender favoring performance over reliability. This fundamental choice means any instability
 or resource constraint will cause data to be lost in transmission.
 
-* When practical and cost effective considering the importance of completeness as a requirement, place the sc4s
+* When practical and cost effective (considering the importance of completeness as a requirement), place the sc4s
 instance in the same VLAN as the source device.
 
 * Avoid crossing a Wireless network, WAN, Firewall, Load Balancer, or inline IDS.
@@ -29,9 +29,10 @@ environment.
 
 ### Create Indexes
 
-SC4S is pre-configured to map each sourcetype to a typical index, for new installations best practice is to create the following
-indexes in Splunk. The indexes can be customized easily if desired. If using defaults create the following indexes on Splunk:
+SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create them in Splunk when 
+using the SC4S defaults. SC4S can be easily customized to use different indexes if desired.
 
+* email
 * netauth
 * netfw
 * netids
@@ -54,7 +55,7 @@ Install the following:
 session.  Alternatively, a list of HEC endpoint URLs can be configured in SC4S if no load balancer is in place.  In either case, it is
 recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of HWFs.
 - Create a HEC token that will be used by SC4S and ensure the token has access to place events in main, em_metrics, and all indexes used as
-event destinations
+event destinations.
 - Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
 or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your
 Splunk type.
@@ -78,6 +79,6 @@ Splunk type.
 
 # Scale out
 
-Additional hosts can be deployed for syslog collection from additional network zones and locations
+Additional hosts can be deployed for syslog collection from additional network zones and locations:
 
 ![SC4S deployment diagram](SC4S%20deployment.png)

docs/gettingstarted/docker-swarm-general.md

@@ -5,7 +5,7 @@ Refer to [Getting Started](https://docs.docker.com/get-started/)
 
 # SC4S Configuration
 
-* Create a directory on the server for configuration. This should be available to all administrators, for example:
+* Create a directory on the server for local configurations. This should be available to all administrators, for example:
 ``/opt/sc4s/``
 * Create a docker-compose.yml file in the directory created above, based on the following template:
 
@@ -33,21 +33,33 @@ services:
       - /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf
 #Uncomment the following line if custom TLS certs are provided
       - /opt/sc4s/tls:/opt/syslog-ng/tls
-
 ```
 
+* NOTE:  While strictly optional, it is recommended that you create and/or download all files and directories referenced in the yml template 
+above (`volumes` declarations) according to the configuration steps that follow.  The TLS options are described in the "Configuration" section.
+Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors.
+
 ## Configure the SC4S environment
 
-Create the following file ``/opt/sc4s/env_file``
+Create the following file ``/opt/sc4s/env_file`` and add the environment variables below:
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment
 
+* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints.  If the endpoint is a VIP,
+match this value to the total number of indexers behind the load balancer.
+
+* NOTE:  Splunk Connect for Syslog defaults to secure configurations.  If you are not using trusted SSL certificates, be sure to
+uncomment the last line in the example below.
+
 ```dotenv
 SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
+SC4S_DEST_SPLUNK_HEC_WORKERS=6
 SPLUNK_CONNECT_METHOD=hec
 SPLUNK_DEFAULT_INDEX=main
 SPLUNK_METRICS_INDEX=em_metrics
+#Uncomment the following line if using untrusted SSL certificates
+#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
 
 ## Configure index destinations for Splunk 
@@ -77,13 +89,12 @@ sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/mas
 ```
 * Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax.
 
-# Start SC4S
+## Start/Restart SC4S
 
 ```bash
 docker stack deploy --compose-file docker-compose.yml sc4s
 ```
 
-
 # Scale out
 
 Additional hosts can be deployed for syslog collection from additional network zones and locations.
@@ -135,16 +146,23 @@ services:
       - /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf
 #Uncomment the following line if custom TLS certs are provided
       - /opt/sc4s/tls:/opt/syslog-ng/tls
-
 ```
 
-* Modify the following file ``/opt/sc4s/default/env_file`` after including the port-specific environment variable(s).
+* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s).  See the "Sources" 
+section for more information on your specific device(s).
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment
 
+* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints.  If the endpoint is a VIP,
+match this value to the total number of indexers behind the load balancer.
+
+* NOTE:  Splunk Connect for Syslog defaults to secure configurations.  If you are not using trusted SSL certificates, be sure to
+uncomment the last line in the example below.
+
 ```dotenv
 SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
+SC4S_DEST_SPLUNK_HEC_WORKERS=6
 SPLUNK_CONNECT_METHOD=hec
 SPLUNK_DEFAULT_INDEX=main
 SPLUNK_METRICS_INDEX=em_metrics
@@ -153,8 +171,57 @@ SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
 
-* Start SC4S.
+* Restart SC4S (below)
+
+## Start/Restart SC4S
 
 ```bash
 docker stack deploy --compose-file docker-compose.yml sc4s
 ```
+
+# Stop SC4S
+
+Start by obtaining the stack name (ID):
+```bash
+docker stack ls
+```
+Then, remove the stack:
+```bash
+docker stack rm <ID>
+```
+# Verify Proper Operation
+
+SC4S has a number of "preflight" checks to ensure that the container starts properly and that the syntax of the underlying syslog-ng
+configuration is correct.  After this step completes, to verify SC4S is properly communicating with Splunk,
+execute the following search in Splunk:
+
+```ini
+index=* sourcetype=sc4s:events "starting up"
+```
+This should yield the following event:
+```ini
+syslog-ng starting up; version='3.22.1'
+``` 
+when the startup process proceeds normally (without syntax errors). If you do not see this,
+follow the steps below before proceeding to deeper-level troubleshooting:
+
+* Check to see that the URL, token, and TLS/SSL settings are correct, and that the appropriate firewall ports are open (8088 or 443).
+
+* Check to see that the proper indexes are created in Splunk, and that the token has access to them.
+
+* Ensure the proper operation of the load balancer if used.
+
+* Lastly, execute the following command to check the internal logs of the syslog-ng process running in the container.  Depending on the
+traffic load, there may be quite a bit of output in the syslog-ng logs.
+```bash
+docker logs SC4S
+```
+You should see events similar to those below in the output:
+```ini
+Oct  1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.22.1'
+Oct  1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)'
+Oct  1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)'
+```
+If you see http server errors such as 4xx or 5xx responses from the http (HEC) endpoint, one or more of the items above are likely set
+incorrectly.  If validating/fixing the configuration fails to correct the problem, proceed to the "Troubleshooting" section for more
+information.

docs/gettingstarted/docker-swarm-rhel7.md

@@ -33,7 +33,7 @@ sudo docker swarm init
 
 # SC4S Configuration
 
-* Create a directory on the server for configuration. This should be available to all administrators, for example:
+* Create a directory on the server for local configurations. This should be available to all administrators, for example:
 ``/opt/sc4s/``
 * Create a docker-compose.yml file in the directory created above, based on the following template:
 
@@ -63,21 +63,33 @@ services:
       - /opt/sc4s/tls:/opt/syslog-ng/tls
 ```
 
+* NOTE:  While strictly optional, it is recommended that you create and/or download all files and directories referenced in the yml template 
+above (`volumes` declarations) according to the configuration steps that follow.  The TLS options are described in the "Configuration" section.
+Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors.
+
 ## Configure the SC4S environment
 
-Create the following file ``/opt/sc4s/env_file``
+Create the following file ``/opt/sc4s/env_file`` and add the environment variables below:
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment
 
+* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints.  If the endpoint is a VIP,
+match this value to the total number of indexers behind the load balancer.
+
+* NOTE:  Splunk Connect for Syslog defaults to secure configurations.  If you are not using trusted SSL certificates, be sure to
+uncomment the last line in the example below.
+
 ```dotenv
 SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
+SC4S_DEST_SPLUNK_HEC_WORKERS=6
 SPLUNK_CONNECT_METHOD=hec
 SPLUNK_DEFAULT_INDEX=main
 SPLUNK_METRICS_INDEX=em_metrics
+#Uncomment the following line if using untrusted SSL certificates
+#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
 
-
 ## Configure index destinations for Splunk 
 
 Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow
@@ -105,7 +117,7 @@ sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/mas
 ```
 * Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax.
 
-# Start SC4S
+## Start/Restart SC4S
 
 ```bash
 sudo docker stack deploy --compose-file docker-compose.yml sc4s
@@ -164,22 +176,80 @@ services:
       - /opt/sc4s/tls:/opt/syslog-ng/tls
 ```
 
-* Modify the following file ``/opt/sc4s/default/env_file`` 
+* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" 
+section for more information on your specific device(s).
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment
 
+* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints.  If the endpoint is a VIP,
+match this value to the total number of indexers behind the load balancer.
+
+* NOTE:  Splunk Connect for Syslog defaults to secure configurations.  If you are not using trusted SSL certificates, be sure to
+uncomment the last line in the example below.
+
 ```dotenv
 SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
 SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
+SC4S_DEST_SPLUNK_HEC_WORKERS=6
 SPLUNK_CONNECT_METHOD=hec
 SPLUNK_DEFAULT_INDEX=main
 SPLUNK_METRICS_INDEX=em_metrics
 SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
 #Uncomment the following line if using untrusted SSL certificates
 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
 ```
-* Start SC4S.
+
+* Restart SC4S (below)
+
+## Start/Restart SC4S
 
 ```bash
 docker stack deploy --compose-file docker-compose.yml sc4s
 ```
+
+# Stop SC4S
+
+Start by obtaining the stack name (ID):
+```bash
+docker stack ls
+```
+Then, remove the stack:
+```bash
+docker stack rm <ID>
+```
+# Verify Proper Operation
+
+SC4S has a number of "preflight" checks to ensure that the container starts properly and that the syntax of the underlying syslog-ng
+configuration is correct.  After this step completes, to verify SC4S is properly communicating with Splunk,
+execute the following search in Splunk:
+
+```ini
+index=* sourcetype=sc4s:events "starting up"
+```
+This should yield the following event:
+```ini
+syslog-ng starting up; version='3.22.1'
+``` 
+when the startup process proceeds normally (without syntax errors). If you do not see this,
+follow the steps below before proceeding to deeper-level troubleshooting:
+
+* Check to see that the URL, token, and TLS/SSL settings are correct, and that the appropriate firewall ports are open (8088 or 443).
+
+* Check to see that the proper indexes are created in Splunk, and that the token has access to them.
+
+* Ensure the proper operation of the load balancer if used.
+
+* Lastly, execute the following command to check the internal logs of the syslog-ng process running in the container.  Depending on the
+traffic load, there may be quite a bit of output in the syslog-ng logs.
+```bash
+docker logs SC4S
+```
+You should see events similar to those below in the output:
+```ini
+Oct  1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.22.1'
+Oct  1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)'
+Oct  1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)'
+```
+If you see http server errors such as 4xx or 5xx responses from the http (HEC) endpoint, one or more of the items above are likely set
+incorrectly.  If validating/fixing the configuration fails to correct the problem, proceed to the "Troubleshooting" section for more
+information.