Release/1.0.3 (#176)

* This change makes the location of env_file consistent however this is a procedural breaking change a deployment using the earlier docs will have the file in the wrong place. Users should move the file from opt/sc4s/default/env_file to opt/sc4s/env_file and revise the sc4s.service file to correct * Sync content from readme to RTD index page * Set proper sourcetype for checkpoint string (#175)
CSVD · Oct 25, 2019 · be74c0c · be74c0c
1 parent 01935f2
commit be74c0c
Show file tree

Hide file tree

Showing 9 changed files with 54 additions and 24 deletions.
diff --git a/README.md b/README.md
@@ -20,19 +20,27 @@ For full usage instructions, please visit the Splunk Connect for Syslog [documen
 
 ## Support
 
-Please use the GitHub issue tracker to submit bugs or request features.
-
-If you have questions or need support, you can:
+Splunk Connect for Syslog is an open source product developed by Splunkers with contributions from the community of partners and customers. This unique product will be enhanced, maintained and supported by the community, led by Splunkers with deep subject matter expertise. The primary reason why Splunk is taking this approach is to push product development closer to those that use and depend upon it. This direct connection will help us all be more successful and move at a rapid pace.
 
 Post a question to Splunk Answers using the tag "Splunk Connect For Syslog"
-Join the #splunk-connect-for-syslog room in the splunk-usergroups Slack Workspace
+
+Join the #splunk-connect-for-syslog room in the splunk-usergroups Slack Workspace. If you don't yet have an account [sign up](https://docs.splunk.com/Documentation/Community/1.0/community/Chat)
+
+Please use the GitHub issue tracker to submit bugs or request enhancements: https://github.com/splunk/splunk-connect-for-syslog/issues
+
+Get involved, try it out, ask questions, contribute filters, and make new friends!
 
 ## Contributing
 
 We welcome feedback and contributions from the community! Please see our [contribution guidelines](CONTRIBUTING.md) for more information on how to get involved.
 
 ## License
 
-Configuration and documentation licensed subject to [CC0](LICENSE-CC0)
+* Configuration and documentation licensed subject to [CC0](LICENSE-CC0)
+
+* Code and scripts licensed subject to [BSD-2-Clause](LICENSE-BSD2) 
+
+* Third Party Red Hat Universal Base Image see [License](https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf)
+
+* Third Party Syslog-NG (OSE) [License](https://github.com/balabit/syslog-ng)
 
-Code and scripts licensed subject to [BSD-2-Clause](LICENSE-BSD2) 
diff --git a/docs/configuration.md b/docs/configuration.md
@@ -39,7 +39,7 @@ and variables needed to properly configure SC4S for your environment.
 * Create a folder ``/opt/sc4s/tls``
 * Save the server private key in PEM format with NO PASSWORD to ``/opt/sc4s/tls/server.key``
 * Save the server certificate in PEM format to ``/opt/sc4s/tls/server.pem``
-* Add the following line to ``/opt/sc4s/default/env_file``
+* Add the following line to ``/opt/sc4s/env_file``
 
 ```dotenv
 SC4S_SOURCE_TLS_ENABLE=yes

diff --git a/docs/gettingstarted/byoe-rhel7.md b/docs/gettingstarted/byoe-rhel7.md
@@ -84,7 +84,7 @@ ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid
 ExecReload=/bin/kill -HUP $MAINPID
 EnvironmentFile=-/etc/default/syslog-ng
 EnvironmentFile=-/etc/sysconfig/syslog-ng
-EnvironmentFile=/opt/sc4s/default/env_file
+EnvironmentFile=/opt/sc4s/env_file
 StandardOutput=journal
 StandardError=journal
 Restart=on-failure
@@ -125,7 +125,7 @@ mkdir -p /opt/syslog-ng/var/data/disk-buffer/
 sudo bash /opt/sc4s/bin/preconfig.sh 
 ```
 
-* Create the file ``/opt/sc4s/default/env_file`` and add the following environment variables:
+* Create the file ``/opt/sc4s/env_file`` and add the following environment variables:
 
 ```dotenv
 SYSLOGNG_OPTS=-f /opt/syslog-ng/etc/syslog-ng.conf 

diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md
@@ -167,7 +167,7 @@ services:
 #     - /opt/sc4s/tls:/opt/syslog-ng/tls
 ```
 
-* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s).  See the "Sources" 
+* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s).  See the "Sources" 
 section for more information on your specific device(s).
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment

diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md
@@ -196,7 +196,7 @@ services:
 #     - /opt/sc4s/tls:/opt/syslog-ng/tls
 ```
 
-* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" 
+* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" 
 section for more information on your specific device(s).
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment

diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md
@@ -30,12 +30,12 @@ Restart=always
 
 ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE
 ExecStartPre=/usr/bin/docker run \
-        --env-file=/opt/sc4s/default/env_file \
+        --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         --name SC4S_preflight --rm \
         $SC4S_IMAGE -s
 ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp \
-        --env-file=/opt/sc4s/default/env_file \
+        --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \
         --name SC4S --rm \
@@ -68,7 +68,7 @@ unit file above.  Failure to do this will cause SC4S to abort at startup.
 
 ## Configure the SC4S environment
 
-Create a file named ``/opt/sc4s/default/env_file`` and add the following environment variables:
+Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
 
 ```dotenv
 SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
@@ -157,20 +157,20 @@ TimeoutStartSec=0
 Restart=always
 ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE
 ExecStartPre=/usr/bin/docker run \
-        --env-file=/opt/sc4s/default/env_file \
+        --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         --name SC4S_preflight --rm \
         $SC4S_IMAGE -s
 ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \
-        --env-file=/opt/sc4s/default/env_file \
+        --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \
         --name SC4S \
         --rm \
 $SC4S_IMAGE
 ```
 
-* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" 
+* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" 
 section for more information on your specific device(s).
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment

diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md
@@ -30,12 +30,12 @@ Restart=always
 
 ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE
 ExecStartPre=/usr/bin/podman run \
-        --env-file=/opt/sc4s/default/env_file \
+        --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         --name SC4S_preflight --rm \
         $SC4S_IMAGE -s
 ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp \
-        --env-file=/opt/sc4s/default/env_file \
+        --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \
         --name SC4S --rm \
@@ -68,7 +68,7 @@ unit file above.  Failure to do this will cause SC4S to abort at startup.
 
 ## Configure the sc4s environment
 
-Create a file named ``/opt/sc4s/default/env_file`` and add the following environment variables:
+Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
 
 ```dotenv
 SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event
@@ -157,20 +157,20 @@ TimeoutStartSec=0
 Restart=always
 ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE
 ExecStartPre=/usr/bin/podman run \
-        --env-file=/opt/sc4s/default/env_file \
+        --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         --name SC4S_preflight --rm \
         $SC4S_IMAGE -s
 ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \
-        --env-file=/opt/sc4s/default/env_file \
+        --env-file=/opt/sc4s/env_file \
         "$SC4S_LOCAL_CONFIG_MOUNT" \
         "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \
         --name SC4S \
         --rm \
 $SC4S_IMAGE
 ```
 
-* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" 
+* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" 
 section for more information on your specific device(s).
 
 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment

diff --git a/docs/index.md b/docs/index.md
@@ -12,6 +12,21 @@ HTTP event Collector.
 * Reduce latency and improve scale by balancing event distribution across Splunk Indexers
 
 
+## Support
+
+Splunk Connect for Syslog is an open source product developed by Splunkers with contributions from the community of partners and customers. This unique product will be enhanced, maintained and supported by the community, led by Splunkers with deep subject matter expertise. The primary reason why Splunk is taking this approach is to push product development closer to those that use and depend upon it. This direct connection will help us all be more successful and move at a rapid pace.
+
+Post a question to Splunk Answers using the tag "Splunk Connect For Syslog"
+
+Join the #splunk-connect-for-syslog room in the splunk-usergroups Slack Workspace. If you don't yet have an account [sign up](https://docs.splunk.com/Documentation/Community/1.0/community/Chat)
+
+Please use the GitHub issue tracker to submit bugs or request enhancements: https://github.com/splunk/splunk-connect-for-syslog/issues
+
+Get involved, try it out, ask questions, contribute filters, and make new friends!
+
+## Contributing
+
+We welcome feedback and contributions from the community! Please see our [contribution guidelines](CONTRIBUTING.md) for more information on how to get involved.
 
 ## License
 

diff --git a/docs/sources.md b/docs/sources.md
@@ -1,3 +1,10 @@
+# Introduction
+When using Splunk Connect for Syslog to onboard a data source, the SC4S filter performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source. 
+
+SC4S "unique" filters are based either on the port upon which events arrive or the hostname/CIDR block from which they are sent. The "soup" filters run for events that arrive on port 514 (default for syslog), and contain regex and other syslog-specific parsers to identify events from a specific source, apply the correct sourcetype, and set other metadata. Data sources which generate events that are not unique enough to accurately identify with soup filters _must_ employ the "unique" filters (port/hostname/CIDR block) instead -- the soup filters are unavailable for these sources. 
+
+If SC4S receives an event on port 514 which has no soup filter, that event will be given a "fallback" sourcetype. If you see events in Splunk with the fallback sourcetype, then you should figure out what source the events are from and determine why these events are not being sourcetyped correctly. The most common reason for events categorized as "fallback" is the lack of a SC4S filter for that source, and in some cases a misconfigured relay which alters the integrity of the message format. In most cases this means a new SC4S filter must be developed. In this situation you can either build a filter or file an issue with the community to request help. 
+
 # Vendor - Checkpoint
 
 ## Product - Log Exporter (Splunk)
@@ -54,7 +61,7 @@ MSG Parse: This filter parses message content
 Use the following search to validate events are present
 
 ```
-index=<asconfigured> sourcetype=cisco:asa
+index=<asconfigured> sourcetype=cp_log
 ```
 
 Verify timestamp, and host values match as expected