From be74c0c28c83af1802af513ae016426b769dfac9 Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Thu, 24 Oct 2019 21:06:43 -0700 Subject: [PATCH] Release/1.0.3 (#176) * This change makes the location of env_file consistent however this is a procedural breaking change a deployment using the earlier docs will have the file in the wrong place. Users should move the file from opt/sc4s/default/env_file to opt/sc4s/env_file and revise the sc4s.service file to correct * Sync content from readme to RTD index page * Set proper sourcetype for checkpoint string (#175) --- README.md | 20 +++++++++++++------ docs/configuration.md | 2 +- docs/gettingstarted/byoe-rhel7.md | 4 ++-- docs/gettingstarted/docker-swarm-general.md | 2 +- docs/gettingstarted/docker-swarm-rhel7.md | 2 +- docs/gettingstarted/docker-systemd-general.md | 12 +++++------ docs/gettingstarted/podman-systemd-general.md | 12 +++++------ docs/index.md | 15 ++++++++++++++ docs/sources.md | 9 ++++++++- 9 files changed, 54 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 64043d5..3ba9077 100644 --- a/README.md +++ b/README.md @@ -20,12 +20,15 @@ For full usage instructions, please visit the Splunk Connect for Syslog [documen ## Support -Please use the GitHub issue tracker to submit bugs or request features. - -If you have questions or need support, you can: +Splunk Connect for Syslog is an open source product developed by Splunkers with contributions from the community of partners and customers. This unique product will be enhanced, maintained and supported by the community, led by Splunkers with deep subject matter expertise. The primary reason why Splunk is taking this approach is to push product development closer to those that use and depend upon it. This direct connection will help us all be more successful and move at a rapid pace. Post a question to Splunk Answers using the tag "Splunk Connect For Syslog" -Join the #splunk-connect-for-syslog room in the splunk-usergroups Slack Workspace + +Join the #splunk-connect-for-syslog room in the splunk-usergroups Slack Workspace. If you don't yet have an account [sign up](https://docs.splunk.com/Documentation/Community/1.0/community/Chat) + +Please use the GitHub issue tracker to submit bugs or request enhancements: https://github.com/splunk/splunk-connect-for-syslog/issues + +Get involved, try it out, ask questions, contribute filters, and make new friends! ## Contributing @@ -33,6 +36,11 @@ We welcome feedback and contributions from the community! Please see our [contri ## License -Configuration and documentation licensed subject to [CC0](LICENSE-CC0) +* Configuration and documentation licensed subject to [CC0](LICENSE-CC0) + +* Code and scripts licensed subject to [BSD-2-Clause](LICENSE-BSD2) + +* Third Party Red Hat Universal Base Image see [License](https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf) + +* Third Party Syslog-NG (OSE) [License](https://github.com/balabit/syslog-ng) -Code and scripts licensed subject to [BSD-2-Clause](LICENSE-BSD2) \ No newline at end of file diff --git a/docs/configuration.md b/docs/configuration.md index 701fffa..02c2d18 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -39,7 +39,7 @@ and variables needed to properly configure SC4S for your environment. * Create a folder ``/opt/sc4s/tls`` * Save the server private key in PEM format with NO PASSWORD to ``/opt/sc4s/tls/server.key`` * Save the server certificate in PEM format to ``/opt/sc4s/tls/server.pem`` -* Add the following line to ``/opt/sc4s/default/env_file`` +* Add the following line to ``/opt/sc4s/env_file`` ```dotenv SC4S_SOURCE_TLS_ENABLE=yes diff --git a/docs/gettingstarted/byoe-rhel7.md b/docs/gettingstarted/byoe-rhel7.md index 47d2f7c..73b7b33 100644 --- a/docs/gettingstarted/byoe-rhel7.md +++ b/docs/gettingstarted/byoe-rhel7.md @@ -84,7 +84,7 @@ ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid ExecReload=/bin/kill -HUP $MAINPID EnvironmentFile=-/etc/default/syslog-ng EnvironmentFile=-/etc/sysconfig/syslog-ng -EnvironmentFile=/opt/sc4s/default/env_file +EnvironmentFile=/opt/sc4s/env_file StandardOutput=journal StandardError=journal Restart=on-failure @@ -125,7 +125,7 @@ mkdir -p /opt/syslog-ng/var/data/disk-buffer/ sudo bash /opt/sc4s/bin/preconfig.sh ``` -* Create the file ``/opt/sc4s/default/env_file`` and add the following environment variables: +* Create the file ``/opt/sc4s/env_file`` and add the following environment variables: ```dotenv SYSLOGNG_OPTS=-f /opt/syslog-ng/etc/syslog-ng.conf diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index 9a03de3..8f536d1 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -167,7 +167,7 @@ services: # - /opt/sc4s/tls:/opt/syslog-ng/tls ``` -* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" +* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" section for more information on your specific device(s). * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index 0d27850..ec93e82 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -196,7 +196,7 @@ services: # - /opt/sc4s/tls:/opt/syslog-ng/tls ``` -* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" +* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" section for more information on your specific device(s). * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 51dda09..27fa688 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -30,12 +30,12 @@ Restart=always ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE ExecStartPre=/usr/bin/docker run \ - --env-file=/opt/sc4s/default/env_file \ + --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp \ - --env-file=/opt/sc4s/default/env_file \ + --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ --name SC4S --rm \ @@ -68,7 +68,7 @@ unit file above. Failure to do this will cause SC4S to abort at startup. ## Configure the SC4S environment -Create a file named ``/opt/sc4s/default/env_file`` and add the following environment variables: +Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event @@ -157,12 +157,12 @@ TimeoutStartSec=0 Restart=always ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE ExecStartPre=/usr/bin/docker run \ - --env-file=/opt/sc4s/default/env_file \ + --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ - --env-file=/opt/sc4s/default/env_file \ + --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ --name SC4S \ @@ -170,7 +170,7 @@ ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 - $SC4S_IMAGE ``` -* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" +* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" section for more information on your specific device(s). * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index e0964a5..1dea7bd 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -30,12 +30,12 @@ Restart=always ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE ExecStartPre=/usr/bin/podman run \ - --env-file=/opt/sc4s/default/env_file \ + --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp \ - --env-file=/opt/sc4s/default/env_file \ + --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ --name SC4S --rm \ @@ -68,7 +68,7 @@ unit file above. Failure to do this will cause SC4S to abort at startup. ## Configure the sc4s environment -Create a file named ``/opt/sc4s/default/env_file`` and add the following environment variables: +Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event @@ -157,12 +157,12 @@ TimeoutStartSec=0 Restart=always ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE ExecStartPre=/usr/bin/podman run \ - --env-file=/opt/sc4s/default/env_file \ + --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ - --env-file=/opt/sc4s/default/env_file \ + --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ --name SC4S \ @@ -170,7 +170,7 @@ ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 - $SC4S_IMAGE ``` -* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" +* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" section for more information on your specific device(s). * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment diff --git a/docs/index.md b/docs/index.md index 19dfcad..c7c293a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -12,6 +12,21 @@ HTTP event Collector. * Reduce latency and improve scale by balancing event distribution across Splunk Indexers +## Support + +Splunk Connect for Syslog is an open source product developed by Splunkers with contributions from the community of partners and customers. This unique product will be enhanced, maintained and supported by the community, led by Splunkers with deep subject matter expertise. The primary reason why Splunk is taking this approach is to push product development closer to those that use and depend upon it. This direct connection will help us all be more successful and move at a rapid pace. + +Post a question to Splunk Answers using the tag "Splunk Connect For Syslog" + +Join the #splunk-connect-for-syslog room in the splunk-usergroups Slack Workspace. If you don't yet have an account [sign up](https://docs.splunk.com/Documentation/Community/1.0/community/Chat) + +Please use the GitHub issue tracker to submit bugs or request enhancements: https://github.com/splunk/splunk-connect-for-syslog/issues + +Get involved, try it out, ask questions, contribute filters, and make new friends! + +## Contributing + +We welcome feedback and contributions from the community! Please see our [contribution guidelines](CONTRIBUTING.md) for more information on how to get involved. ## License diff --git a/docs/sources.md b/docs/sources.md index f7e85cb..5b912bc 100644 --- a/docs/sources.md +++ b/docs/sources.md @@ -1,3 +1,10 @@ +# Introduction +When using Splunk Connect for Syslog to onboard a data source, the SC4S filter performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source. + +SC4S "unique" filters are based either on the port upon which events arrive or the hostname/CIDR block from which they are sent. The "soup" filters run for events that arrive on port 514 (default for syslog), and contain regex and other syslog-specific parsers to identify events from a specific source, apply the correct sourcetype, and set other metadata. Data sources which generate events that are not unique enough to accurately identify with soup filters _must_ employ the "unique" filters (port/hostname/CIDR block) instead -- the soup filters are unavailable for these sources. + +If SC4S receives an event on port 514 which has no soup filter, that event will be given a "fallback" sourcetype. If you see events in Splunk with the fallback sourcetype, then you should figure out what source the events are from and determine why these events are not being sourcetyped correctly. The most common reason for events categorized as "fallback" is the lack of a SC4S filter for that source, and in some cases a misconfigured relay which alters the integrity of the message format. In most cases this means a new SC4S filter must be developed. In this situation you can either build a filter or file an issue with the community to request help. + # Vendor - Checkpoint ## Product - Log Exporter (Splunk) @@ -54,7 +61,7 @@ MSG Parse: This filter parses message content Use the following search to validate events are present ``` -index= sourcetype=cisco:asa +index= sourcetype=cp_log ``` Verify timestamp, and host values match as expected