From a40218301160a97851e0b56f6a5f5bb2c8195e31 Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Thu, 10 Oct 2019 21:11:09 -0400 Subject: [PATCH 1/3] Feature/docsupdates (#134) * Fixes #116 * REMOVES CHANGELOG which is not maintained --- CHANGELOG | 6 ------ README.md | 46 ++++++++++++++++++---------------------------- docs/demo.md | 36 ++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 4 files changed, 55 insertions(+), 34 deletions(-) delete mode 100644 CHANGELOG create mode 100644 docs/demo.md diff --git a/CHANGELOG b/CHANGELOG deleted file mode 100644 index 4b84880..0000000 --- a/CHANGELOG +++ /dev/null @@ -1,6 +0,0 @@ -0.4.0 - - Add source support for Cisco NX - - Add source support for Symantec Proxy SG and ASG (Formerly bluecoat) - - Add support for mapping network source IP OR parsed host to specific vendor product where MSG parsing is impossible - - Code cleanup and simplification - - Begin using SEMVER for releases \ No newline at end of file diff --git a/README.md b/README.md index 9e4aa53..c7c8559 100644 --- a/README.md +++ b/README.md @@ -1,47 +1,37 @@ -# splunk-connect-for-syslog +# README Splunk Connect for Syslog is an open source packaged solution for getting data into Splunk using syslog-ng (OSE) and the Splunk HTTP event Collector. -# Use the demo +## Purpose -The Splunk Connect for syslog demo uses docker and docker compose -to configure a instance of Splunk along with syslog-ng and a test -harness to simulate a mix of events. Ensure git, docker and docker-compose -are pre-installed and working prior to continuing. +Splunk Connect for Syslog (SC4S) is a community project focused on reducing the pain of getting syslog data sources into Splunk. The primary pain points SC4S addresses include the following… +* Shortage of deep syslog expertise in the community +* Inconsistency between syslog server deployments creates a support challenge +* Data sources tagged with catch-all sourcetype “syslog” which limits Splunk analytics +* Uneven data distribution between Splunk indexers impacts search performance +* Splunk Connect for Syslog should be used by any Splunk customer needing to onboard data sources via syslog to Splunk. -- Clone the repository and cd into directory +## Usage -```bash -git clone git@github.com:splunk/splunk-connect-for-syslog.git -cd splunk-connect-for-syslog -``` +For full usage instructions, please visit the Splunk Connect for Syslog documentation page. -- Create a working .env file * Note for demo purposes this file does not need to be modified +## Support -```bash -cp .env.template .env -``` +Please use the GitHub issue tracker to submit bugs or request features. -- Update the splunkbase username and password in .env this allows the splunk container to install required add-ons for the demo +If you have questions or need support, you can: -- Start the demo environment +Post a question to Splunk Answers using the tag "Splunk Connect For Syslog" +Join the #splunk-connect-for-syslog room in the splunk-usergroups Slack Workspace -```bash -./demo-with-compose.sh -``` +## Contributing -- Login to splunk by browsing to http://127.0.0.1:8000 user name admin password "Changed@11" +We welcome feedback and contributions from the community! Please see our [contribution guidelines](CONTRIBUTING.md) for more information on how to get involved. -- Search the main index to see indexed events - -```spl -index = main -``` - -# License +## License Configuration and documentation licensed subject to [CC0](LICENSE-CC0) diff --git a/docs/demo.md b/docs/demo.md new file mode 100644 index 0000000..8d80112 --- /dev/null +++ b/docs/demo.md @@ -0,0 +1,36 @@ +# Use the demo + +The Splunk Connect for syslog demo uses docker and docker compose +to configure a instance of Splunk along with syslog-ng and a test +harness to simulate a mix of events. Ensure git, docker and docker-compose +are pre-installed and working prior to continuing. + + +- Clone the repository and cd into directory + +```bash +git clone git@github.com:splunk/splunk-connect-for-syslog.git +cd splunk-connect-for-syslog +``` + +- Create a working .env file * Note for demo purposes this file does not need to be modified + +```bash +cp .env.template .env +``` + +- Update the splunkbase username and password in .env this allows the splunk container to install required add-ons for the demo + +- Start the demo environment + +```bash +./demo-with-compose.sh +``` + +- Login to splunk by browsing to http://127.0.0.1:8000 user name admin password "Changed@11" + +- Search the main index to see indexed events + +```spl +index = * +``` diff --git a/mkdocs.yml b/mkdocs.yml index 3e07a94..cc7f4e8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -4,6 +4,7 @@ nav: - Home: 'index.md' - Performance: 'performance.md' - Getting Started: 'gettingstarted.md' + - Demo Lab: 'demo.md' - Configuration: 'configuration.md' - Sources: 'sources.md' - Troubleshooting: 'troubleshooting.md' From 92c77a53660536faf64a2e5283e5c85d4a5705d7 Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Thu, 10 Oct 2019 21:12:05 -0400 Subject: [PATCH 2/3] Document Community RPM installation for SC4S on RHEL/CENTOS 7 without containers (#135) Add instructions for deployment of SC4S in a BYOE --- docs/gettingstarted.md | 2 +- docs/gettingstarted/byoe-rhel7.md | 145 ++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+), 1 deletion(-) create mode 100644 docs/gettingstarted/byoe-rhel7.md diff --git a/docs/gettingstarted.md b/docs/gettingstarted.md index a43e906..a7fff9d 100644 --- a/docs/gettingstarted.md +++ b/docs/gettingstarted.md @@ -76,7 +76,7 @@ Splunk type. | [Docker CE + systemd single node](gettingstarted/docker-systemd-general.md) | First choice for Debian, Ubuntu, and CentOS distributions with limited existing docker experience | | [Docker CE + Swarm single node](gettingstarted/docker-swarm-general.md) | Option for Debian, Ubuntu, and CentOS desiring swarm orchestration | | [Docker CE + Swarm single node RHEL 7.7](gettingstarted/docker-swarm-rhel7.md) | Option for RedHat 7.7 desiring swarm orchestration | - +| [Bring your own Envionment](gettingstarted/byoe-rhel7.md) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers | # Scale out diff --git a/docs/gettingstarted/byoe-rhel7.md b/docs/gettingstarted/byoe-rhel7.md new file mode 100644 index 0000000..cc973d6 --- /dev/null +++ b/docs/gettingstarted/byoe-rhel7.md @@ -0,0 +1,145 @@ +#Warning + +The "Bring Your Own Environment" instructions that follow allow administrators to utilize the SC4S syslog-ng +config files directly on the host OS running on a hardware server or virtual machine. Administrators must provide an +appropriate host OS as well as an up-to-date syslog-ng installation either built from source (not documented) or +installed from community-built RPMs. Modification of the base configuration will be required for most customer +environments due to enterprise infrastructure variations. + +* NOTE: Installing or modifying system configurations can have unexpected consequences, and rudimentary linux system +administratrion and syslog-ng configuration experience is assumed. + +Read this [explanation](https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-ng-on-rhel-and-other-rpm-distributions) +on the reason syslog-ng builds are so dated in most RHEL/Debian distributions. + + +* Install CentOS or RHEL 7.7 +* Enable EPEL + * Centos 7 + + ```bash + sudo yum install epel-release + ``` + + * RHEL 7 + + ```bash + cd /tmp + wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + sudo yum install ./epel-release-latest-*.noarch.rpm -y + ``` + +* Enable the optional repo for RHEL 7 only + + ```bash + sudo subscription-manager repos --enable rhel-7-server-optional-rpms + ``` +* Enable the "stable" unoffical repo for syslog-ng + + ```bash + cd /etc/yum.repos.d/ + sudo wget https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-stable/repo/epel-7/czanik-syslog-ng-stable-epel-7.repo + sudo yum install syslog-ng syslog-ng-http syslog-ng-python + ``` + +* Optional step: Disable the OOB syslog-ng unit file, as the syslog-ng process configured here will run as the `sc4s` +service. rsyslog will continue to be the system logger, and can be left enabled _only_ if it is configured to not +listen on the same ports as sc4s. + +```bash +systemctl stop syslog-ng +systemctl disable syslog-ng +``` +* Download the latest bare_metal.tar from [releases](https://github.com/splunk/splunk-connect-for-syslog/releases) on github and untar the package + +```bash +cd /tmp +sudo wget https://github.com/splunk/splunk-connect-for-syslog/releases/download/0.12.1/baremetal.tar +tar -xf baremetal.tar +sudo mkdir -p /opt/syslog-ng/etc +sudo mkdir -p /opt/syslog-ng/var +sudo cp -R etc/* /opt/syslog-ng/etc/ +``` + +* Install and verify gomplate verify the output is 3.5.0 or newer + +```bash +sudo curl -o /usr/local/bin/gomplate -sSL https://github.com/hairyhenderson/gomplate/releases/download/v3.5.0/gomplate_linux-amd64 +sudo chmod 755 /usr/local/bin/gomplate +gomplate --help +``` + +* create the sc4s unit file drop in ``/etc/systemd/system/sc4s.service`` and add the following content + +```ini +[Unit] +Description=SC4S Syslog Daemon +Documentation=man:syslog-ng(8) +Wants=network.target network-online.target +After=network.target network-online.target + +[Service] +Type=notify +ExecStartPre=/opt/sc4s/bin/preconfig.sh +ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid +ExecReload=/bin/kill -HUP $MAINPID +EnvironmentFile=-/etc/default/syslog-ng +EnvironmentFile=-/etc/sysconfig/syslog-ng +EnvironmentFile=/opt/sc4s/default/env_file +StandardOutput=journal +StandardError=journal +Restart=on-failure + +[Install] +WantedBy=multi-user.target +``` + +* create the file ``/opt/sc4s/bin/preconfig.sh`` and add the following content + +```bash +#!/usr/bin/env bash +source scl_source enable rh-python36 + +cd /opt/syslog-ng +for d in $(find /opt/syslog-ng/etc -type d) +do + echo Templating conf for $d + gomplate \ + --input-dir=$d \ + --template t=etc/go_templates/ \ + --exclude=*.conf --exclude=*.csv --exclude=*.t --exclude=.*\ + --output-map="$d/{{ .in | strings.ReplaceAll \".conf.tmpl\" \".conf\" }}" +done + +mkdir -p /opt/syslog-ng/etc/conf.d/local/context/ +mkdir -p /opt/syslog-ng/etc/conf.d/local/config/ +cp --verbose -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context/ +cp --verbose -R -n /opt/syslog-ng/etc/local_config/* /opt/syslog-ng/etc/conf.d/local/config/ +mkdir -p /opt/syslog-ng/var/data/disk-buffer/ +``` + +* set execute permissions on the file +``` +sudo chmod 755 /opt/sc4s/bin/preconfig.sh +``` + +* Create the file ``/opt/sc4s/default/env_file`` and add the following environment variables: + +```dotenv +SYSLOGNG_OPTS=-f /opt/syslog-ng/etc/syslog-ng.conf +SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event +SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 +SC4S_DEST_SPLUNK_HEC_WORKERS=6 +SPLUNK_CONNECT_METHOD=hec +SPLUNK_DEFAULT_INDEX=main +SPLUNK_METRICS_INDEX=em_metrics +#Uncomment the following line if using untrusted SSL certificates +#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no +``` + +* Reload systemctl and restart syslog-ng + +```bash +sudo systemctl daemon-reload +sudo systemctl start sc4s +``` From 9f42d487cd59ea3c08e712a835870d0d5c18cf1e Mon Sep 17 00:00:00 2001 From: mbonsack Date: Thu, 10 Oct 2019 18:15:50 -0700 Subject: [PATCH 3/3] Fix unique port unit/yml files (#136) Fix port ranges Fix comment for docker-compose.yml Add UDP ports to unique port unit/yml files --- docs/gettingstarted/docker-swarm-general.md | 14 ++++++------- docs/gettingstarted/docker-swarm-rhel7.md | 14 ++++++------- docs/gettingstarted/docker-systemd-general.md | 21 +++++++++---------- docs/gettingstarted/podman-systemd-general.md | 21 +++++++++---------- 4 files changed, 32 insertions(+), 38 deletions(-) diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index cbc5558..a1c0f72 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -114,7 +114,7 @@ For collection of such sources we provide a means of dedicating a unique listeni Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. -In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify the individual ports or a +In the following example the target port ranges allow for up to 21 technology-specific ports. Modify individual ports or a range as appropriate for your network. * Modify the unit file ``/opt/sc4s/docker-compose.yml`` @@ -134,22 +134,20 @@ services: protocol: udp #Comment the following line out if using docker-compose mode: host - - target: 5000-5021 - published: 5000-5021 + - target: 5000-5020 + published: 5000-5020 protocol: tcp #Comment the following line out if using docker-compose mode: host - - target: 5000-5021 - published: 5000-5021 + - target: 5000-5020 + published: 5000-5020 protocol: udp #Comment the following line out if using docker-compose mode: host env_file: - /opt/sc4s/env_file volumes: - - /opt/sc4s/default/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv - - /opt/sc4s/default/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv - - /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf + - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local #Uncomment the following line if custom TLS certs are provided - /opt/sc4s/tls:/opt/syslog-ng/tls ``` diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index 87106c8..c06849b 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -144,7 +144,7 @@ For collection of such sources we provide a means of dedicating a unique listeni Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. -In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify the individual ports or a +In the following example the target port ranges allow for up to 21 technology-specific ports. Modify individual ports or a range as appropriate for your network. * Modify the unit file ``/opt/sc4s/docker-compose.yml`` @@ -164,22 +164,20 @@ services: protocol: udp #Comment the following line out if using docker-compose mode: host - - target: 5000-5021 - published: 5000-5021 + - target: 5000-5020 + published: 5000-5020 protocol: tcp #Comment the following line out if using docker-compose mode: host - - target: 5000-5021 - published: 5000-5021 + - target: 5000-5020 + published: 5000-5020 protocol: udp #Comment the following line out if using docker-compose mode: host env_file: - /opt/sc4s/env_file volumes: - - /opt/sc4s/default/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv - - /opt/sc4s/default/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv - - /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf + - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local #Uncomment the following line if custom TLS certs are provided - /opt/sc4s/tls:/opt/syslog-ng/tls ``` diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index a7931ed..32eb897 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -117,7 +117,7 @@ For collection of such sources we provide a means of dedicating a unique listeni Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. -In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify the individual ports or a +In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify individual ports or a range as appropriate for your network. * Modify the unit file ``/lib/systemd/system/sc4s.service`` @@ -130,25 +130,24 @@ Requires=network.service [Service] Environment="SC4S_IMAGE=splunk/scs:latest" -#Note Uncomment this line to use custom index names AND download the splunk_index.csv file template per getting started -Environment="SC4S_UNIT_SPLUNK_INDEX=-v /opt/sc4s/default/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv" -#Note Uncomment the following two linese for host and ip based source type mapping AND download the two file templates per getting started -#Environment="SC4S_UNIT_VP_CSV=-v /opt/sc4s/default/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv" -#Environment="SC4S_UNIT_VP_CONF=-v /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf" -#Uncomment the following line if custom TLS certs are provided -#Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" +# Optional mount point for local overrides and configurations; see notes in docs + +Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local" + +# Uncomment the following line if custom TLS certs are provided +# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" TimeoutStartSec=0 Restart=always ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE ExecStartPre=/usr/bin/docker run \ --env-file=/opt/sc4s/default/env_file \ - "$SC4S_UNIT_SPLUNK_INDEX" "$SC4S_UNIT_VP_CSV" "$SC4S_UNIT_VP_CONF" "$SC4S_TLS_DIR" \ + "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s -ExecStart=/usr/bin/docker run -p 514:514 -p 5000-5020:5000-5020 \ +ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ --env-file=/opt/sc4s/default/env_file \ - "$SC4S_UNIT_SPLUNK_INDEX" "$SC4S_UNIT_VP_CSV" "$SC4S_UNIT_VP_CONF" "$SC4S_TLS_DIR" \ + "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S \ --rm \ $SC4S_IMAGE diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 2ce8a2b..394762d 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -117,7 +117,7 @@ For collection of such sources we provide a means of dedicating a unique listeni Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. -In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify the individual ports or a +In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify individual ports or a range as appropriate for your network. * Modify the unit file ``/lib/systemd/system/sc4s.service`` @@ -130,25 +130,24 @@ Requires=network.service [Service] Environment="SC4S_IMAGE=splunk/scs:latest" -#Note Uncomment this line to use custom index names AND download the splunk_index.csv file template per getting started -Environment="SC4S_UNIT_SPLUNK_INDEX=-v /opt/sc4s/default/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv" -#Note Uncomment the following two linese for host and ip based source type mapping AND download the two file templates per getting started -#Environment="SC4S_UNIT_VP_CSV=-v /opt/sc4s/default/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv" -#Environment="SC4S_UNIT_VP_CONF=-v /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf" -#Uncomment the following line if custom TLS certs are provided -#Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" +# Optional mount point for local overrides and configurations; see notes in docs + +Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local" + +# Uncomment the following line if custom TLS certs are provided +# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" TimeoutStartSec=0 Restart=always ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE ExecStartPre=/usr/bin/podman run \ --env-file=/opt/sc4s/default/env_file \ - "$SC4S_UNIT_SPLUNK_INDEX" "$SC4S_UNIT_VP_CSV" "$SC4S_UNIT_VP_CONF" "$SC4S_TLS_DIR" \ + "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s -ExecStart=/usr/bin/podman run -p 514:514 -p 5000-5020:5000-5020 \ +ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ --env-file=/opt/sc4s/default/env_file \ - "$SC4S_UNIT_SPLUNK_INDEX" "$SC4S_UNIT_VP_CSV" "$SC4S_UNIT_VP_CONF" "$SC4S_TLS_DIR" \ + "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S \ --rm \ $SC4S_IMAGE