update compose for swarm (#76)

Documentation updates for beta users covering podman, docker, and docker swarm
CSVD · Sep 10, 2019 · 2422049 · 2422049
1 parent 3173ff9
commit 2422049
Show file tree

Hide file tree

Showing 8 changed files with 470 additions and 3 deletions.
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -17,7 +17,10 @@ services:
     volumes:
       - sc4s-results:/work/test-results
     environment:
+      - SPLUNK_USER=admin
       - SPLUNK_PASSWORD=${SPLUNK_PASSWORD}
+      - SPLUNK_HOST=splunk
+      - SYSLOG_HOST=sc4s
 
   sc4s:
     image: splunk/scs:latest

diff --git a/docs/gettingstarted.md b/docs/gettingstarted.md
@@ -1,4 +1,73 @@
+# Getting Started
 
+Splunk Connect for Syslog is a containerized distribution of syslog-ng with a configuration framework
+designed to simplify getting syslog data into Splunk Enterprise and Splunk Cloud. Our approach is
+to provide a container runtime agnostic solution allowing customers to follow our guidance or adapt the solution
+to their enterprise container strategy.
+
+
+# Planning Deployment
+
+Syslog is an overloaded term that refers to multiple message formats AND optionally a wire protocol for
+transmission of events between computer systems over UDP, TCP, or TLS. The protocol is designed to minimize
+overhead on the sender favoring performance over reliability. This fundamental choice means any instability
+or resource constraint will cause data to be lost in transmission.
+
+* When practical and cost effective considering the importance of completeness as a requirement, place the scs
+instance in the same VLAN as the source device.
+
+* Avoid crossing a Wireless network, WAN, Firewall, Load Balancer, or inline IDS.
+* When High Availability of a single instance of SCS is required, implement multi node clustering of the container 
+environment.
+* Avoid TCP except where the source is unable to contain the event to a single UDP packet.
+* Avoid TLS except where the event may cross a untrusted network.
+
+
+# Implementation
+
+## Setup indexes in Splunk
+
+SCS is pre-configured to map each sourcetype to a typical index, for new installations best practice is to create the following
+indexes in Splunk. The indexes can be customized easily if desired. If using defaults create the following indexes on Splunk:
+
+* netauth
+* netfw
+* netids
+* netops
+* netproxy
+* netipam
+* em_metrics (ensure this is created as a metrics index)
+
+## Install Related Splunk Apps
+
+Install the following:
+
+* [Splunk App for Infrastructure](https://splunkbase.splunk.com/app/3975/)
+* [Splunk Add-on for Infrastructure](https://splunkbase.splunk.com/app/4217/)
+* [Splunk Metrics Workspace](https://splunkbase.splunk.com/app/4192/) *NOTE Included in Splunk 7.3.0 and above* 
+
+## Setup Splunk HTTP Event Collector
+
+- Set up the Splunk HTTP Event Collector with the HEC endpoints behind a load balancer (VIP) configured for https round robin *WITHOUT* sticky session.  Alternatively, a list of HEC endpoint URLs can be configured in SC4S if no load balancer is in place.  In either case, it is recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of HWFs.
+- Create a HEC token that will be used by SCS and ensure the token has access to place events in main, em_metrics, and all indexes used as event destinations
+
+### Splunk Cloud
+
+Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
+
+### Splunk Enterprise
+
+Refer to [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q)
+
+## Implement a container run time and SCS
+
+| Container and Orchestration | Notes |
+|-----------------------------|-------|
+| [Podman + systemd single node](gettingstarted/podman-systemd-general.md) | First choice for RedHat 7.x and 8.x, second choice for Debian and Ubuntu (packages provided via PPA) |
+| [Docker CE + systemd single node](gettingstarted/docker-systemd-general.md) | First choice for Debian, Ubuntu, and CentOS distributions with limited existing docker experience |
+| [Docker CE + Swarm single node](gettingstarted/docker-swarm-general.md) | Option for Debian, Ubuntu, and CentOS  desiring swarm orchestration |
+| [Docker CE + Swarm single node RHEL 7.7](gettingstarted/docker-swarm-rhel7.md) | Option for RedHat 7.7 desiring swarm orchestration |
+=======
 # Pre-req
 
 * Linux host with Docker 19.x or newer with Docker Swarm enabled

diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md
@@ -0,0 +1,111 @@
+
+# Install Docker CE and Swarm
+
+Refer to [Getting Started](https://docs.docker.com/get-started/)
+
+# Setup
+
+* Create a directory on the server for configuration. This should be available to all administrators, for example:
+``/opt/scs/``
+* Create a docker-compose.yml file and place it in the directory created above based on the following template:
+
+```yaml
+version: "3.7"
+services:
+  sc4s:
+    image: splunk/scs:latest
+    ports:  
+       - target: 514
+         published: 514
+         protocol: tcp
+#Comment the following line out if using docker-compose
+         mode: host
+       - target: 514
+         published: 514
+         protocol: udp
+#Comment the following line out if using docker-compose         
+         mode: host
+    environment:
+      - SPLUNK_HEC_URL=https://inputs-hec.kops.spl.guru/services/collector/event
+      - SPLUNK_HEC_TOKEN=02450979-d363-4e6c-b6c9-796d8b546a6e
+      - SPLUNK_CONNECT_METHOD=hec
+      - SPLUNK_DEFAULT_INDEX=main
+      - SPLUNK_METRICS_INDEX=em_metrics
+    volumes:
+#Uncomment the following line if overriding index destinations    
+#      - ./sc4s-juniper/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv
+#Uncomment the following lines if using a host or network based filter and log_path
+#      - ./sc4s-juniper/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv
+#      - ./sc4s-juniper/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf
+
+```
+
+## Configure index destinations for Splunk 
+
+Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow customization of index destinations.
+
+* Download the latest context.csv file to a subdirectory sc4s below the docker-compose.yml file created above.
+
+```bash
+sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv
+```
+* Edit splunk_index.csv review the index configuration and revise as required for sourcetypes utilized in your environment.
+
+## Configure sources by source IP or host name
+
+Legacy sources and non-standard-compliant source require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation. 
+
+* Download the latest vendor_product_by_source.conf file to a subdirectory sc4s below the docker-compose.yml file created above.
+```bash
+sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf
+sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv
+```
+* Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax.
+
+* Start SC4S.
+
+```bash
+docker stack deploy --compose-file docker-compose.yml sc4s
+```
+
+
+## Scale out
+
+Additional hosts can be deployed for syslog collection from additional network zones and locations.
+
+
+## Single Source Technology instance - Alpha
+
+For certain source technologies message categorization by content is impossible. To support collection 
+of such legacy nonstandard sources, we provide a means of dedicating a container to a specific source using
+an alternate port. In the following configration example a dedicated port is opened (6514) for legacy juniper netscreen devices.
+
+This approach is "alpha" and subject to change.
+
+```yaml
+version: "3"
+services:
+  sc4s-juniper-netscreen:
+    image: splunk/scs:latest
+    hostname: sc4s-juniper-netscreen
+    ports:  
+       - target: 514
+         published: 6514
+         protocol: tcp
+#Comment the following line out if using docker-compose
+         mode: host
+       - target: 514
+         published: 6514
+         protocol: udp
+#Comment the following line out if using docker-compose         
+         mode: host
+    environment:
+      - SPLUNK_HEC_URL=https://foo:8088/services/collector/event
+      - SPLUNK_HEC_TOKEN=<token>
+      - SPLUNK_CONNECT_METHOD=hec
+      - SPLUNK_DEFAULT_INDEX=<defaultindex>
+      - SPLUNK_METRICS_INDEX=em_metrics
+      - SYSLOG_PRESUME_FILTER=f_juniper_netscreen
+    volumes:
+    - ./sc4s-juniper/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv
+```
diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md
@@ -0,0 +1,136 @@
+
+# Install Docker CE and Swarm
+
+*Warning* this method of installing docker on RHEL does not appear to be supported:
+
+Enable required repositories
+```bash
+subscription-manager repos --enable=rhel-7-server-rpms
+subscription-manager repos --enable=rhel-7-server-extras-rpms
+subscription-manager repos --enable=rhel-7-server-optional-rpms
+yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
+```
+
+Enable EPEL
+```bash
+yum install wget -y
+cd /tmp
+wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
+rpm -Uvh epel-release-latest-7.noarch.rpm
+```
+
+Install required packages and enable docker
+```bash
+yum install docker-ce -y
+systemctl enable docker.service
+systemctl start docker.service
+```
+
+Initialize Docker Swarm
+```bash
+sudo docker swarm init
+```
+
+# Setup
+
+* Create a directory on the server for configuration. This should be available to all administrators, for example:
+``/opt/scs/``
+* Create a docker-compose.yml file and place it in the directory created above based on the following template:
+
+```yaml
+version: "3.7"
+services:
+  sc4s:
+    image: splunk/scs:latest
+    ports:  
+       - target: 514
+         published: 514
+         protocol: tcp
+#Comment the following line out if using docker-compose
+         mode: host
+       - target: 514
+         published: 514
+         protocol: udp
+#Comment the following line out if using docker-compose         
+         mode: host
+    environment:
+      - SPLUNK_HEC_URL=https://inputs-hec.kops.spl.guru/services/collector/event
+      - SPLUNK_HEC_TOKEN=02450979-d363-4e6c-b6c9-796d8b546a6e
+      - SPLUNK_CONNECT_METHOD=hec
+      - SPLUNK_DEFAULT_INDEX=main
+      - SPLUNK_METRICS_INDEX=em_metrics
+    volumes:
+#Uncomment the following line if overriding index destinations    
+#      - ./sc4s-juniper/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv
+#Uncomment the following lines if using a host or network based filter and log_path
+#      - ./sc4s-juniper/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv
+#      - ./sc4s-juniper/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf
+
+```
+
+## Configure index destinations for Splunk 
+
+Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow customization of index destinations.
+
+* Download the latest context.csv file to a subdirectory sc4s below the docker-compose.yml file created above.
+
+```bash
+sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv
+```
+* Edit splunk_index.csv review the index configuration and revise as required for sourcertypes utilized in your environment.
+
+## Configure sources by source IP or host name
+
+Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation. 
+
+* Download the latest vendor_product_by_source.conf file to a subdirectory sc4s below the docker-compose.yml file created above
+```bash
+sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf
+sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv
+```
+* Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax.
+
+* Start SC4S.
+
+```bash
+sudo docker stack deploy --compose-file docker-compose.yml sc4s
+```
+
+## Scale out
+
+Additional hosts can be deployed for syslog collection from additional network zones and locations.
+
+
+## Single Source Technology instance - Alpha
+
+For certain source technologies message categorization by content is impossible. To support collection of such legacy nonstandard sources, we provide a means of dedicating a container to a specific source using an alternate port. In the following configration example a dedicated port is opened (6514) for legacy juniper netscreen devices.
+
+This approach is "alpha" and subject to change.
+
+```yaml
+version: "3"
+services:
+  sc4s-juniper-netscreen:
+    image: splunk/scs:latest
+    hostname: sc4s-juniper-netscreen
+    ports:  
+       - target: 514
+         published: 6514
+         protocol: tcp
+#Comment the following line out if using docker-compose
+         mode: host
+       - target: 514
+         published: 6514
+         protocol: udp
+#Comment the following line out if using docker-compose         
+         mode: host
+    environment:
+      - SPLUNK_HEC_URL=https://foo:8088/services/collector/event
+      - SPLUNK_HEC_TOKEN=<token>
+      - SPLUNK_CONNECT_METHOD=hec
+      - SPLUNK_DEFAULT_INDEX=<defaultindex>
+      - SPLUNK_METRICS_INDEX=em_metrics
+      - SYSLOG_PRESUME_FILTER=f_juniper_netscreen
+    volumes:
+    - ./sc4s-juniper/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv
+```