Skip to content

Commit

Permalink
Fix/doc gettingstarted (#85)
Browse files Browse the repository at this point in the history 
Docs cleanup
  • Loading branch information
mbonsack authored and Ryan Faircloth committed Sep 20, 2019
1 parent 48f0d7c commit 540c2fa
Show file tree
Hide file tree
Showing 5 changed files with 99 additions and 72 deletions.
53 changes: 24 additions & 29 deletions docs/gettingstarted.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

Splunk Connect for Syslog is a containerized distribution of syslog-ng with a configuration framework
designed to simplify getting syslog data into Splunk Enterprise and Splunk Cloud. Our approach is
to provide a container runtime agnostic solution allowing customers to follow our guidance or adapt the solution
to their enterprise container strategy.
to provide a runtime-agnostic solution allowing customers to deploy using the container runtime
environment of choice.


# Planning Deployment
Expand All @@ -13,19 +13,21 @@ transmission of events between computer systems over UDP, TCP, or TLS. The proto
overhead on the sender favoring performance over reliability. This fundamental choice means any instability
or resource constraint will cause data to be lost in transmission.

* When practical and cost effective considering the importance of completeness as a requirement, place the SC4s
* When practical and cost effective considering the importance of completeness as a requirement, place the sc4s
instance in the same VLAN as the source device.

* Avoid crossing a Wireless network, WAN, Firewall, Load Balancer, or inline IDS.
* When High Availability of a single instance of SC4S is required, implement multi node clustering of the container
* When High Availability of a single instance of SC4S is required, implement multi node clustering of the container
environment.
* Avoid TCP except where the source is unable to contain the event to a single UDP packet.
* Avoid TLS except where the event may cross a untrusted network.


# Implementation

## Setup indexes in Splunk
## Splunk Setup

### Create Indexes

SC4S is pre-configured to map each sourcetype to a typical index, for new installations best practice is to create the following
indexes in Splunk. The indexes can be customized easily if desired. If using defaults create the following indexes on Splunk:
Expand All @@ -38,28 +40,33 @@ indexes in Splunk. The indexes can be customized easily if desired. If using def
* netipam
* em_metrics (ensure this is created as a metrics index)

## Install Related Splunk Apps
### Install Related Splunk Apps

Install the following:

* [Splunk App for Infrastructure](https://splunkbase.splunk.com/app/3975/)
* [Splunk Add-on for Infrastructure](https://splunkbase.splunk.com/app/4217/)
* [Splunk Metrics Workspace](https://splunkbase.splunk.com/app/4192/) *NOTE Included in Splunk 7.3.0 and above*
* [Splunk Metrics Workspace](https://splunkbase.splunk.com/app/4192/) *NOTE Included in Splunk 7.3.0 and above*

## Setup Splunk HTTP Event Collector
### Configure the Splunk HTTP Event Collector

- Set up the Splunk HTTP Event Collector with the HEC endpoints behind a load balancer (VIP) configured for https round robin *WITHOUT* sticky session. Alternatively, a list of HEC endpoint URLs can be configured in SC4S if no load balancer is in place. In either case, it is recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of HWFs.
- Create a HEC token that will be used by SC4S and ensure the token has access to place events in main, em_metrics, and all indexes used as event destinations

### Splunk Cloud
- Set up the Splunk HTTP Event Collector with the HEC endpoints behind a load balancer (VIP) configured for https round robin *WITHOUT* sticky
session. Alternatively, a list of HEC endpoint URLs can be configured in SC4S if no load balancer is in place. In either case, it is
recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of HWFs.
- Create a HEC token that will be used by SC4S and ensure the token has access to place events in main, em_metrics, and all indexes used as
event destinations
- Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your
Splunk type.

Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
## Implement a Container Runtime and SC4S

### Splunk Enterprise
### Prerequisites

Refer to [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q)
* Linux host with Docker (CE 19.x or greater with Docker Swarm) or Podman enabled, depending on runtime choice (below).
* A network load balancer (NLB) configured for round robin. Note: Special consideration may be required when more advanced products are used. The optimal configuration of the load balancer will round robin each http POST request (not each connection).

## Implement a container run time and sc4s
### Select a Container Runtime and SC4S Configuration

| Container and Orchestration | Notes |
|-----------------------------|-------|
Expand All @@ -68,21 +75,9 @@ Refer to [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE
| [Docker CE + Swarm single node](gettingstarted/docker-swarm-general.md) | Option for Debian, Ubuntu, and CentOS desiring swarm orchestration |
| [Docker CE + Swarm single node RHEL 7.7](gettingstarted/docker-swarm-rhel7.md) | Option for RedHat 7.7 desiring swarm orchestration |

# Pre-req

* Linux host with Docker 19.x or newer with Docker Swarm enabled
* [Getting Started](https://docs.docker.com/get-started/)
* A Splunk index for metrics typically "em_metrics"
* One or more Splunk indexes for events collected by SC4S
* Splunk HTTP event collector enabled with a token dedicated for SC4S
* [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q)
* [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
* A network load balancer (NLB) configured for round robin. Note: Special consideration may be required when more advanced products are used. The optimal configuration of the load balancer will round robin each http POST request (not each connection)


## Scale out
# Scale out

Additional hosts can be deployed for syslog collection from additional network zones and locations

![SC4S deployment diagram](SC4S%20deployment.png)

30 changes: 19 additions & 11 deletions docs/gettingstarted/docker-swarm-general.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@

Refer to [Getting Started](https://docs.docker.com/get-started/)

# Setup
# SC4S Configuration

* Create a directory on the server for configuration. This should be available to all administrators, for example:
``/opt/sc4s/``
* Create a docker-compose.yml file and place it in the directory created above based on the following template:
* Create a docker-compose.yml file in the directory created above, based on the following template:

```yaml
version: "3.7"
Expand All @@ -29,10 +29,10 @@ services:
- /opt/sc4s/env_file
volumes:
#Uncomment the following line if overriding index destinations
# - ./sc4s-juniper/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv
# - ./context-local/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv
#Uncomment the following lines if using a host or network based filter and log_path
# - ./sc4s-juniper/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv
# - ./sc4s-juniper/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf
# - ./context-local/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv
# - ./context-local/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf

```

Expand All @@ -52,9 +52,11 @@ SPLUNK_METRICS_INDEX=em_metrics

## Configure index destinations for Splunk

Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow customization of index destinations.
Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow
customization of index destinations.

* Download the latest context.csv file to a subdirectory SC4S below the docker-compose.yml file created above.
* Create a subdirectory called ``context-local`` in the directory (e.g. ``/opt/sc4s/``) created in the first step above. From this directory,
execute the command below to download the index context file:

```bash
sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv
Expand All @@ -63,16 +65,19 @@ sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/mas

## Configure sources by source IP or host name

Legacy sources and non-standard-compliant source require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation.
Legacy sources and non-standard-compliant source require configuration by source IP or hostname as included in the event. The following steps
apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation.

* If not already done in the step immediately above, create a subdirectory called ``context-local`` in the directory (e.g. ``/opt/sc4s/``)
created in the first step above. From this directory, execute the commands below to download the vendor context files:

* Download the latest vendor_product_by_source.conf file to a subdirectory SC4S below the docker-compose.yml file created above.
```bash
sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf
sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv
```
* Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax.

* Start SC4S.
# Start SC4S

```bash
docker stack deploy --compose-file docker-compose.yml sc4s
Expand All @@ -91,7 +96,10 @@ of such legacy nonstandard sources we provide a means of dedicating a container
an alternate port.
Refer to the Sources documentation to identify the specific variable used to enable a specific port for the technology in use.

In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology specific ports modify the range as appropriate
Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use.

In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify the individual ports or a
range as appropriate for your network.

* Modify the unit file ``/opt/sc4s/docker-compose.yml``
```yaml
Expand Down
36 changes: 21 additions & 15 deletions docs/gettingstarted/docker-swarm-rhel7.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,39 +3,39 @@

*Warning* this method of installing docker on RHEL does not appear to be supported:

Enable required repositories
## Enable required repositories
```bash
subscription-manager repos --enable=rhel-7-server-rpms
subscription-manager repos --enable=rhel-7-server-extras-rpms
subscription-manager repos --enable=rhel-7-server-optional-rpms
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
```

Enable EPEL
## Enable EPEL
```bash
yum install wget -y
cd /tmp
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -Uvh epel-release-latest-7.noarch.rpm
```

Install required packages and enable docker
## Install required packages and enable docker
```bash
yum install docker-ce -y
systemctl enable docker.service
systemctl start docker.service
```

Initialize Docker Swarm
## Initialize Docker Swarm
```bash
sudo docker swarm init
```

# Setup
# SC4S Configuration

* Create a directory on the server for configuration. This should be available to all administrators, for example:
``/opt/sc4s/``
* Create a docker-compose.yml file and place it in the directory created above based on the following template:
* Create a docker-compose.yml file in the directory created above, based on the following template:

```yaml
version: "3.7"
Expand All @@ -57,10 +57,10 @@ services:
- /opt/sc4s/env_file
volumes:
#Uncomment the following line if overriding index destinations
# - ./sc4s-juniper/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv
# - ./context-local/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv
#Uncomment the following lines if using a host or network based filter and log_path
# - ./sc4s-juniper/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv
# - ./sc4s-juniper/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf
# - ./context-local/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv
# - ./context-local/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf

```

Expand All @@ -81,9 +81,11 @@ SPLUNK_METRICS_INDEX=em_metrics

## Configure index destinations for Splunk

Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow customization of index destinations.
Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow
customization of index destinations.

* Download the latest context.csv file to a subdirectory SC4S below the docker-compose.yml file created above.
* Create a subdirectory called ``context-local`` in the directory (e.g. ``/opt/sc4s/``) created in the first step above. From this directory,
execute the command below to download the index context file:

```bash
sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv
Expand All @@ -92,16 +94,19 @@ sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/mas

## Configure sources by source IP or host name

Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation.
Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps
apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation.

* If not already done in the step immediately above, create a subdirectory called ``context-local`` in the directory (e.g. ``/opt/sc4s/``)
created in the first step above. From this directory, execute the commands below to download the vendor context files:

* Download the latest vendor_product_by_source.conf file to a subdirectory SC4S below the docker-compose.yml file created above
```bash
sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf
sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv
```
* Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax.

* Start SC4S.
# Start SC4S

```bash
sudo docker stack deploy --compose-file docker-compose.yml sc4s
Expand All @@ -119,7 +124,8 @@ of such legacy nonstandard sources we provide a means of dedicating a container
an alternate port.
Refer to the Sources documentation to identify the specific variable used to enable a specific port for the technology in use.

In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology specific ports modify the range as appropriate
In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify the individual ports or a
range as appropriate for your network.

* Modify the unit file ``/opt/sc4s/docker-compose.yml``
```yaml
Expand Down

0 comments on commit 540c2fa

Please sign in to comment.