Skip to content

Commit

Permalink
Merge branch 'release/1.6.0'
Browse files Browse the repository at this point in the history
  • Loading branch information
rfaircloth-splunk committed Jan 14, 2020
2 parents 8369f7f + 515f9dc commit 4baa513
Show file tree
Hide file tree
Showing 45 changed files with 689 additions and 587 deletions.
8 changes: 4 additions & 4 deletions docker-compose.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,12 @@ services:
- SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
- SC4S_SOURCE_TLS_ENABLE=no
- SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
- SC4S_LISTEN_DEFAULT_TCP_PORT=514
- SC4S_LISTEN_DEFAULT_UDP_PORT=514
# - SC4S_LISTEN_DEFAULT_TLS_PORT=6514
# - SC4S_LISTEN_DEFAULT_TCP_PORT=514
# - SC4S_LISTEN_DEFAULT_UDP_PORT=514
# - SC4S_LISTEN_DEFAULT_TLS_PORT=6514
- SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
- SC4S_LISTEN_CHECKPOINT_SPLUNK_TCP_PORT=6000
# - SC4S_ARCHIVE_CHECKPOINT=yes
# - SC4S_ARCHIVE_CHECKPOINT=yes
- SC4S_ARCHIVE_GLOBAL=yes
volumes:
- ./tls:/opt/syslog-ng/tls
Expand Down
20 changes: 19 additions & 1 deletion docs/gettingstarted/byoe-rhel7.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ administration and syslog-ng configuration experience is assumed when using the

* NOTE: Do _not_ depend on the distribution-supplied version of syslog-ng, as it will likely be far too old.
Read this [explanation](https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-ng-on-rhel-and-other-rpm-distributions)
for the reason why syslog-ng builds are so dated in most RHEL/Debian distributions.
for the reason why syslog-ng builds are so dated in almost all RHEL/Debian distributions.

# BYOE Installation Instructions

Expand Down Expand Up @@ -157,3 +157,21 @@ sudo systemctl daemon-reload
sudo systemctl enable sc4s
sudo systemctl start sc4s
```
## Configure SC4S Listening Ports

Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS.
The standard SC4S configuration reflect these defaults. These defaults can be changed by adding the following
additional environment variables with appropriate values to the ``env_file`` above:
```dotenv
SC4S_LISTEN_DEFAULT_TCP_PORT=514
SC4S_LISTEN_DEFAULT_UDP_PORT=514
SC4S_LISTEN_DEFAULT_TLS_PORT=6514
```
### Dedicated (Unique) Listening Ports

For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in
the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise.
For collection of such sources we provide a means of dedicating a unique listening port to a specific source.

Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology
in use.
141 changes: 71 additions & 70 deletions docs/gettingstarted/docker-swarm-general.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ reboot:
net.ipv4.ip_forward=1
```

# SC4S Configuration
# SC4S Initial Configuration

* Create a directory on the server for local configurations and disk buffering. This should be available to all
administrators, for example:
Expand All @@ -46,6 +46,11 @@ services:
protocol: udp
# Comment the following line out if using docker-compose
mode: host
- target: 6514
published: 6514
protocol: tcp
# Comment the following line out if using docker-compose
mode: host
env_file:
- /opt/sc4s/env_file
volumes:
Expand Down Expand Up @@ -88,9 +93,10 @@ document for details on the directory structure the archive uses.
* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the
`docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup.

## Configure the SC4S environment
# Configure the SC4S environment

Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:
SC4S is almost entirely controlled through environment variables, which are read from a file at starteup. Create a file named
``/opt/sc4s/env_file`` and add the following environment variables and values:

```dotenv
SPLUNK_HEC_URL=https://splunk.smg.aws:8088
Expand All @@ -102,64 +108,46 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6

* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment.

* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP,
match this value to the total number of indexers behind the load balancer.
* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32.
If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer.

* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to
uncomment the last line in the example above.

## Modify index destinations for Splunk

Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers.

* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start.
* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your
environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in
this table that pertain to the individual data source filters that are included with SC4S.
* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further
information is covered in the "Log Path overrides" section of the Configuration document.

## Configure source filtering by source IP or host name

Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps
apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation.

* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start.
* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type.
* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents.
* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`.

## Configure compliance index/metadata overrides

In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion.
The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to
the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file
lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is
covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document.
## Configure SC4S Listening Ports

## Start/Restart SC4S
Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS.
The docker compose file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container
port mapping can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the
``published`` port(s) in the docker compose file (which represents the actual listening ports on the host machine), like so:

```bash
docker stack deploy --compose-file docker-compose.yml sc4s
```
ports:
- target: 514
published: 614
protocol: tcp
#Comment the following line out if using docker-compose
mode: host
```
This snippet above instructs the _host_ to listen on TCP port 614 and map that port to the default TCP 514 port on the _container_.
No changes to the underlying SC4S default configuration (environment variables) are needed.

# Scale out

Additional hosts can be deployed for syslog collection from additional network zones and locations.


# Configure Dedicated Listening Ports
### Dedicated (Unique) Listening Ports

For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in
the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise.
For collection of such sources we provide a means of dedicating a unique listening port to a specific source.
the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise.
For collection of such sources, we provide a means of dedicating a unique listening port to a specific source.

Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use.
The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured
by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the
``target`` and ``published`` lines provide for 21 additional technology-specific UDP and TCP ports.

In the following example the target port ranges allow for up to 21 technology-specific ports. Modify individual ports or a
range as appropriate for your network.
Follow these steps to configure unique ports:

* Modify the unit file ``/opt/sc4s/docker-compose.yml``
* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources"
documentation to identify the specific environment variables that are mapped to each data source vendor/technology.
* Modify the compose file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below.
* Restart SC4S using the command in the "Start/Restart SC4S" section below.
```yaml
version: "3.7"
services:
Expand All @@ -176,6 +164,11 @@ services:
protocol: udp
#Comment the following line out if using docker-compose
mode: host
- target: 6514
published: 6514
protocol: tcp
# Comment the following line out if using docker-compose
mode: host
- target: 5000-5020
published: 5000-5020
protocol: tcp
Expand All @@ -191,38 +184,46 @@ services:
volumes:
- /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z
- /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z
#Uncomment the following line if custom TLS certs are provided
# - /opt/sc4s/tls:/opt/syslog-ng/tls
# Uncomment the following line if local disk archiving is desired
# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z
# Uncomment the following line if custom TLS certs are provided
# - /opt/sc4s/tls:/opt/syslog-ng/tls:z
```

* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources"
section for more information on your specific device(s).
## Modify index destinations for Splunk

* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment
Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers.

* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP,
match this value to the total number of indexers behind the load balancer.
* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start.
* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your
environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in
this table that pertain to the individual data source filters that are included with SC4S.
* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further
information is covered in the "Log Path overrides" section of the Configuration document.

* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to
uncomment the last line in the example below.
## Configure source filtering by source IP or host name

```dotenv
SPLUNK_HEC_URL=https://splunk.smg.aws:8088
SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
SC4S_DEST_SPLUNK_HEC_WORKERS=6
SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
#Uncomment the following line if using untrusted SSL certificates
#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
```
Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps
apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation.

* Restart SC4S (below)
* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start.
* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type.
* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents.
* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`.

## Start/Restart SC4S
## Configure compliance index/metadata overrides

In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion.
The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to
the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file
lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is
covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document.

# Start/Restart SC4S

```bash
docker stack deploy --compose-file docker-compose.yml sc4s
```

# Stop SC4S

Start by obtaining the stack name (ID):
Expand All @@ -244,7 +245,7 @@ index=* sourcetype=sc4s:events "starting up"
```
This should yield the following event:
```ini
syslog-ng starting up; version='3.22.1'
syslog-ng starting up; version='3.25.1'
```
when the startup process proceeds normally (without syntax errors). If you do not see this,
follow the steps below before proceeding to deeper-level troubleshooting:
Expand All @@ -262,7 +263,7 @@ docker logs SC4S
```
You should see events similar to those below in the output:
```ini
Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.24.1'
Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.25.1'
Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)'
Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)'
```
Expand Down

0 comments on commit 4baa513

Please sign in to comment.