Skip to content

Commit

Permalink
Update docs for default and unique ports
Browse files Browse the repository at this point in the history 
* Update all runtime docs for default and unique ports
* Add default TLS port (6514) to all docker-compose and unit file examples
* Add default env variables to BYOE docs
* Runtime docs cleanup for flow/readability
  • Loading branch information
Mark Bonsack committed Jan 10, 2020
1 parent f860d79 commit fd77a51
Show file tree
Hide file tree
Showing 6 changed files with 258 additions and 248 deletions.
20 changes: 19 additions & 1 deletion docs/gettingstarted/byoe-rhel7.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ administration and syslog-ng configuration experience is assumed when using the

* NOTE: Do _not_ depend on the distribution-supplied version of syslog-ng, as it will likely be far too old.
Read this [explanation](https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-ng-on-rhel-and-other-rpm-distributions)
for the reason why syslog-ng builds are so dated in most RHEL/Debian distributions.
for the reason why syslog-ng builds are so dated in almost all RHEL/Debian distributions.

# BYOE Installation Instructions

Expand Down Expand Up @@ -157,3 +157,21 @@ sudo systemctl daemon-reload
sudo systemctl enable sc4s
sudo systemctl start sc4s
```
## Configure SC4S Listening Ports

Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS.
The docker compose file and standard SC4S configurations reflect these defaults. These defaults can be changed by adding the following
additional environment variables with appropriate values to the ``env_file`` above:
```dotenv
SC4S_LISTEN_DEFAULT_TCP_PORT=514
SC4S_LISTEN_DEFAULT_UDP_PORT=514
SC4S_LISTEN_DEFAULT_TLS_PORT=6514
```
### Dedicated (Unique) Listening Ports

For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in
the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise.
For collection of such sources we provide a means of dedicating a unique listening port to a specific source.

Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology
in use.
126 changes: 64 additions & 62 deletions docs/gettingstarted/docker-swarm-general.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ reboot:
net.ipv4.ip_forward=1
```

# SC4S Configuration
# SC4S Initial Configuration

* Create a directory on the server for local configurations and disk buffering. This should be available to all
administrators, for example:
Expand All @@ -46,6 +46,11 @@ services:
protocol: udp
# Comment the following line out if using docker-compose
mode: host
- target: 6514
published: 6514
protocol: tcp
# Comment the following line out if using docker-compose
mode: host
env_file:
- /opt/sc4s/env_file
volumes:
Expand Down Expand Up @@ -88,7 +93,7 @@ document for details on the directory structure the archive uses.
* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the
`docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup.

## Configure the SC4S environment
# Configure the SC4S environment

Create a file named ``/opt/sc4s/env_file`` and add the following environment variables:

Expand All @@ -108,58 +113,40 @@ match this value to the total number of indexers behind the load balancer.
* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to
uncomment the last line in the example above.

## Modify index destinations for Splunk

Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers.

* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start.
* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your
environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in
this table that pertain to the individual data source filters that are included with SC4S.
* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further
information is covered in the "Log Path overrides" section of the Configuration document.

## Configure source filtering by source IP or host name

Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps
apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation.

* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start.
* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type.
* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents.
* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`.

## Configure compliance index/metadata overrides
## Configure SC4S Listening Ports

In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion.
The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to
the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file
lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is
covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document.

## Start/Restart SC4S
Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS.
The docker compose file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container
port mapping can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the
``published`` port(s) in the docker compose file (which represents the actual listening ports on the host machine), like so:

```bash
docker stack deploy --compose-file docker-compose.yml sc4s
```
ports:
- target: 514
published: 614
protocol: tcp
#Comment the following line out if using docker-compose
mode: host
```
This snippet above instructs the _host_ to listen on TCP port 614 and map that port to the default TCP 514 port on the _container_.
No changes to the underlying SC4S default configuration (environment variables) are needed.

# Scale out

Additional hosts can be deployed for syslog collection from additional network zones and locations.


# Configure Dedicated Listening Ports
### Dedicated (Unique) Listening Ports

For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in
the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise.
For collection of such sources we provide a means of dedicating a unique listening port to a specific source.

Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use.
Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology
in use.

In the following example the target port ranges allow for up to 21 technology-specific ports. Modify individual ports or a
range as appropriate for your network.
The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured
by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the
``target`` and ``published`` lines provide for 21 additional technology-specific ports. Follow these steps to configure unique ports:

* Modify the unit file ``/opt/sc4s/docker-compose.yml``
* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources"
section for more information on your specific device(s).
* Modify the unit file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below:
```yaml
version: "3.7"
services:
Expand All @@ -176,6 +163,11 @@ services:
protocol: udp
#Comment the following line out if using docker-compose
mode: host
- target: 6514
published: 6514
protocol: tcp
# Comment the following line out if using docker-compose
mode: host
- target: 5000-5020
published: 5000-5020
protocol: tcp
Expand All @@ -195,34 +187,44 @@ services:
# - /opt/sc4s/tls:/opt/syslog-ng/tls
```

* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources"
section for more information on your specific device(s).
## Modify index destinations for Splunk

* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment
Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers.

* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP,
match this value to the total number of indexers behind the load balancer.
* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start.
* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your
environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in
this table that pertain to the individual data source filters that are included with SC4S.
* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further
information is covered in the "Log Path overrides" section of the Configuration document.

* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to
uncomment the last line in the example below.
## Configure source filtering by source IP or host name

```dotenv
SPLUNK_HEC_URL=https://splunk.smg.aws:8088
SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94
SC4S_DEST_SPLUNK_HEC_WORKERS=6
SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000
#Uncomment the following line if using untrusted SSL certificates
#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
```
Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps
apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation.

* Restart SC4S (below)
* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start.
* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type.
* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents.
* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`.

## Start/Restart SC4S
## Configure compliance index/metadata overrides

In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion.
The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to
the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file
lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is
covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document.

# Scale out

Additional hosts can be deployed for syslog collection from additional network zones and locations.

# Start/Restart SC4S

```bash
docker stack deploy --compose-file docker-compose.yml sc4s
```

# Stop SC4S

Start by obtaining the stack name (ID):
Expand Down Expand Up @@ -262,7 +264,7 @@ docker logs SC4S
```
You should see events similar to those below in the output:
```ini
Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.24.1'
Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.25.1'
Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)'
Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)'
```
Expand Down

0 comments on commit fd77a51

Please sign in to comment.