From fd77a51bb589af6b631172f87191878226b03d8e Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Fri, 10 Jan 2020 12:31:53 -0800 Subject: [PATCH 1/2] Update docs for default and unique ports * Update all runtime docs for default and unique ports * Add default TLS port (6514) to all docker-compose and unit file examples * Add default env variables to BYOE docs * Runtime docs cleanup for flow/readability --- docs/gettingstarted/byoe-rhel7.md | 20 ++- docs/gettingstarted/docker-swarm-general.md | 126 ++++++++--------- docs/gettingstarted/docker-swarm-rhel7.md | 128 +++++++++--------- docs/gettingstarted/docker-systemd-general.md | 111 ++++++++------- docs/gettingstarted/index.md | 10 +- docs/gettingstarted/podman-systemd-general.md | 111 ++++++++------- 6 files changed, 258 insertions(+), 248 deletions(-) diff --git a/docs/gettingstarted/byoe-rhel7.md b/docs/gettingstarted/byoe-rhel7.md index 59baff5..437deb8 100644 --- a/docs/gettingstarted/byoe-rhel7.md +++ b/docs/gettingstarted/byoe-rhel7.md @@ -17,7 +17,7 @@ administration and syslog-ng configuration experience is assumed when using the * NOTE: Do _not_ depend on the distribution-supplied version of syslog-ng, as it will likely be far too old. Read this [explanation](https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-ng-on-rhel-and-other-rpm-distributions) -for the reason why syslog-ng builds are so dated in most RHEL/Debian distributions. +for the reason why syslog-ng builds are so dated in almost all RHEL/Debian distributions. # BYOE Installation Instructions @@ -157,3 +157,21 @@ sudo systemctl daemon-reload sudo systemctl enable sc4s sudo systemctl start sc4s ``` +## Configure SC4S Listening Ports + +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The docker compose file and standard SC4S configurations reflect these defaults. These defaults can be changed by adding the following +additional environment variables with appropriate values to the ``env_file`` above: +```dotenv +SC4S_LISTEN_DEFAULT_TCP_PORT=514 +SC4S_LISTEN_DEFAULT_UDP_PORT=514 +SC4S_LISTEN_DEFAULT_TLS_PORT=6514 +``` +### Dedicated (Unique) Listening Ports + +For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in +the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. +For collection of such sources we provide a means of dedicating a unique listening port to a specific source. + +Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology +in use. \ No newline at end of file diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index 44d2255..7f72222 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -22,7 +22,7 @@ reboot: net.ipv4.ip_forward=1 ``` -# SC4S Configuration +# SC4S Initial Configuration * Create a directory on the server for local configurations and disk buffering. This should be available to all administrators, for example: @@ -46,6 +46,11 @@ services: protocol: udp # Comment the following line out if using docker-compose mode: host + - target: 6514 + published: 6514 + protocol: tcp +# Comment the following line out if using docker-compose + mode: host env_file: - /opt/sc4s/env_file volumes: @@ -88,7 +93,7 @@ document for details on the directory structure the archive uses. * IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the `docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup. -## Configure the SC4S environment +# Configure the SC4S environment Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: @@ -108,58 +113,40 @@ match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example above. -## Modify index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. - -* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your -environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in -this table that pertain to the individual data source filters that are included with SC4S. -* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further -information is covered in the "Log Path overrides" section of the Configuration document. - -## Configure source filtering by source IP or host name - -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. - -* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. - -## Configure compliance index/metadata overrides +## Configure SC4S Listening Ports -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. -The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to -the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is -covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. - -## Start/Restart SC4S +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The docker compose file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container +port mapping can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the +``published`` port(s) in the docker compose file (which represents the actual listening ports on the host machine), like so: -```bash -docker stack deploy --compose-file docker-compose.yml sc4s ``` + ports: + - target: 514 + published: 614 + protocol: tcp +#Comment the following line out if using docker-compose + mode: host +``` +This snippet above instructs the _host_ to listen on TCP port 614 and map that port to the default TCP 514 port on the _container_. +No changes to the underlying SC4S default configuration (environment variables) are needed. -# Scale out - -Additional hosts can be deployed for syslog collection from additional network zones and locations. - - -# Configure Dedicated Listening Ports +### Dedicated (Unique) Listening Ports For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. For collection of such sources we provide a means of dedicating a unique listening port to a specific source. -Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. +Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology +in use. -In the following example the target port ranges allow for up to 21 technology-specific ports. Modify individual ports or a -range as appropriate for your network. +The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured +by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the +``target`` and ``published`` lines provide for 21 additional technology-specific ports. Follow these steps to configure unique ports: -* Modify the unit file ``/opt/sc4s/docker-compose.yml`` +* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" +section for more information on your specific device(s). +* Modify the unit file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below: ```yaml version: "3.7" services: @@ -176,6 +163,11 @@ services: protocol: udp #Comment the following line out if using docker-compose mode: host + - target: 6514 + published: 6514 + protocol: tcp +# Comment the following line out if using docker-compose + mode: host - target: 5000-5020 published: 5000-5020 protocol: tcp @@ -195,34 +187,44 @@ services: # - /opt/sc4s/tls:/opt/syslog-ng/tls ``` -* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). +## Modify index destinations for Splunk -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +## Configure source filtering by source IP or host name -```dotenv -SPLUNK_HEC_URL=https://splunk.smg.aws:8088 -SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 -SC4S_DEST_SPLUNK_HEC_WORKERS=6 -SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 -#Uncomment the following line if using untrusted SSL certificates -#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no -``` +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* Restart SC4S (below) +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. -## Start/Restart SC4S +## Configure compliance index/metadata overrides + +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. + +# Scale out + +Additional hosts can be deployed for syslog collection from additional network zones and locations. + +# Start/Restart SC4S ```bash docker stack deploy --compose-file docker-compose.yml sc4s ``` - # Stop SC4S Start by obtaining the stack name (ID): @@ -262,7 +264,7 @@ docker logs SC4S ``` You should see events similar to those below in the output: ```ini -Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.24.1' +Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.25.1' Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' ``` diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index 0af8b6b..09f8179 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -31,7 +31,7 @@ systemctl start docker.service sudo docker swarm init ``` -# SC4S Configuration +# SC4S Initial Configuration * Create a directory on the server for local configurations and disk buffering. This should be available to all administrators, for example: ``/opt/sc4s/`` @@ -52,6 +52,11 @@ services: - target: 514 published: 514 protocol: udp +# Comment the following line out if using docker-compose + mode: host + - target: 6514 + published: 6514 + protocol: tcp # Comment the following line out if using docker-compose mode: host env_file: @@ -96,7 +101,7 @@ document for details on the directory structure the archive uses. * IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the `docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup. -## Configure the SC4S environment +# Configure the SC4S environment Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: @@ -116,60 +121,40 @@ match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example below. +## Configure SC4S Listening Ports -## Modify index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. - -* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your -environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in -this table that pertain to the individual data source filters that are included with SC4S. -* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further -information is covered in the "Log Path overrides" section of the Configuration document. - - -## Configure source filtering by source IP or host name - -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. - -* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. - -## Configure compliance index/metadata overrides - -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. -The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to -the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is -covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. - -## Start/Restart SC4S +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The docker compose file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container +port mapping can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the +``published`` port(s) in the docker compose file (which represents the actual listening ports on the host machine), like so: -```bash -sudo docker stack deploy --compose-file docker-compose.yml sc4s ``` + ports: + - target: 514 + published: 614 + protocol: tcp +#Comment the following line out if using docker-compose + mode: host +``` +This snippet above instructs the _host_ to listen on TCP port 614 and map that port to the default TCP 514 port on the _container_. +No changes to the underlying SC4S default configuration (environment variables) are needed. -# Scale out - -Additional hosts can be deployed for syslog collection from additional network zones and locations. - - -# Configure Dedicated Listening Ports +### Dedicated (Unique) Listening Ports For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. For collection of such sources we provide a means of dedicating a unique listening port to a specific source. -Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. +Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology +in use. -In the following example the target port ranges allow for up to 21 technology-specific ports. Modify individual ports or a -range as appropriate for your network. +The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured +by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the +``target`` and ``published`` lines provide for 21 additional technology-specific ports. Follow these steps to configure unique ports: -* Modify the unit file ``/opt/sc4s/docker-compose.yml`` +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). See the "Sources" +section for more information on your specific device(s). +* Modify the unit file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below: ```yaml version: "3.7" services: @@ -186,6 +171,11 @@ services: protocol: udp #Comment the following line out if using docker-compose mode: host + - target: 6514 + published: 6514 + protocol: tcp +# Comment the following line out if using docker-compose + mode: host - target: 5000-5020 published: 5000-5020 protocol: tcp @@ -205,34 +195,44 @@ services: # - /opt/sc4s/tls:/opt/syslog-ng/tls ``` -* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). +## Modify index destinations for Splunk -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +## Configure source filtering by source IP or host name -```dotenv -SPLUNK_HEC_URL=https://splunk.smg.aws:8088 -SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 -SC4S_DEST_SPLUNK_HEC_WORKERS=6 -SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 -#Uncomment the following line if using untrusted SSL certificates -#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no -``` +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. + +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. + +## Configure compliance index/metadata overrides -* Restart SC4S (below) +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. -## Start/Restart SC4S +# Scale out + +Additional hosts can be deployed for syslog collection from additional network zones and locations. + +# Start/Restart SC4S ```bash docker stack deploy --compose-file docker-compose.yml sc4s ``` - # Stop SC4S Start by obtaining the stack name (ID): @@ -272,7 +272,7 @@ docker logs SC4S ``` You should see events similar to those below in the output: ```ini -Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.24.1' +Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.25.1' Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' ``` diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 7eae5b0..27834bf 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -21,7 +21,7 @@ reboot: net.ipv4.ip_forward=1 ``` -# Setup +# Initial Setup * Create the systemd unit file `/lib/systemd/system/sc4s.service` based on the following template: @@ -54,7 +54,7 @@ ExecStartPre=/usr/bin/docker run \ "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s -ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp \ +ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ @@ -95,7 +95,7 @@ document for details on the directory structure the archive uses. * IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the unit file above. Failure to do this will cause SC4S to abort at startup. -## Configure the SC4S environment +# Configure the SC4S environment Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: @@ -115,55 +115,35 @@ match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example. -## Modify index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. - -* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your -environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in -this table that pertain to the individual data source filters that are included with SC4S. -* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further -information is covered in the "Log Path overrides" section of the Configuration document. +## Configure SC4S Listening Ports -## Configure source filtering by source IP or host name - -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. - -* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The unit file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container port mapping +can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the initial port in the +`ExecStart` line for the main container (which represents the actual listening port on the host machine), like so: -## Configure compliance index/metadata overrides - -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. -The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to -the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is -covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. - -## Configure SC4S for systemd and start SC4S - -```bash -sudo systemctl daemon-reload -sudo systemctl enable sc4s -sudo systemctl start sc4s ``` +-p 614:514 -p 714:514/udp -p 8514:6514 +``` +This instructs the _host_ to listen on TCP port 614, UDP 714, and TCP 8514 (for TLS) and map them to the standard UDP/TCP 514 and 6514 ports +on the _container_. No changes to the underlying SC4S default configuration (environment variables) are needed. -# Configure Dedicated Listening Ports +### Dedicated (Unique) Listening Ports For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. For collection of such sources we provide a means of dedicating a unique listening port to a specific source. -Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. +Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology +in use. -In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify individual ports or a -range as appropriate for your network. +The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the +environment variable(s). In the following example, the `ExecStart` line for the main SC4S container is modified, where +``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Follow these steps to configure unique ports: -* Modify the unit file ``/lib/systemd/system/sc4s.service`` +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). See the "Sources" +section for more information on your specific device(s). +* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below: ```ini [Unit] Description=SC4S Container @@ -190,7 +170,7 @@ ExecStartPre=/usr/bin/docker run \ "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s -ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ +ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ @@ -199,27 +179,42 @@ ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 - $SC4S_IMAGE ``` -* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). +## Modify index destinations for Splunk -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +## Configure source filtering by source IP or host name -```dotenv -SPLUNK_HEC_URL=https://splunk.smg.aws:8088 -SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 -SC4S_DEST_SPLUNK_HEC_WORKERS=6 -SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 -#Uncomment the following line if using untrusted SSL certificates -#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no -``` +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. + +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. + +## Configure compliance index/metadata overrides -* Restart SC4S (below) +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. + +## Configure SC4S for systemd and start SC4S + +```bash +sudo systemctl daemon-reload +sudo systemctl enable sc4s +sudo systemctl start sc4s +``` # Start SC4S diff --git a/docs/gettingstarted/index.md b/docs/gettingstarted/index.md index f686dc0..b8258f6 100644 --- a/docs/gettingstarted/index.md +++ b/docs/gettingstarted/index.md @@ -76,11 +76,11 @@ Splunk type. | Container and Orchestration | Notes | |-----------------------------|-------| -| [Podman + systemd single node](gettingstarted/podman-systemd-general.md) | First choice for RedHat 7.x/8.x and CentOS, second choice for Debian and Ubuntu (packages provided via PPA) | -| [Docker CE + systemd single node](gettingstarted/docker-systemd-general.md) | First choice for Debian and Ubuntu; second choice for CentOS for those with limited existing Docker experience | -| [Docker CE + Swarm single node](gettingstarted/docker-swarm-general.md) | Option for Debian, Ubuntu, CentOS, and Desktop Docker desiring Docker Compose or Swarm orchestration | -| [Docker CE + Swarm single node RHEL 7.7](gettingstarted/docker-swarm-rhel7.md) | Option for RedHat 7.7 desiring Docker Compose or Swarm orchestration | -| [Bring your own Envionment](gettingstarted/byoe-rhel7.md) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers | +| [Podman + systemd single node](gettingstarted/podman-systemd-general) | First choice for RedHat 7.x/8.x and CentOS, second choice for Debian and Ubuntu (packages provided via PPA) | +| [Docker CE + systemd single node](gettingstarted/docker-systemd-general) | First choice for Debian and Ubuntu; second choice for CentOS for those with limited existing Docker experience | +| [Docker CE + Swarm single node](gettingstarted/docker-swarm-general) | Option for Debian, Ubuntu, CentOS, and Desktop Docker desiring Docker Compose or Swarm orchestration | +| [Docker CE + Swarm single node RHEL 7.7](gettingstarted/docker-swarm-rhel7) | Option for RedHat 7.7 desiring Docker Compose or Swarm orchestration | +| [Bring your own Envionment](gettingstarted/byoe-rhel7) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers | ### Offline Container Installation diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 18f8f82..2ecc610 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -3,7 +3,7 @@ Refer to [Installation](https://podman.io/getting-started/installation) -# Setup +# Initial Setup * Create the systemd unit file `/lib/systemd/system/sc4s.service` based on the following template: @@ -36,7 +36,7 @@ ExecStartPre=/usr/bin/podman run \ "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s -ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp \ +ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ @@ -77,7 +77,7 @@ document for details on the directory structure the archive uses. * IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the unit file above. Failure to do this will cause SC4S to abort at startup. -## Configure the sc4s environment +# Configure the sc4s environment Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: @@ -97,55 +97,35 @@ match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example. -## Modify index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. - -* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your -environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in -this table that pertain to the individual data source filters that are included with SC4S. -* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further -information is covered in the "Log Path overrides" section of the Configuration document. +## Configure SC4S Listening Ports -## Configure source filtering by source IP or host name - -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. - -* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The unit file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container port mapping +can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the initial port in the +`ExecStart` line for the main container (which represents the actual listening port on the host machine), like so: -## Configure compliance index/metadata overrides - -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. -The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to -the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is -covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. - -## Configure SC4S for systemd and start SC4S - -```bash -sudo systemctl daemon-reload -sudo systemctl enable sc4s -sudo systemctl start sc4s ``` +-p 614:514 -p 714:514/udp -p 8514:6514 +``` +This instructs the _host_ to listen on TCP port 614, UDP 714, and TCP 8514 (for TLS) and map them to the standard UDP/TCP 514 and 6514 ports +on the _container_. No changes to the underlying SC4S default configuration (environment variables) are needed. -# Configure Dedicated Listening Ports +### Dedicated (Unique) Listening Ports For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. For collection of such sources we provide a means of dedicating a unique listening port to a specific source. -Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. +Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology +in use. -In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify individual ports or a -range as appropriate for your network. +The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the +environment variable(s). In the following example, the `ExecStart` line for the main SC4S container is modified, where +``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Follow these steps to configure unique ports: -* Modify the unit file ``/lib/systemd/system/sc4s.service`` +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). See the "Sources" +section for more information on your specific device(s). +* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below: ```ini [Unit] Description=SC4S Container @@ -172,7 +152,7 @@ ExecStartPre=/usr/bin/podman run \ "$SC4S_LOCAL_CONFIG_MOUNT" \ --name SC4S_preflight --rm \ $SC4S_IMAGE -s -ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ +ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ @@ -181,27 +161,42 @@ ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 - $SC4S_IMAGE ``` -* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). +## Modify index destinations for Splunk -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +## Configure source filtering by source IP or host name -```dotenv -SPLUNK_HEC_URL=https://splunk.smg.aws:8088 -SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 -SC4S_DEST_SPLUNK_HEC_WORKERS=6 -SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 -#Uncomment the following line if using untrusted SSL certificates -#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no -``` +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* Restart SC4S (below) +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. + +## Configure compliance index/metadata overrides + +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. + +## Configure SC4S for systemd and start SC4S + +```bash +sudo systemctl daemon-reload +sudo systemctl enable sc4s +sudo systemctl start sc4s +``` # Start SC4S From f52c5aa45d61715d4f21634b65d6b68d5d986282 Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Sat, 11 Jan 2020 17:23:26 -0800 Subject: [PATCH 2/2] Runtime docs cleanup * Clean up `docker-compose.yml` and systemd `unit` files to include archive options * Clean up compose/unit file command layout * Clarify instructions for unique port setup --- docs/gettingstarted/docker-swarm-general.md | 29 +++++---- docs/gettingstarted/docker-swarm-rhel7.md | 31 +++++---- docs/gettingstarted/docker-systemd-general.md | 64 +++++++++++-------- docs/gettingstarted/podman-systemd-general.md | 58 ++++++++++------- 4 files changed, 104 insertions(+), 78 deletions(-) diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index 7f72222..672f1d8 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -95,7 +95,8 @@ document for details on the directory structure the archive uses. # Configure the SC4S environment -Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: +SC4S is almost entirely controlled through environment variables, which are read from a file at starteup. Create a file named +``/opt/sc4s/env_file`` and add the following environment variables and values: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088 @@ -107,8 +108,8 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. +If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example above. @@ -135,18 +136,18 @@ No changes to the underlying SC4S default configuration (environment variables) For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. -For collection of such sources we provide a means of dedicating a unique listening port to a specific source. - -Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology -in use. +For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the -``target`` and ``published`` lines provide for 21 additional technology-specific ports. Follow these steps to configure unique ports: +``target`` and ``published`` lines provide for 21 additional technology-specific UDP and TCP ports. + +Follow these steps to configure unique ports: -* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). -* Modify the unit file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below: +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" +documentation to identify the specific environment variables that are mapped to each data source vendor/technology. +* Modify the compose file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below. +* Restart SC4S using the command in the "Start/Restart SC4S" section below. ```yaml version: "3.7" services: @@ -183,8 +184,10 @@ services: volumes: - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z -#Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls +# Uncomment the following line if local disk archiving is desired +# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z +# Uncomment the following line if custom TLS certs are provided +# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` ## Modify index destinations for Splunk diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index 09f8179..2fbe187 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -103,7 +103,8 @@ document for details on the directory structure the archive uses. # Configure the SC4S environment -Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: +SC4S is almost entirely controlled through environment variables, which are read from a file at starteup. Create a file named +``/opt/sc4s/env_file`` and add the following environment variables and values: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088 @@ -115,11 +116,11 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. +If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +uncomment the last line in the example above. ## Configure SC4S Listening Ports @@ -143,18 +144,18 @@ No changes to the underlying SC4S default configuration (environment variables) For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. -For collection of such sources we provide a means of dedicating a unique listening port to a specific source. - -Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology -in use. +For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the -``target`` and ``published`` lines provide for 21 additional technology-specific ports. Follow these steps to configure unique ports: +``target`` and ``published`` lines provide for 21 additional technology-specific UDP and TCP ports. + +Follow these steps to configure unique ports: -* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). -* Modify the unit file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below: +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" +documentation to identify the specific environment variables that are mapped to each data source vendor/technology. +* Modify the compose file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below. +* Restart SC4S using the command in the "Start/Restart SC4S" section below. ```yaml version: "3.7" services: @@ -191,8 +192,10 @@ services: volumes: - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z -#Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls +# Uncomment the following line if local disk archiving is desired +# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z +# Uncomment the following line if custom TLS certs are provided +# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` ## Modify index destinations for Splunk diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 27834bf..03f3d8f 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -38,10 +38,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest" Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" +# Optional mount point for local disk archive (EWMM output) files + +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" -# Uncomment the following line if local disk archiving is desired -# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Uncomment the following line if custom TLS certs are provided # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z" @@ -52,16 +55,16 @@ ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE ExecStartPre=/usr/bin/docker run \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight --rm \ - $SC4S_IMAGE -s -ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 \ + --name SC4S_preflight \ + --rm $SC4S_IMAGE -s +ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ "$SC4S_LOCAL_ARCHIVE_MOUNT" \ "$SC4S_TLS_DIR" \ - --name SC4S --rm \ -$SC4S_IMAGE + --name SC4S \ + --rm $SC4S_IMAGE ``` * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. @@ -95,9 +98,10 @@ document for details on the directory structure the archive uses. * IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the unit file above. Failure to do this will cause SC4S to abort at startup. -# Configure the SC4S environment +# Configure the sc4s environment -Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: +SC4S is almost entirely controlled through environment variables, which are read from a file at starteup. Create a file named +``/opt/sc4s/env_file`` and add the following environment variables and values: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088 @@ -109,18 +113,18 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. +If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example. +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example above. ## Configure SC4S Listening Ports Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. The unit file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container port mapping can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the initial port in the -`ExecStart` line for the main container (which represents the actual listening port on the host machine), like so: +`ExecStart` line in the unit file for the main container (which represents the actual listening port on the host machine), like so: ``` -p 614:514 -p 714:514/udp -p 8514:6514 @@ -132,18 +136,18 @@ on the _container_. No changes to the underlying SC4S default configuration (en For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. -For collection of such sources we provide a means of dedicating a unique listening port to a specific source. - -Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology -in use. +For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the -environment variable(s). In the following example, the `ExecStart` line for the main SC4S container is modified, where -``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Follow these steps to configure unique ports: +environment variable(s). In the example below, the `ExecStart` line for the main SC4S container is modified, where +``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. -* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). -* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below: +Follow these steps to configure unique ports: + +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" +documentation to identify the specific environment variables that are mapped to each data source vendor/technology. +* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below. +* Ensure that you reload the unit file as well as restarting SC4S. See the "Configure SC4S for systemd and start SC4S" section below. ```ini [Unit] Description=SC4S Container @@ -157,8 +161,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest" Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" +# Optional mount point for local disk archive (EWMM output) files + +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" + # Uncomment the following line if custom TLS certs are provided # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" @@ -168,15 +177,16 @@ ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE ExecStartPre=/usr/bin/docker run \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight --rm \ - $SC4S_IMAGE -s + --name SC4S_preflight \ + --rm $SC4S_IMAGE -s ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ + "$SC4S_LOCAL_ARCHIVE_MOUNT" \ + "$SC4S_TLS_DIR" \ --name SC4S \ - --rm \ -$SC4S_IMAGE + --rm $SC4S_IMAGE ``` ## Modify index destinations for Splunk diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 2ecc610..51356a8 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -20,10 +20,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest" Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" +# Optional mount point for local disk archive (EWMM output) files + +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" -# Uncomment the following line if local disk archiving is desired -# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Uncomment the following line if custom TLS certs are provided # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z" @@ -34,16 +37,16 @@ ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE ExecStartPre=/usr/bin/podman run \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight --rm \ - $SC4S_IMAGE -s + --name SC4S_preflight \ + --rm $SC4S_IMAGE -s ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ "$SC4S_LOCAL_ARCHIVE_MOUNT" \ "$SC4S_TLS_DIR" \ - --name SC4S --rm \ -$SC4S_IMAGE + --name SC4S \ + --rm $SC4S_IMAGE ``` * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. @@ -79,7 +82,8 @@ unit file above. Failure to do this will cause SC4S to abort at startup. # Configure the sc4s environment -Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: +SC4S is almost entirely controlled through environment variables, which are read from a file at starteup. Create a file named +``/opt/sc4s/env_file`` and add the following environment variables and values: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088 @@ -91,18 +95,18 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. +If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example. +uncomment the last line in the example above. ## Configure SC4S Listening Ports Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. The unit file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container port mapping can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the initial port in the -`ExecStart` line for the main container (which represents the actual listening port on the host machine), like so: +`ExecStart` line in the unit file for the main container (which represents the actual listening port on the host machine), like so: ``` -p 614:514 -p 714:514/udp -p 8514:6514 @@ -114,18 +118,18 @@ on the _container_. No changes to the underlying SC4S default configuration (en For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. -For collection of such sources we provide a means of dedicating a unique listening port to a specific source. - -Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology -in use. +For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the -environment variable(s). In the following example, the `ExecStart` line for the main SC4S container is modified, where -``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Follow these steps to configure unique ports: +environment variable(s). In the example below, the `ExecStart` line for the main SC4S container is modified, where +``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. -* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). -* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below: +Follow these steps to configure unique ports: + +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" +documentation to identify the specific environment variables that are mapped to each data source vendor/technology. +* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below. +* Ensure that you reload the unit file as well as restarting SC4S. See the "Configure SC4S for systemd and start SC4S" section below. ```ini [Unit] Description=SC4S Container @@ -139,8 +143,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest" Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" +# Optional mount point for local disk archive (EWMM output) files + +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" + # Uncomment the following line if custom TLS certs are provided # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" @@ -150,15 +159,16 @@ ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE ExecStartPre=/usr/bin/podman run \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight --rm \ - $SC4S_IMAGE -s + --name SC4S_preflight \ + --rm $SC4S_IMAGE -s ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ + "$SC4S_LOCAL_ARCHIVE_MOUNT" \ + "$SC4S_TLS_DIR" \ --name SC4S \ - --rm \ -$SC4S_IMAGE + --rm $SC4S_IMAGE ``` ## Modify index destinations for Splunk