diff --git a/docs/gettingstarted/byoe-rhel7.md b/docs/gettingstarted/byoe-rhel7.md index 59baff5..437deb8 100644 --- a/docs/gettingstarted/byoe-rhel7.md +++ b/docs/gettingstarted/byoe-rhel7.md @@ -17,7 +17,7 @@ administration and syslog-ng configuration experience is assumed when using the * NOTE: Do _not_ depend on the distribution-supplied version of syslog-ng, as it will likely be far too old. Read this [explanation](https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-ng-on-rhel-and-other-rpm-distributions) -for the reason why syslog-ng builds are so dated in most RHEL/Debian distributions. +for the reason why syslog-ng builds are so dated in almost all RHEL/Debian distributions. # BYOE Installation Instructions @@ -157,3 +157,21 @@ sudo systemctl daemon-reload sudo systemctl enable sc4s sudo systemctl start sc4s ``` +## Configure SC4S Listening Ports + +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The docker compose file and standard SC4S configurations reflect these defaults. These defaults can be changed by adding the following +additional environment variables with appropriate values to the ``env_file`` above: +```dotenv +SC4S_LISTEN_DEFAULT_TCP_PORT=514 +SC4S_LISTEN_DEFAULT_UDP_PORT=514 +SC4S_LISTEN_DEFAULT_TLS_PORT=6514 +``` +### Dedicated (Unique) Listening Ports + +For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in +the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. +For collection of such sources we provide a means of dedicating a unique listening port to a specific source. + +Refer to the "Sources" documentation to identify the specific environment variables used to enable unique listening ports for the technology +in use. \ No newline at end of file diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index 44d2255..672f1d8 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -22,7 +22,7 @@ reboot: net.ipv4.ip_forward=1 ``` -# SC4S Configuration +# SC4S Initial Configuration * Create a directory on the server for local configurations and disk buffering. This should be available to all administrators, for example: @@ -46,6 +46,11 @@ services: protocol: udp # Comment the following line out if using docker-compose mode: host + - target: 6514 + published: 6514 + protocol: tcp +# Comment the following line out if using docker-compose + mode: host env_file: - /opt/sc4s/env_file volumes: @@ -88,9 +93,10 @@ document for details on the directory structure the archive uses. * IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the `docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup. -## Configure the SC4S environment +# Configure the SC4S environment -Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: +SC4S is almost entirely controlled through environment variables, which are read from a file at starteup. Create a file named +``/opt/sc4s/env_file`` and add the following environment variables and values: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088 @@ -102,64 +108,46 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. +If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example above. -## Modify index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. - -* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your -environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in -this table that pertain to the individual data source filters that are included with SC4S. -* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further -information is covered in the "Log Path overrides" section of the Configuration document. - -## Configure source filtering by source IP or host name +## Configure SC4S Listening Ports -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The docker compose file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container +port mapping can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the +``published`` port(s) in the docker compose file (which represents the actual listening ports on the host machine), like so: -* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. - -## Configure compliance index/metadata overrides - -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. -The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to -the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is -covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. - -## Start/Restart SC4S - -```bash -docker stack deploy --compose-file docker-compose.yml sc4s ``` + ports: + - target: 514 + published: 614 + protocol: tcp +#Comment the following line out if using docker-compose + mode: host +``` +This snippet above instructs the _host_ to listen on TCP port 614 and map that port to the default TCP 514 port on the _container_. +No changes to the underlying SC4S default configuration (environment variables) are needed. -# Scale out - -Additional hosts can be deployed for syslog collection from additional network zones and locations. - - -# Configure Dedicated Listening Ports +### Dedicated (Unique) Listening Ports For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. -For collection of such sources we provide a means of dedicating a unique listening port to a specific source. +For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. -Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. +The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured +by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the +``target`` and ``published`` lines provide for 21 additional technology-specific UDP and TCP ports. -In the following example the target port ranges allow for up to 21 technology-specific ports. Modify individual ports or a -range as appropriate for your network. +Follow these steps to configure unique ports: -* Modify the unit file ``/opt/sc4s/docker-compose.yml`` +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" +documentation to identify the specific environment variables that are mapped to each data source vendor/technology. +* Modify the compose file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below. +* Restart SC4S using the command in the "Start/Restart SC4S" section below. ```yaml version: "3.7" services: @@ -176,6 +164,11 @@ services: protocol: udp #Comment the following line out if using docker-compose mode: host + - target: 6514 + published: 6514 + protocol: tcp +# Comment the following line out if using docker-compose + mode: host - target: 5000-5020 published: 5000-5020 protocol: tcp @@ -191,38 +184,50 @@ services: volumes: - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z -#Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls +# Uncomment the following line if local disk archiving is desired +# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z +# Uncomment the following line if custom TLS certs are provided +# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` -* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). +## Modify index destinations for Splunk -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +## Configure source filtering by source IP or host name -```dotenv -SPLUNK_HEC_URL=https://splunk.smg.aws:8088 -SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 -SC4S_DEST_SPLUNK_HEC_WORKERS=6 -SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 -#Uncomment the following line if using untrusted SSL certificates -#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no -``` +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* Restart SC4S (below) +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. -## Start/Restart SC4S +## Configure compliance index/metadata overrides + +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. + +# Scale out + +Additional hosts can be deployed for syslog collection from additional network zones and locations. + +# Start/Restart SC4S ```bash docker stack deploy --compose-file docker-compose.yml sc4s ``` - # Stop SC4S Start by obtaining the stack name (ID): @@ -262,7 +267,7 @@ docker logs SC4S ``` You should see events similar to those below in the output: ```ini -Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.24.1' +Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.25.1' Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' ``` diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index 0af8b6b..2fbe187 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -31,7 +31,7 @@ systemctl start docker.service sudo docker swarm init ``` -# SC4S Configuration +# SC4S Initial Configuration * Create a directory on the server for local configurations and disk buffering. This should be available to all administrators, for example: ``/opt/sc4s/`` @@ -52,6 +52,11 @@ services: - target: 514 published: 514 protocol: udp +# Comment the following line out if using docker-compose + mode: host + - target: 6514 + published: 6514 + protocol: tcp # Comment the following line out if using docker-compose mode: host env_file: @@ -96,9 +101,10 @@ document for details on the directory structure the archive uses. * IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the `docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup. -## Configure the SC4S environment +# Configure the SC4S environment -Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: +SC4S is almost entirely controlled through environment variables, which are read from a file at starteup. Create a file named +``/opt/sc4s/env_file`` and add the following environment variables and values: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088 @@ -110,66 +116,46 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. +If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. - - -## Modify index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. - -* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your -environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in -this table that pertain to the individual data source filters that are included with SC4S. -* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further -information is covered in the "Log Path overrides" section of the Configuration document. - - -## Configure source filtering by source IP or host name - -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. - -* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. - -## Configure compliance index/metadata overrides +uncomment the last line in the example above. -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. -The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to -the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is -covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. +## Configure SC4S Listening Ports -## Start/Restart SC4S +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The docker compose file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container +port mapping can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the +``published`` port(s) in the docker compose file (which represents the actual listening ports on the host machine), like so: -```bash -sudo docker stack deploy --compose-file docker-compose.yml sc4s ``` + ports: + - target: 514 + published: 614 + protocol: tcp +#Comment the following line out if using docker-compose + mode: host +``` +This snippet above instructs the _host_ to listen on TCP port 614 and map that port to the default TCP 514 port on the _container_. +No changes to the underlying SC4S default configuration (environment variables) are needed. -# Scale out - -Additional hosts can be deployed for syslog collection from additional network zones and locations. - - -# Configure Dedicated Listening Ports +### Dedicated (Unique) Listening Ports For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. -For collection of such sources we provide a means of dedicating a unique listening port to a specific source. +For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. -Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. +The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured +by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the +``target`` and ``published`` lines provide for 21 additional technology-specific UDP and TCP ports. -In the following example the target port ranges allow for up to 21 technology-specific ports. Modify individual ports or a -range as appropriate for your network. +Follow these steps to configure unique ports: -* Modify the unit file ``/opt/sc4s/docker-compose.yml`` +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" +documentation to identify the specific environment variables that are mapped to each data source vendor/technology. +* Modify the compose file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below. +* Restart SC4S using the command in the "Start/Restart SC4S" section below. ```yaml version: "3.7" services: @@ -186,6 +172,11 @@ services: protocol: udp #Comment the following line out if using docker-compose mode: host + - target: 6514 + published: 6514 + protocol: tcp +# Comment the following line out if using docker-compose + mode: host - target: 5000-5020 published: 5000-5020 protocol: tcp @@ -201,38 +192,50 @@ services: volumes: - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z -#Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls +# Uncomment the following line if local disk archiving is desired +# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z +# Uncomment the following line if custom TLS certs are provided +# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` -* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). +## Modify index destinations for Splunk -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +## Configure source filtering by source IP or host name -```dotenv -SPLUNK_HEC_URL=https://splunk.smg.aws:8088 -SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 -SC4S_DEST_SPLUNK_HEC_WORKERS=6 -SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 -#Uncomment the following line if using untrusted SSL certificates -#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no -``` +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* Restart SC4S (below) +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. -## Start/Restart SC4S +## Configure compliance index/metadata overrides + +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. + +# Scale out + +Additional hosts can be deployed for syslog collection from additional network zones and locations. + +# Start/Restart SC4S ```bash docker stack deploy --compose-file docker-compose.yml sc4s ``` - # Stop SC4S Start by obtaining the stack name (ID): @@ -272,7 +275,7 @@ docker logs SC4S ``` You should see events similar to those below in the output: ```ini -Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.24.1' +Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.25.1' Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' ``` diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 7eae5b0..03f3d8f 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -21,7 +21,7 @@ reboot: net.ipv4.ip_forward=1 ``` -# Setup +# Initial Setup * Create the systemd unit file `/lib/systemd/system/sc4s.service` based on the following template: @@ -38,10 +38,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest" Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" +# Optional mount point for local disk archive (EWMM output) files + +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" -# Uncomment the following line if local disk archiving is desired -# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Uncomment the following line if custom TLS certs are provided # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z" @@ -52,16 +55,16 @@ ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE ExecStartPre=/usr/bin/docker run \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight --rm \ - $SC4S_IMAGE -s -ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp \ + --name SC4S_preflight \ + --rm $SC4S_IMAGE -s +ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ "$SC4S_LOCAL_ARCHIVE_MOUNT" \ "$SC4S_TLS_DIR" \ - --name SC4S --rm \ -$SC4S_IMAGE + --name SC4S \ + --rm $SC4S_IMAGE ``` * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. @@ -95,9 +98,10 @@ document for details on the directory structure the archive uses. * IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the unit file above. Failure to do this will cause SC4S to abort at startup. -## Configure the SC4S environment +# Configure the sc4s environment -Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: +SC4S is almost entirely controlled through environment variables, which are read from a file at starteup. Create a file named +``/opt/sc4s/env_file`` and add the following environment variables and values: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088 @@ -109,61 +113,41 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. - -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example. - -## Modify index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. - -* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your -environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in -this table that pertain to the individual data source filters that are included with SC4S. -* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further -information is covered in the "Log Path overrides" section of the Configuration document. - -## Configure source filtering by source IP or host name - -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. - -* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. +If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. -## Configure compliance index/metadata overrides +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example above. -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. -The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to -the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is -covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. +## Configure SC4S Listening Ports -## Configure SC4S for systemd and start SC4S +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The unit file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container port mapping +can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the initial port in the +`ExecStart` line in the unit file for the main container (which represents the actual listening port on the host machine), like so: -```bash -sudo systemctl daemon-reload -sudo systemctl enable sc4s -sudo systemctl start sc4s ``` +-p 614:514 -p 714:514/udp -p 8514:6514 +``` +This instructs the _host_ to listen on TCP port 614, UDP 714, and TCP 8514 (for TLS) and map them to the standard UDP/TCP 514 and 6514 ports +on the _container_. No changes to the underlying SC4S default configuration (environment variables) are needed. -# Configure Dedicated Listening Ports +### Dedicated (Unique) Listening Ports For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. -For collection of such sources we provide a means of dedicating a unique listening port to a specific source. +For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. -Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. +The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the +environment variable(s). In the example below, the `ExecStart` line for the main SC4S container is modified, where +``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. -In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify individual ports or a -range as appropriate for your network. +Follow these steps to configure unique ports: -* Modify the unit file ``/lib/systemd/system/sc4s.service`` +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" +documentation to identify the specific environment variables that are mapped to each data source vendor/technology. +* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below. +* Ensure that you reload the unit file as well as restarting SC4S. See the "Configure SC4S for systemd and start SC4S" section below. ```ini [Unit] Description=SC4S Container @@ -177,8 +161,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest" Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" +# Optional mount point for local disk archive (EWMM output) files + +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" + # Uncomment the following line if custom TLS certs are provided # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" @@ -188,38 +177,54 @@ ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE ExecStartPre=/usr/bin/docker run \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight --rm \ - $SC4S_IMAGE -s -ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ + --name SC4S_preflight \ + --rm $SC4S_IMAGE -s +ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ + "$SC4S_LOCAL_ARCHIVE_MOUNT" \ + "$SC4S_TLS_DIR" \ --name SC4S \ - --rm \ -$SC4S_IMAGE + --rm $SC4S_IMAGE ``` -* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). +## Modify index destinations for Splunk -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +## Configure source filtering by source IP or host name -```dotenv -SPLUNK_HEC_URL=https://splunk.smg.aws:8088 -SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 -SC4S_DEST_SPLUNK_HEC_WORKERS=6 -SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 -#Uncomment the following line if using untrusted SSL certificates -#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no -``` +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. + +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. -* Restart SC4S (below) +## Configure compliance index/metadata overrides + +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. + +## Configure SC4S for systemd and start SC4S + +```bash +sudo systemctl daemon-reload +sudo systemctl enable sc4s +sudo systemctl start sc4s +``` # Start SC4S diff --git a/docs/gettingstarted/index.md b/docs/gettingstarted/index.md index f686dc0..b8258f6 100644 --- a/docs/gettingstarted/index.md +++ b/docs/gettingstarted/index.md @@ -76,11 +76,11 @@ Splunk type. | Container and Orchestration | Notes | |-----------------------------|-------| -| [Podman + systemd single node](gettingstarted/podman-systemd-general.md) | First choice for RedHat 7.x/8.x and CentOS, second choice for Debian and Ubuntu (packages provided via PPA) | -| [Docker CE + systemd single node](gettingstarted/docker-systemd-general.md) | First choice for Debian and Ubuntu; second choice for CentOS for those with limited existing Docker experience | -| [Docker CE + Swarm single node](gettingstarted/docker-swarm-general.md) | Option for Debian, Ubuntu, CentOS, and Desktop Docker desiring Docker Compose or Swarm orchestration | -| [Docker CE + Swarm single node RHEL 7.7](gettingstarted/docker-swarm-rhel7.md) | Option for RedHat 7.7 desiring Docker Compose or Swarm orchestration | -| [Bring your own Envionment](gettingstarted/byoe-rhel7.md) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers | +| [Podman + systemd single node](gettingstarted/podman-systemd-general) | First choice for RedHat 7.x/8.x and CentOS, second choice for Debian and Ubuntu (packages provided via PPA) | +| [Docker CE + systemd single node](gettingstarted/docker-systemd-general) | First choice for Debian and Ubuntu; second choice for CentOS for those with limited existing Docker experience | +| [Docker CE + Swarm single node](gettingstarted/docker-swarm-general) | Option for Debian, Ubuntu, CentOS, and Desktop Docker desiring Docker Compose or Swarm orchestration | +| [Docker CE + Swarm single node RHEL 7.7](gettingstarted/docker-swarm-rhel7) | Option for RedHat 7.7 desiring Docker Compose or Swarm orchestration | +| [Bring your own Envionment](gettingstarted/byoe-rhel7) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers | ### Offline Container Installation diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 18f8f82..51356a8 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -3,7 +3,7 @@ Refer to [Installation](https://podman.io/getting-started/installation) -# Setup +# Initial Setup * Create the systemd unit file `/lib/systemd/system/sc4s.service` based on the following template: @@ -20,10 +20,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest" Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" +# Optional mount point for local disk archive (EWMM output) files + +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" -# Uncomment the following line if local disk archiving is desired -# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Uncomment the following line if custom TLS certs are provided # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z" @@ -34,16 +37,16 @@ ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE ExecStartPre=/usr/bin/podman run \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight --rm \ - $SC4S_IMAGE -s -ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp \ + --name SC4S_preflight \ + --rm $SC4S_IMAGE -s +ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ "$SC4S_LOCAL_ARCHIVE_MOUNT" \ "$SC4S_TLS_DIR" \ - --name SC4S --rm \ -$SC4S_IMAGE + --name SC4S \ + --rm $SC4S_IMAGE ``` * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. @@ -77,9 +80,10 @@ document for details on the directory structure the archive uses. * IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the unit file above. Failure to do this will cause SC4S to abort at startup. -## Configure the sc4s environment +# Configure the sc4s environment -Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: +SC4S is almost entirely controlled through environment variables, which are read from a file at starteup. Create a file named +``/opt/sc4s/env_file`` and add the following environment variables and values: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088 @@ -91,61 +95,41 @@ SC4S_DEST_SPLUNK_HEC_WORKERS=6 * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints, up to a maxiumum of 32. +If the endpoint is a VIP, match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example. - -## Modify index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. - -* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your -environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in -this table that pertain to the individual data source filters that are included with SC4S. -* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further -information is covered in the "Log Path overrides" section of the Configuration document. +uncomment the last line in the example above. -## Configure source filtering by source IP or host name +## Configure SC4S Listening Ports -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. +The unit file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container port mapping +can be used to change the defaults without altering the underlying SC4S configuration. To do this, simply change the initial port in the +`ExecStart` line in the unit file for the main container (which represents the actual listening port on the host machine), like so: -* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. -* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. - -## Configure compliance index/metadata overrides - -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. -The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to -the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is -covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. - -## Configure SC4S for systemd and start SC4S - -```bash -sudo systemctl daemon-reload -sudo systemctl enable sc4s -sudo systemctl start sc4s ``` +-p 614:514 -p 714:514/udp -p 8514:6514 +``` +This instructs the _host_ to listen on TCP port 614, UDP 714, and TCP 8514 (for TLS) and map them to the standard UDP/TCP 514 and 6514 ports +on the _container_. No changes to the underlying SC4S default configuration (environment variables) are needed. -# Configure Dedicated Listening Ports +### Dedicated (Unique) Listening Ports For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. -For collection of such sources we provide a means of dedicating a unique listening port to a specific source. +For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. -Refer to the "Sources" documentation to identify the specific variable used to enable a specific port for the technology in use. +The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the +environment variable(s). In the example below, the `ExecStart` line for the main SC4S container is modified, where +``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. -In the following example ``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. Modify individual ports or a -range as appropriate for your network. +Follow these steps to configure unique ports: -* Modify the unit file ``/lib/systemd/system/sc4s.service`` +* Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" +documentation to identify the specific environment variables that are mapped to each data source vendor/technology. +* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below. +* Ensure that you reload the unit file as well as restarting SC4S. See the "Configure SC4S for systemd and start SC4S" section below. ```ini [Unit] Description=SC4S Container @@ -159,8 +143,13 @@ Environment="SC4S_IMAGE=splunk/scs:latest" Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" +# Optional mount point for local disk archive (EWMM output) files + +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" + # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" + # Uncomment the following line if custom TLS certs are provided # Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" @@ -170,38 +159,54 @@ ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE ExecStartPre=/usr/bin/podman run \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight --rm \ - $SC4S_IMAGE -s -ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ + --name SC4S_preflight \ + --rm $SC4S_IMAGE -s +ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ + "$SC4S_LOCAL_ARCHIVE_MOUNT" \ + "$SC4S_TLS_DIR" \ --name SC4S \ - --rm \ -$SC4S_IMAGE + --rm $SC4S_IMAGE ``` -* Modify the following file ``/opt/sc4s/env_file`` to include the port-specific environment variable(s). See the "Sources" -section for more information on your specific device(s). +## Modify index destinations for Splunk -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +## Configure source filtering by source IP or host name -```dotenv -SPLUNK_HEC_URL=https://splunk.smg.aws:8088 -SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 -SC4S_DEST_SPLUNK_HEC_WORKERS=6 -SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 -#Uncomment the following line if using untrusted SSL certificates -#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no -``` +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. + +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. -* Restart SC4S (below) +## Configure compliance index/metadata overrides + +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. + +## Configure SC4S for systemd and start SC4S + +```bash +sudo systemctl daemon-reload +sudo systemctl enable sc4s +sudo systemctl start sc4s +``` # Start SC4S