diff --git a/docs/configuration.md b/docs/configuration.md index f314f89..949bf47 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,4 +1,9 @@ -# Global Configuration +# SC4S Configuration Variables + +Other than device filter creation, SC4S is almost entirely controlled by environment variables. Here are the categories +and variables needed to properly configure SC4S for your environment. + +## Global Configuration | Variable | Values | Description | |----------|---------------|-------------| @@ -6,16 +11,17 @@ | SPLUNK_HEC_TOKEN | string | Splunk HTTP Event Collector Token | -# Splunk HEC destination Configuration +## Splunk HEC Destination Configuration | Variable | Values | Description | |----------|---------------|-------------| +| SC4S_DEST_SPLUNK_HEC_WORKERS | numeric | Number of destination workers (threads). Set this to the number of HEC endpoints up to a max of 32. | | SC4S_DEST_SPLUNK_HEC_TLS_VERIFY | yes(default) or no | verify HTTP(s) certificate | | SC4S_DEST_SPLUNK_HEC_CIPHER_SUITE | comma separated list | Open SSL cipher suite list | | SC4S_DEST_SPLUNK_HEC_SSL_VERSION | comma separated list | Open SSL version list | | SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE | path | Custom trusted cert file | -# Syslog Source Configuration +## Syslog Source Configuration | Variable | Values/Default | Description | |----------|----------------|-------------| @@ -28,7 +34,7 @@ | SC4S_SOURCE_UDP_SO_RCVBUFF | 425984 | UDP server buffer size in bytes | -# Syslog Source TLS Certificate Configuration +## Syslog Source TLS Certificate Configuration * Create a folder ``/opt/sc4s/tls`` * Save the server private key in PEM format with NO PASSWORD to ``/opt/sc4s/tls/server.key`` diff --git a/docs/gettingstarted.md b/docs/gettingstarted.md index a5ea9ab..7a29b37 100644 --- a/docs/gettingstarted.md +++ b/docs/gettingstarted.md @@ -13,7 +13,7 @@ transmission of events between computer systems over UDP, TCP, or TLS. The proto overhead on the sender favoring performance over reliability. This fundamental choice means any instability or resource constraint will cause data to be lost in transmission. -* When practical and cost effective considering the importance of completeness as a requirement, place the sc4s +* When practical and cost effective (considering the importance of completeness as a requirement), place the sc4s instance in the same VLAN as the source device. * Avoid crossing a Wireless network, WAN, Firewall, Load Balancer, or inline IDS. @@ -29,9 +29,10 @@ environment. ### Create Indexes -SC4S is pre-configured to map each sourcetype to a typical index, for new installations best practice is to create the following -indexes in Splunk. The indexes can be customized easily if desired. If using defaults create the following indexes on Splunk: +SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create them in Splunk when +using the SC4S defaults. SC4S can be easily customized to use different indexes if desired. +* email * netauth * netfw * netids @@ -54,7 +55,7 @@ Install the following: session. Alternatively, a list of HEC endpoint URLs can be configured in SC4S if no load balancer is in place. In either case, it is recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of HWFs. - Create a HEC token that will be used by SC4S and ensure the token has access to place events in main, em_metrics, and all indexes used as -event destinations +event destinations. - Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud) or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your Splunk type. @@ -78,6 +79,6 @@ Splunk type. # Scale out -Additional hosts can be deployed for syslog collection from additional network zones and locations +Additional hosts can be deployed for syslog collection from additional network zones and locations: ![SC4S deployment diagram](SC4S%20deployment.png) \ No newline at end of file diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index 3928114..9fef61c 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -5,7 +5,7 @@ Refer to [Getting Started](https://docs.docker.com/get-started/) # SC4S Configuration -* Create a directory on the server for configuration. This should be available to all administrators, for example: +* Create a directory on the server for local configurations. This should be available to all administrators, for example: ``/opt/sc4s/`` * Create a docker-compose.yml file in the directory created above, based on the following template: @@ -33,21 +33,33 @@ services: - /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf #Uncomment the following line if custom TLS certs are provided - /opt/sc4s/tls:/opt/syslog-ng/tls - ``` +* NOTE: While strictly optional, it is recommended that you create and/or download all files and directories referenced in the yml template +above (`volumes` declarations) according to the configuration steps that follow. The TLS options are described in the "Configuration" section. +Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors. + ## Configure the SC4S environment -Create the following file ``/opt/sc4s/env_file`` +Create the following file ``/opt/sc4s/env_file`` and add the environment variables below: * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example below. + ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 +SC4S_DEST_SPLUNK_HEC_WORKERS=6 SPLUNK_CONNECT_METHOD=hec SPLUNK_DEFAULT_INDEX=main SPLUNK_METRICS_INDEX=em_metrics +#Uncomment the following line if using untrusted SSL certificates +#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` ## Configure index destinations for Splunk @@ -77,13 +89,12 @@ sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/mas ``` * Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax. -# Start SC4S +## Start/Restart SC4S ```bash docker stack deploy --compose-file docker-compose.yml sc4s ``` - # Scale out Additional hosts can be deployed for syslog collection from additional network zones and locations. @@ -135,16 +146,23 @@ services: - /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf #Uncomment the following line if custom TLS certs are provided - /opt/sc4s/tls:/opt/syslog-ng/tls - ``` -* Modify the following file ``/opt/sc4s/default/env_file`` after including the port-specific environment variable(s). +* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" +section for more information on your specific device(s). * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example below. + ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 +SC4S_DEST_SPLUNK_HEC_WORKERS=6 SPLUNK_CONNECT_METHOD=hec SPLUNK_DEFAULT_INDEX=main SPLUNK_METRICS_INDEX=em_metrics @@ -153,8 +171,57 @@ SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` -* Start SC4S. +* Restart SC4S (below) + +## Start/Restart SC4S ```bash docker stack deploy --compose-file docker-compose.yml sc4s ``` + +# Stop SC4S + +Start by obtaining the stack name (ID): +```bash +docker stack ls +``` +Then, remove the stack: +```bash +docker stack rm +``` +# Verify Proper Operation + +SC4S has a number of "preflight" checks to ensure that the container starts properly and that the syntax of the underlying syslog-ng +configuration is correct. After this step completes, to verify SC4S is properly communicating with Splunk, +execute the following search in Splunk: + +```ini +index=* sourcetype=sc4s:events "starting up" +``` +This should yield the following event: +```ini +syslog-ng starting up; version='3.22.1' +``` +when the startup process proceeds normally (without syntax errors). If you do not see this, +follow the steps below before proceeding to deeper-level troubleshooting: + +* Check to see that the URL, token, and TLS/SSL settings are correct, and that the appropriate firewall ports are open (8088 or 443). + +* Check to see that the proper indexes are created in Splunk, and that the token has access to them. + +* Ensure the proper operation of the load balancer if used. + +* Lastly, execute the following command to check the internal logs of the syslog-ng process running in the container. Depending on the +traffic load, there may be quite a bit of output in the syslog-ng logs. +```bash +docker logs SC4S +``` +You should see events similar to those below in the output: +```ini +Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.22.1' +Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' +Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' +``` +If you see http server errors such as 4xx or 5xx responses from the http (HEC) endpoint, one or more of the items above are likely set +incorrectly. If validating/fixing the configuration fails to correct the problem, proceed to the "Troubleshooting" section for more +information. \ No newline at end of file diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index a9dbcc8..eba329c 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -33,7 +33,7 @@ sudo docker swarm init # SC4S Configuration -* Create a directory on the server for configuration. This should be available to all administrators, for example: +* Create a directory on the server for local configurations. This should be available to all administrators, for example: ``/opt/sc4s/`` * Create a docker-compose.yml file in the directory created above, based on the following template: @@ -63,21 +63,33 @@ services: - /opt/sc4s/tls:/opt/syslog-ng/tls ``` +* NOTE: While strictly optional, it is recommended that you create and/or download all files and directories referenced in the yml template +above (`volumes` declarations) according to the configuration steps that follow. The TLS options are described in the "Configuration" section. +Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors. + ## Configure the SC4S environment -Create the following file ``/opt/sc4s/env_file`` +Create the following file ``/opt/sc4s/env_file`` and add the environment variables below: * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example below. + ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 +SC4S_DEST_SPLUNK_HEC_WORKERS=6 SPLUNK_CONNECT_METHOD=hec SPLUNK_DEFAULT_INDEX=main SPLUNK_METRICS_INDEX=em_metrics +#Uncomment the following line if using untrusted SSL certificates +#SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` - ## Configure index destinations for Splunk Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow @@ -105,7 +117,7 @@ sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/mas ``` * Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax. -# Start SC4S +## Start/Restart SC4S ```bash sudo docker stack deploy --compose-file docker-compose.yml sc4s @@ -164,13 +176,21 @@ services: - /opt/sc4s/tls:/opt/syslog-ng/tls ``` -* Modify the following file ``/opt/sc4s/default/env_file`` +* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" +section for more information on your specific device(s). * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example below. + ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 +SC4S_DEST_SPLUNK_HEC_WORKERS=6 SPLUNK_CONNECT_METHOD=hec SPLUNK_DEFAULT_INDEX=main SPLUNK_METRICS_INDEX=em_metrics @@ -178,8 +198,58 @@ SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 #Uncomment the following line if using untrusted SSL certificates #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` -* Start SC4S. + +* Restart SC4S (below) + +## Start/Restart SC4S ```bash docker stack deploy --compose-file docker-compose.yml sc4s ``` + +# Stop SC4S + +Start by obtaining the stack name (ID): +```bash +docker stack ls +``` +Then, remove the stack: +```bash +docker stack rm +``` +# Verify Proper Operation + +SC4S has a number of "preflight" checks to ensure that the container starts properly and that the syntax of the underlying syslog-ng +configuration is correct. After this step completes, to verify SC4S is properly communicating with Splunk, +execute the following search in Splunk: + +```ini +index=* sourcetype=sc4s:events "starting up" +``` +This should yield the following event: +```ini +syslog-ng starting up; version='3.22.1' +``` +when the startup process proceeds normally (without syntax errors). If you do not see this, +follow the steps below before proceeding to deeper-level troubleshooting: + +* Check to see that the URL, token, and TLS/SSL settings are correct, and that the appropriate firewall ports are open (8088 or 443). + +* Check to see that the proper indexes are created in Splunk, and that the token has access to them. + +* Ensure the proper operation of the load balancer if used. + +* Lastly, execute the following command to check the internal logs of the syslog-ng process running in the container. Depending on the +traffic load, there may be quite a bit of output in the syslog-ng logs. +```bash +docker logs SC4S +``` +You should see events similar to those below in the output: +```ini +Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.22.1' +Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' +Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' +``` +If you see http server errors such as 4xx or 5xx responses from the http (HEC) endpoint, one or more of the items above are likely set +incorrectly. If validating/fixing the configuration fails to correct the problem, proceed to the "Troubleshooting" section for more +information. \ No newline at end of file diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 9a683dd..80721ec 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -5,11 +5,7 @@ Refer to [Getting Started](https://docs.docker.com/get-started/) # Setup -* Create a systemd unit file use to start the container with the host os. - -``sudo vi /lib/systemd/system/sc4s.service`` - -Enter insert mode and paste the following +* Create the systemd unit file `/lib/systemd/system/sc4s.service` based on the following template: ```ini [Unit] @@ -20,10 +16,13 @@ Requires=network.service [Service] Environment="SC4S_IMAGE=splunk/sc4s:latest" -#Note Uncomment this line to use custom index names AND download the splunk_index.csv file template per getting started +#Note Uncomment the following line to use custom index names AND download the splunk_index.csv file template per getting started Environment="SC4S_UNIT_SPLUNK_INDEX=-v /opt/sc4s/default/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv" + +#Note Uncomment the following two lines for host and ip based source type mapping AND download the two file templates per getting started Environment="SC4S_UNIT_VP_CSV=-v /opt/sc4s/default/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv" Environment="SC4S_UNIT_VP_CONF=-v /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf" + #Uncomment the following line if custom TLS certs are provided #Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" @@ -39,20 +38,29 @@ ExecStartPre=/usr/bin/docker run \ ExecStart=/usr/bin/docker run -p 514:514 \ --env-file=/opt/sc4s/default/env_file \ "$SC4S_UNIT_SPLUNK_INDEX" "$SC4S_UNIT_VP_CSV" "$SC4S_UNIT_VP_CONF" "$SC4S_TLS_DIR" \ - --name SC4S \ - --rm \ + --name SC4S --rm \ $SC4S_IMAGE ``` +* NOTE: While strictly optional, it is recommended that you create and/or download all files and directories referenced in the unit +file above (the `Environment` assignments) according to the configuration steps that follow. The TLS options are described in the "Configuration" section. + ## Configure the SC4S environment -Create the following file ``/opt/sc4s/default/env_file`` +Create the following file ``/opt/sc4s/default/env_file`` and add the environment variables below: * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example below. + ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 +SC4S_DEST_SPLUNK_HEC_WORKERS=6 SPLUNK_CONNECT_METHOD=hec SPLUNK_DEFAULT_INDEX=main SPLUNK_METRICS_INDEX=em_metrics @@ -88,7 +96,7 @@ sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/mas ``` * Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax. -# Start SC4S +## Configure SC4S for systemd and start SC4S ```bash sudo systemctl daemon-reload @@ -96,7 +104,6 @@ sudo systemctl enable sc4s sudo systemctl start sc4s ``` - # Configure Dedicated Listening Ports For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in @@ -142,13 +149,21 @@ ExecStart=/usr/bin/docker run -p 514:514 -p 5000-5020:5000-5020 \ $SC4S_IMAGE ``` -* Modify the following file ``/opt/sc4s/default/env_file`` +* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" +section for more information on your specific device(s). * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example below. + ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 +SC4S_DEST_SPLUNK_HEC_WORKERS=6 SPLUNK_CONNECT_METHOD=hec SPLUNK_DEFAULT_INDEX=main SPLUNK_METRICS_INDEX=em_metrics @@ -157,8 +172,67 @@ SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` -* Restart SC4S. +* Restart SC4S (below) + +# Start SC4S + +```bash +sudo systemctl start sc4s +``` + +# Restart SC4S ```bash sudo systemctl restart sc4s ``` + +If changes were made to the configuration Unit file above (e.g. to configure with dedicated ports), you must first stop SC4S and re-run +the systemd configuration commands: +```bash +sudo systemctl stop sc4s +sudo systemctl daemon-reload +sudo systemctl enable sc4s +sudo systemctl start sc4s +``` + +# Stop SC4S + +```bash +sudo systemctl stop sc4s +``` +# Verify Proper Operation + +SC4S has a number of "preflight" checks to ensure that the container starts properly and that the syntax of the underlying syslog-ng +configuration is correct. After this step completes, to verify SC4S is properly communicating with Splunk, +execute the following search in Splunk: + +```ini +index=* sourcetype=sc4s:events "starting up" +``` +This should yield the following event: +```ini +syslog-ng starting up; version='3.22.1' +``` +when the startup process proceeds normally (without syntax errors). If you do not see this, +follow the steps below before proceeding to deeper-level troubleshooting: + +* Check to see that the URL, token, and TLS/SSL settings are correct, and that the appropriate firewall ports are open (8088 or 443). + +* Check to see that the proper indexes are created in Splunk, and that the token has access to them. + +* Ensure the proper operation of the load balancer if used. + +* Lastly, execute the following command to check the internal logs of the syslog-ng process running in the container. Depending on the +traffic load, there may be quite a bit of output in the syslog-ng logs. +```bash +docker logs SC4S +``` +You should see events similar to those below in the output: +```ini +Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.22.1' +Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' +Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' +``` +If you see http server errors such as 4xx or 5xx responses from the http (HEC) endpoint, one or more of the items above are likely set +incorrectly. If validating/fixing the configuration fails to correct the problem, proceed to the "Troubleshooting" section for more +information. \ No newline at end of file diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 0ddb896..54d8df6 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -5,11 +5,7 @@ Refer to [Installation](https://podman.io/getting-started/installation) # Setup -* Create a systemd unit file use to start the container with the host os. - -``sudo vi /lib/systemd/system/sc4s.service`` - -Enter insert mode and paste the following +* Create the systemd unit file `/lib/systemd/system/sc4s.service` based on the following template: ```ini [Unit] @@ -20,16 +16,19 @@ Requires=network.service [Service] Environment="SC4S_IMAGE=splunk/scs:latest" -#Note Uncomment this line to use custom index names AND download the splunk_index.csv file template per getting started +#Note Uncomment the following line to use custom index names AND download the splunk_index.csv file template per getting started Environment="SC4S_UNIT_SPLUNK_INDEX=-v /opt/sc4s/default/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv" -#Note Uncomment the following two linese for host and ip based source type mapping AND download the two file templates per getting started + +#Note Uncomment the following two lines for host and ip based source type mapping AND download the two file templates per getting started #Environment="SC4S_UNIT_VP_CSV=-v /opt/sc4s/default/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv" #Environment="SC4S_UNIT_VP_CONF=-v /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf" + #Uncomment the following line if custom TLS certs are provided #Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" TimeoutStartSec=0 Restart=always + ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE ExecStartPre=/usr/bin/podman run \ --env-file=/opt/sc4s/default/env_file \ @@ -41,18 +40,27 @@ ExecStart=/usr/bin/podman run -p 514:514 \ "$SC4S_UNIT_SPLUNK_INDEX" "$SC4S_UNIT_VP_CSV" "$SC4S_UNIT_VP_CONF" "$SC4S_TLS_DIR" \ --name SC4S --rm \ $SC4S_IMAGE - ``` +* NOTE: While strictly optional, it is recommended that you create and/or download all files and directories referenced in the unit +file above (the `Environment` assignments) according to the configuration steps that follow. The TLS options are described in the "Configuration" section. + ## Configure the sc4s environment -Create the following file ``/opt/sc4s/default/env_file`` +Create the following file ``/opt/sc4s/default/env_file`` and add the environment variables below: * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example below. + ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 +SC4S_DEST_SPLUNK_HEC_WORKERS=6 SPLUNK_CONNECT_METHOD=hec SPLUNK_DEFAULT_INDEX=main SPLUNK_METRICS_INDEX=em_metrics @@ -88,7 +96,7 @@ sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/mas ``` * Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax. -# Start SC4S +## Configure SC4S for systemd and start SC4S ```bash sudo systemctl daemon-reload @@ -139,16 +147,23 @@ ExecStart=/usr/bin/podman run -p 514:514 -p 5000-5020:5000-5020 \ --name SC4S \ --rm \ $SC4S_IMAGE - ``` -* Modify the following file ``/opt/sc4s/default/env_file`` +* Modify the following file ``/opt/sc4s/default/env_file`` to include the port-specific environment variable(s). See the "Sources" +section for more information on your specific device(s). * Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example below. + ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event SPLUNK_HEC_TOKEN=a778f63a-5dff-4e3c-a72c-a03183659e94 +SC4S_DEST_SPLUNK_HEC_WORKERS=6 SPLUNK_CONNECT_METHOD=hec SPLUNK_DEFAULT_INDEX=main SPLUNK_METRICS_INDEX=em_metrics @@ -157,8 +172,67 @@ SC4S_LISTEN_JUNIPER_NETSCREEN_TCP_PORT=5000 #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` -* Restart SC4S. +* Restart SC4S (below) + +# Start SC4S + +```bash +sudo systemctl start sc4s +``` + +# Restart SC4S ```bash sudo systemctl restart sc4s ``` + +If changes were made to the configuration Unit file above (e.g. to configure with dedicated ports), you must first stop SC4S and re-run +the systemd configuration commands: +```bash +sudo systemctl stop sc4s +sudo systemctl daemon-reload +sudo systemctl enable sc4s +sudo systemctl start sc4s +``` + +# Stop SC4S + +```bash +sudo systemctl stop sc4s +``` +# Verify Proper Operation + +SC4S has a number of "preflight" checks to ensure that the container starts properly and that the syntax of the underlying syslog-ng +configuration is correct. After this step completes, to verify SC4S is properly communicating with Splunk, +execute the following search in Splunk: + +```ini +index=* sourcetype=sc4s:events "starting up" +``` +This should yield the following event: +```ini +syslog-ng starting up; version='3.22.1' +``` +when the startup process proceeds normally (without syntax errors). If you do not see this, +follow the steps below before proceeding to deeper-level troubleshooting: + +* Check to see that the URL, token, and TLS/SSL settings are correct, and that the appropriate firewall ports are open (8088 or 443). + +* Check to see that the proper indexes are created in Splunk, and that the token has access to them. + +* Ensure the proper operation of the load balancer if used. + +* Lastly, execute the following command to check the internal logs of the syslog-ng process running in the container. Depending on the +traffic load, there may be quite a bit of output in the syslog-ng logs. +```bash +podman logs SC4S +``` +You should see events similar to those below in the output: +```ini +Oct 1 03:13:35 77cd4776af41 syslog-ng[1]: syslog-ng starting up; version='3.22.1' +Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection accepted; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' +Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', client='AF_INET(10.0.1.18:55010)', local='AF_INET(0.0.0.0:514)' +``` +If you see http server errors such as 4xx or 5xx responses from the http (HEC) endpoint, one or more of the items above are likely set +incorrectly. If validating/fixing the configuration fails to correct the problem, proceed to the "Troubleshooting" section for more +information. \ No newline at end of file