diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index d32bc91..e14b1ae 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,23 +1,25 @@ #Troubleshooting -## Not getting starting up message in Splunk! +## Startup + +Most issues that occur with startup and operation of sc4s typically involve syntax errors or duplicate listening ports. If you are +running out of systemd, you may see this at startup: -### Are there any syntax errors or duplicate listening ports? -Check for errors with the following command. ```bash [root@sc4s syslog-ng]# systemctl start sc4s Job for sc4s.service failed because the control process exited with error code. See "systemctl status sc4s.service" and "journalctl -xe" for details. ``` -### Is your container running? - Check using `podman logs SC4S` or `podman ps`. A more informative command than `journalctl -xe` is the following, +In other cases, there may be nothing untoward after starting with systemd, but the container is not running at all +after checking with `podman logs SC4S` or `podman ps`. A more informative command than `journalctl -xe` is the following, ``` journalctl -b -u sc4s | tail -100 ``` which will print the last 100 lines of the system journal in far more detail, which should be sufficient to see the specific failure (syntax or runtime) and guide you in troubleshooting why the container exited unexpectedly. -### Is your container working outside of the systemd startup environment? -The following command will launch the container directly from the CLI. This command assumes the local mounted directories are set up as shown in the "getting started" examples: +As an alternative to launching via systemd during the initial installation phase, you may wish to test the container startup outside of the +systemd startup environment. The following commmand will launch the container directly from the CLI. This command assumes the local mounted +directories are set up as shown in the "getting started" examples: ```bash /usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ @@ -31,7 +33,7 @@ The following command will launch the container directly from the CLI. This com If you are using docker, substitute "docker" for "podman" for the container runtime command above. -### Are there any stale containers in your environment? (podman) +### Stale Containers (podman) In rare instances, (especially when starting/stopping often) an SC4S container might not shut down completely when using podman, leaving a "stale" container behind that is denoted by a very long ID string. You will see this type of output when viewing the journal after a failed @@ -49,7 +51,16 @@ podman rm -f 894357502b2a7142d097ea3ca1468d1cb4fbc69959a9817a1bbe145a09d37fb9 replacing the long string with whatever container ID is shown in your error message. SC4S should then start normally. -### Are there any HEC errors in the podman logs? +## Verification of TLS Server + +To verify the correct configuration of the TLS server use the following command. Use `podman` or `docker` and replace the IP, FQDN, +and port as appropriate: + +```bash + run -ti drwetter/testssl.sh --severity MEDIUM --ip 127.0.0.1 selfsigned.example.com:6510 +``` + +## Validating HEC/token issues (AKA "No data in Splunk") SC4S performs basic HEC connectivity and index checks at startup. These indicate general connection issues and indexes that may not be accesible and/or configured on the Splunk side. To check the container logs which contain the results of these tests, run: @@ -82,57 +93,6 @@ A `400` error (not 404) is normally caused by an index that has not been created just _one_ bad index will "taint" the entire batch (in this case, 1000 events) and prevent _any_ of them from being sent to Splunk. _It is imperative that the container logs be free of these kinds of errors in production._ -### Is SC4S server out of space? -Check the connection to Splunk. If the connection is lost for a long period it will lead to increase in disk space due to disk buffer backup. -Adjust the size of the disk buffer using env_file. [Disk buffer configuration](https://splunk-connect-for-syslog.readthedocs.io/en/master/configuration/#disk-buffer-variables) - -Check env_file if `SC4S_DEST_GLOBAL_ALTERNATES=d_hec_debug` is enabled and hence archive is consuming disk space. - -Check the method consuming disk space use `df -h` and then `du -sh *` to find out what's causing it. - -Try rebuilding sc4s volume. -``` -podman volume rm splunk-sc4s-var -podman volume create splunk-sc4s-var -``` -Try pruning containers -``` -podman system prune -``` - -### Are there any kernel memory warnings? - -```bash -/usr/bin/ logs SC4S -``` -Note the output. The following warning message is not a failure condition unless we are reaching the upper limit of hardware performance. -``` -The kernel refused to set the receive buffer (SO_RCVBUF) to the requested size, you probably need to adjust buffer related kernel parameters; so_rcvbuf='1703936', so_rcvbuf_set='425984' -``` -Make changes to /etc/sysctl.conf. Changing receive buffer values here to 16 MB: - -``` -net.core.rmem_default = 1703936 -net.core.rmem_max = 1703936. -``` -Run following commands for changes to be affected. -``` -sysctl -p restart SC4S -``` - -## Are there events with incorrect timezone? -By default, SC4S resolves the timezone to GMT. If customer have a preference to use local TZ then set the user TZ preference in Splunk during search time rather than at index time. -[Timezone config documentation](https://docs.splunk.com/Documentation/Splunk/8.0.4/Data/ApplyTimezoneOffsetstotimestamps) - -## Verification of TLS Server - -To verify the correct configuration of the TLS server use the following command. Use `podman` or `docker` and replace the IP, FQDN, -and port as appropriate: - -```bash - run -ti drwetter/testssl.sh --severity MEDIUM --ip 127.0.0.1 selfsigned.example.com:6510 -``` - ## Enabling the Alternate Debug Destination To help debug why the `400` errors are ocurring, it is helpful to enable an alternate destination for syslog traffic that will write