diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index 9e1a85f..0d7dc62 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -115,7 +115,7 @@ If the endpoint is a VIP, match this value to the total number of indexers behin * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example above. -## Configure SC4S Listening Ports +## Configure SC4S Default Listening Ports Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. The docker compose file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container @@ -139,35 +139,16 @@ For certain source technologies, categorization by message content is impossible the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. -The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured -by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the -``target`` and ``published`` lines provide for 21 additional technology-specific UDP and TCP ports. - Follow these steps to configure unique ports: * Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" documentation to identify the specific environment variables that are mapped to each data source vendor/technology. -* Modify the compose file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below. -* Restart SC4S using the command in the "Start/Restart SC4S" section below. -```yaml -version: "3.7" -services: - sc4s: - image: splunk/scs:latest - ports: - - target: 514 - published: 514 - protocol: tcp -#Comment the following line out if using docker-compose - mode: host - - target: 514 - published: 514 - protocol: udp -#Comment the following line out if using docker-compose - mode: host - - target: 6514 - published: 6514 - protocol: tcp +* The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured +by the environment variable(s) added above. Similar to the way the SC4S default listening ports can be changed, the docker compose file +can be ammended with additional ``target`` stanzas in the ``ports`` section of the file. The following additional ``target`` and +``published`` lines provide for 21 additional technology-specific UDP and TCP ports: + +``` # Comment the following line out if using docker-compose mode: host - target: 5000-5020 @@ -178,19 +159,10 @@ services: - target: 5000-5020 published: 5000-5020 protocol: udp -#Comment the following line out if using docker-compose - mode: host - env_file: - - /opt/sc4s/env_file - volumes: - - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - - splunk-sc4s-var:/opt/syslog-ng/var -# Uncomment the following line if local disk archiving is desired -# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z -# Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` +* Restart SC4S using the command in the "Start/Restart SC4S" section below. + ## Modify index destinations for Splunk Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. @@ -209,8 +181,10 @@ apply to support such sources. To identify sources that require this step, refer * If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by +hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local log path is created that is specific to the +environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. ## Configure compliance index/metadata overrides diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index c276f64..af6d519 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -123,7 +123,7 @@ If the endpoint is a VIP, match this value to the total number of indexers behin * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example above. -## Configure SC4S Listening Ports +## Configure SC4S Default Listening Ports Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. The docker compose file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container @@ -147,35 +147,16 @@ For certain source technologies, categorization by message content is impossible the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. -The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured -by the environment variable(s). In the following example, additional ``target`` stanzas are added for the main ``sc4s`` container, where the -``target`` and ``published`` lines provide for 21 additional technology-specific UDP and TCP ports. - Follow these steps to configure unique ports: * Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" documentation to identify the specific environment variables that are mapped to each data source vendor/technology. -* Modify the compose file ``/opt/sc4s/docker-compose.yml`` and add/change port stanzas as appropriate using the example below. -* Restart SC4S using the command in the "Start/Restart SC4S" section below. -```yaml -version: "3.7" -services: - sc4s: - image: splunk/scs:latest - ports: - - target: 514 - published: 514 - protocol: tcp -#Comment the following line out if using docker-compose - mode: host - - target: 514 - published: 514 - protocol: udp -#Comment the following line out if using docker-compose - mode: host - - target: 6514 - published: 6514 - protocol: tcp +* The docker compose file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured +by the environment variable(s) added above. Similar to the way the SC4S default listening ports can be changed, the docker compose file +can be ammended with additional ``target`` stanzas in the ``ports`` section of the file. The following additional ``target`` and +``published`` lines provide for 21 additional technology-specific UDP and TCP ports: + +``` # Comment the following line out if using docker-compose mode: host - target: 5000-5020 @@ -186,19 +167,10 @@ services: - target: 5000-5020 published: 5000-5020 protocol: udp -#Comment the following line out if using docker-compose - mode: host - env_file: - - /opt/sc4s/env_file - volumes: - - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - - splunk-sc4s-var:/opt/syslog-ng/var -# Uncomment the following line if local disk archiving is desired -# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z -# Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` +* Restart SC4S using the command in the "Start/Restart SC4S" section below. + ## Modify index destinations for Splunk Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. @@ -217,8 +189,10 @@ apply to support such sources. To identify sources that require this step, refer * If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by +hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local log path is created that is specific to the +environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. ## Configure compliance index/metadata overrides diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 2f25337..132e80b 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -121,7 +121,7 @@ If the endpoint is a VIP, match this value to the total number of indexers behin * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example above. -## Configure SC4S Listening Ports +## Configure SC4S Default Listening Ports Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. The unit file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container port mapping @@ -129,7 +129,7 @@ can be used to change the defaults without altering the underlying SC4S configur `ExecStart` line in the unit file for the main container (which represents the actual listening port on the host machine), like so: ``` --p 614:514 -p 714:514/udp -p 8514:6514 +ExecStart=/usr/bin/docker run -p 614:514 -p 714:514/udp -p 8514:6514 \ ``` This instructs the _host_ to listen on TCP port 614, UDP 714, and TCP 8514 (for TLS) and map them to the standard UDP/TCP 514 and 6514 ports on the _container_. No changes to the underlying SC4S default configuration (environment variables) are needed. @@ -140,59 +140,21 @@ For certain source technologies, categorization by message content is impossible the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. -The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the -environment variable(s). In the example below, the `ExecStart` line for the main SC4S container is modified, where -``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. - Follow these steps to configure unique ports: * Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" documentation to identify the specific environment variables that are mapped to each data source vendor/technology. -* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below. -* Ensure that you reload the unit file as well as restarting SC4S. See the "Configure SC4S for systemd and start SC4S" section below. -```ini -[Unit] -Description=SC4S Container -Wants=NetworkManager.service network-online.target -After=NetworkManager.service network-online.target - -[Install] -WantedBy=multi-user.target - -[Service] -Environment="SC4S_IMAGE=splunk/scs:latest" - -# Required mount point for syslog-ng persist data (including disk buffer) -Environment="SC4S_PERSIST_VOLUME=-v splunk-sc4s-var:/opt/syslog-ng/var" - -# Optional mount point for local overrides and configurations; see notes in docs -Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" - -# Optional mount point for local disk archive (EWMM output) files -# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" +* The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the +environment variable(s) added above. Similar to the way the SC4S default listening ports can be changed, the `ExecStart` line for +the main SC4S container can also be amended to to include unique listening ports. The following `ExecStart` line in the unit file will +provide for 21 technology-specific UDP and TCP ports: -# Uncomment the following line if custom TLS certs are provided -# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" - -TimeoutStartSec=0 -Restart=always - -ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE -ExecStartPre=/usr/bin/docker run \ - --env-file=/opt/sc4s/env_file \ - "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight \ - --rm $SC4S_IMAGE -s +``` ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ - --env-file=/opt/sc4s/env_file \ - "$SC4S_PERSIST_VOLUME" \ - "$SC4S_LOCAL_CONFIG_MOUNT" \ - "$SC4S_LOCAL_ARCHIVE_MOUNT" \ - "$SC4S_TLS_DIR" \ - --name SC4S \ - --rm $SC4S_IMAGE ``` +* Ensure that you reload the unit file as well as restarting SC4S. See the "Configure SC4S for systemd and start SC4S" section below. + ## Modify index destinations for Splunk Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. @@ -211,8 +173,10 @@ apply to support such sources. To identify sources that require this step, refer * If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by +hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local log path is created that is specific to the +environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. ## Configure compliance index/metadata overrides diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index f38c5b3..8988b50 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -103,7 +103,7 @@ If the endpoint is a VIP, match this value to the total number of indexers behin * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example above. -## Configure SC4S Listening Ports +## Configure SC4S Default Listening Ports Most enterprises use UDP/TCP port 514 as the default as their main listening port for syslog "soup" traffic, and TCP port 6514 for TLS. The unit file and standard SC4S configurations reflect these defaults. If it desired to change some or all of them, container port mapping @@ -111,7 +111,7 @@ can be used to change the defaults without altering the underlying SC4S configur `ExecStart` line in the unit file for the main container (which represents the actual listening port on the host machine), like so: ``` --p 614:514 -p 714:514/udp -p 8514:6514 +ExecStart=/usr/bin/podman run -p 614:514 -p 714:514/udp -p 8514:6514 \ ``` This instructs the _host_ to listen on TCP port 614, UDP 714, and TCP 8514 (for TLS) and map them to the standard UDP/TCP 514 and 6514 ports on the _container_. No changes to the underlying SC4S default configuration (environment variables) are needed. @@ -122,59 +122,21 @@ For certain source technologies, categorization by message content is impossible the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. -The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the -environment variable(s). In the example below, the `ExecStart` line for the main SC4S container is modified, where -``-p 5000-5020:5000-5020`` allows for up to 21 technology-specific ports. - Follow these steps to configure unique ports: * Modify the ``/opt/sc4s/env_file`` file to include the port-specific environment variable(s). Refer to the "Sources" documentation to identify the specific environment variables that are mapped to each data source vendor/technology. -* Modify the unit file ``/lib/systemd/system/sc4s.service`` with the appropriate ``ExecStart`` command line changes using the example below. -* Ensure that you reload the unit file as well as restarting SC4S. See the "Configure SC4S for systemd and start SC4S" section below. -```ini -[Unit] -Description=SC4S Container -Wants=NetworkManager.service network-online.target -After=NetworkManager.service network-online.target - -[Install] -WantedBy=multi-user.target - -[Service] -Environment="SC4S_IMAGE=splunk/scs:latest" - -# Required mount point for syslog-ng persist data (including disk buffer) -Environment="SC4S_PERSIST_VOLUME=-v splunk-sc4s-var:/opt/syslog-ng/var" - -# Optional mount point for local overrides and configurations; see notes in docs -Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z" - -# Optional mount point for local disk archive (EWMM output) files -# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" +* The unit file used to start the SC4S container needs to be modified as well to reflect the additional listening ports configured by the +environment variable(s) added above. Similar to the way the SC4S default listening ports can be changed, the `ExecStart` line for +the main SC4S container can also be amended to to include unique listening ports. The following `ExecStart` line in the unit file will +provide for 21 technology-specific UDP and TCP ports: -# Uncomment the following line if custom TLS certs are provided -# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" - -TimeoutStartSec=0 -Restart=always - -ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE -ExecStartPre=/usr/bin/podman run \ - --env-file=/opt/sc4s/env_file \ - "$SC4S_LOCAL_CONFIG_MOUNT" \ - --name SC4S_preflight \ - --rm $SC4S_IMAGE -s +``` ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp \ - --env-file=/opt/sc4s/env_file \ - "$SC4S_PERSIST_VOLUME" \ - "$SC4S_LOCAL_CONFIG_MOUNT" \ - "$SC4S_LOCAL_ARCHIVE_MOUNT" \ - "$SC4S_TLS_DIR" \ - --name SC4S \ - --rm $SC4S_IMAGE ``` +* Ensure that you reload the unit file as well as restarting SC4S. See the "Configure SC4S for systemd and start SC4S" section below. + ## Modify index destinations for Splunk Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. @@ -193,8 +155,10 @@ apply to support such sources. To identify sources that require this step, refer * If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. -* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. -* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by +hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local log path is created that is specific to the +environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. ## Configure compliance index/metadata overrides diff --git a/docs/sources/CommonEventFormat/index.md b/docs/sources/CommonEventFormat/index.md index b88b329..5688d47 100644 --- a/docs/sources/CommonEventFormat/index.md +++ b/docs/sources/CommonEventFormat/index.md @@ -4,18 +4,37 @@ Each CEF product should have their own source entry in this documentation set. In a departure from normal configuration, all CEF products should use the "CEF" version of the unique port and -archive envrionmetn variable settings (rather than a unique one per product), as the CEF log path +archive environment variable settings (rather than a unique one per product), as the CEF log path handles all products sending events to SC4S in the CEF format. Examples of this include Arcsight, -Imperva, and Cyberark. Therefore, the CEF environment varialbes for unique port, archive, etc. +Imperva, and Cyberark. Therefore, the CEF environment variables for unique port, archive, etc. should be set only _once_. If your deployment has multiple CEF devices that send to more than one port, set the CEF unique port variable(s) to just one of the ports in use. Then, map the others with -container networking to the port chosen. Example: If you have three CEF devices, sending on TCP -ports 2000,2001, and 2002, set `SC4S_LISTEN_CEF_TCP_PORT=2000`. Then, map the other two with -container networking, e.g. `-p 2000:2000 -p 2001:2000 -p 2002:2000`. This will route all -three ports to TCP port 2000 inside the container, and the single CEF log path will properly -process data from all three devices. +container networking to the port chosen, similar to the way default ports are configured (see the +"Getting Started" runtime documents for more details). + +Example: If you have three CEF devices, +sending on TCP ports 2000,2001, and 2002, set `SC4S_LISTEN_CEF_TCP_PORT=2000`. Then, change the +unit/compose files to route the three external ports to the single port 2000 on the container. +Here is the example for podman/systemd: + +``` +ExecStart=/usr/bin/podman -p 514:514 -p 514:514/udp -p 6514:6514 -p 2000-2002:2000 \ +``` + +or this, for docker-compose/swarm installations: + +``` +# Comment the following line out if using docker-compose + mode: host + - target: 2000 + published: 2000-2002 + protocol: tcp +``` + +These changes will route all three ports to TCP port 2000 inside the container, and the single CEF log +path will properly process data from all three devices. The source documentation included below is a reference baseline for any product that sends data using the CEF log path.