From 0e5795243d5aaa6dcbfc5224dae07a570d0b0d36 Mon Sep 17 00:00:00 2001 From: mbonsack Date: Fri, 6 Dec 2019 04:27:00 -0800 Subject: [PATCH] Config/runtime docs refinement (#215) Significant update to the configuration and runtime doc * Added metadata override detail * Added missing mount point detail * Added detail to runtime docs * Misc fixes --- docs/configuration.md | 145 +++++++++++++----- docs/gettingstarted/byoe-rhel7.md | 2 + docs/gettingstarted/docker-swarm-general.md | 20 ++- docs/gettingstarted/docker-swarm-rhel7.md | 22 ++- docs/gettingstarted/docker-systemd-general.md | 24 ++- docs/gettingstarted/podman-systemd-general.md | 24 ++- 6 files changed, 181 insertions(+), 56 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 5926646..588a783 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -44,9 +44,8 @@ may hide this nuance. ## Archive File Configuration -This feature is designed to support "compliance" archival of all messages. To enable this feature update the Unit file -or docker compose to mount an appropriate host folder to the container folder ``/opt/syslog-ng/var/archive``. -The files will be stored in a folder structure using the naming pattern +This feature is designed to support "compliance" archival of all messages. Instructions for enabling this feature are included +in each "getting started" runtime document. The files will be stored in a folder structure using the naming pattern ``${YEAR}/${MONTH}/${DAY}/${fields.sc4s_vendor_product}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}.log"``. This pattern will create one file per "vendor_product" per minute with records formatted using syslog-ng's EWMM template. @@ -76,7 +75,8 @@ and/or move them to an archival system to avoid disk space failures. ## Syslog Source TLS Certificate Configuration -* Create a folder ``/opt/sc4s/tls`` +* Create a folder ``/opt/sc4s/tls`` if not already done as part of the "getting started" process. +* Uncomment the appropriate mount line in the unit or yaml file (again, documented in the "getting started" runtime documents). * Save the server private key in PEM format with NO PASSWORD to ``/opt/sc4s/tls/server.key`` * Save the server certificate in PEM format to ``/opt/sc4s/tls/server.pem`` * Add the following line to ``/opt/sc4s/env_file`` @@ -85,44 +85,115 @@ and/or move them to an archival system to avoid disk space failures. SC4S_SOURCE_TLS_ENABLE=yes ``` -## Override index or metadata based on host, ip, or subnet +## Log Path overrides of index or metadata + +In some cases it is appropriate to override the default SC4S index or other Splunk metadata (such as an +source, host, or sourcetype) for a given data source. This is accomplished by the use of a lookup file that identifies these +source exceptions based on the log path used by the incoming message. These log path overrides are documented in the associated +"sources" document. For each data source, you will see a table of the form + +| key | sourcetype | index | notes | +|----------------|----------------|----------------|----------------| +| foo_bar | fb_log | netfw | none | + +In this case the key, `foo_bar`, will be an entry into the `splunk_indexes.csv` file that is populated in `/opt/sc4s/local/context` when SC4S +is run for the first time. The other columns show the default sourcetype and index when not overriden. This file contruct +is best shown with an example. Here is the table for Juniper Netscreen devices, from the "sources" document: -In some cases it is appropriate to re-direct events to an alternate index or append metadata (such as an -indexed field) based on PCI scope, geography, or other criterion. This is accomplished by the use -of a file that uniquely identifies these source exceptions via syslog-ng filters, -which maps to an associated lookup of alternate indexes, sources, or other metadata. +| key | sourcetype | index | notes | +|------------------------|---------------------|----------------|---------------| +| juniper_netscreen | netscreen:firewall | netfw | none | +| juniper_idp | juniper:idp | netfw | none | -* Get the filter and lookup files -```bash -cd /opt/sc4s/default -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context_templates/compliance_meta_by_source.conf -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context_templates/compliance_meta_by_source.csv +Here is a snippet from the `splunk_indexes.csv` file: + +``` +#juniper_sslvpn,index,netfw +juniper_netscreen,index,ns_index +#juniper_nsm,index,netfw ``` + +The columns in this file are `key`, `metadata`, and `value`. The `key` entries are +by default "commmented out", which is really a half-truth because CSV files don't allow comments. Therefore, to ensure there +is a match from the log path that references this file, be sure to remove the leading `#`. Once this is done, the following changes can be +made by adding one or more rows to the table and specifying one or more of the following `metadata`/`value` pairs for a given `key`: + + * `index` to specify an alternate `value` for index + * `source` to specify an alternate `value` for source + * `sourcetype` to specify an alternate `value` for sourcetype (be _very_ careful when changing this; only do so if a downstream + TA is _not_ being used, or a custom TA (built by you) is being used.) + +In this case, the `juniper_netscreen` key is "uncommented" (thereby enabling it), and the new index used for that data source will be +`ns_index`. + +In general, for most deployments the index should be the only change needed; the defaults for the others should almost +never be overridden (particularly for "Out of the Box" data sources). Even then, care should be taken when considering an alternate +index, as the defaults for SC4S were chosen with best practices in mind. + +This `csv` file can also be appended when building custom SC4S log paths (filters). Care should be taken during filter design to choose +appropriate index and sourctype defaults, so that admins are not compelled to override them. + + +## Override index or metadata based on host, ip, or subnet + +In other cases it is appropriate to provide the same overrides but based on PCI scope, geography, or other criterion rather than globally. +This is accomplished by the use of a file that uniquely identifies these source exceptions via syslog-ng filters, +which maps to an associated lookup of alternate indexes, sources, or other metadata. In addition, (indexed) fields can also be +added to futher classify the data. + +* The `conf` and `csv` files referenced below will be populated into the `/opt/sc4s/local/context` directory when SC4S is run for the first +time after being set up according to the "getting started" runtime documents. * Edit the file ``compliance_meta_by_source.conf`` to supply uniquely named filters to identify events subject to override. -* Edit the file ``compliance_meta_by_source.csv`` to supply appropriate the field(s) and values. -The three columns in the table are `filter name`, `field name`, and `value`. `field name` obeys the following convention: - * ``fields.fieldname`` where `fieldname` will become the name of an indexed field with the supplied value - * ``.splunk.index`` to specify an alternate value for index - * ``.splunk.source`` to specify an alternate value for source +* Edit the file ``compliance_meta_by_source.csv`` to supply appropriate field(s) and values. + +The three columns in the `csv` file are `filter name`, `field name`, and `value`. Filter names in the `conf` file must match one or more +corresonding `filter name` rows in the `csv` file. The `field name` column obeys the following convention: + + * `.splunk.index` to specify an alternate `value` for index + * `.splunk.source` to specify an alternate `value` for source + * `.splunk.sourcetype` to specify an alternate `value` for sourcetype (be _very_ careful when changing this; only do so if a downstream + TA is _not_ being used, or a custom TA (built by you) is being used.) + * `fields.fieldname` where `fieldname` will become the name of an indexed field sent to Splunk with the supplied `value` + +This file construct is best shown by an example. Here is a sample ``compliance_meta_by_source.conf`` file: + +``` +@version: 3.24 +filter f_test_test { + host("something-*" type(glob)) or + netmask(192.168.100.1/24) +}; +``` +and the corresponding ``compliance_meta_by_source.csv`` file: + +``` +f_test_test,.splunk.index,"pciindex" +f_test_test,fields.compliance,"pci" +``` + +First off, ensure that the proper version string exists at the top of the `conf` file, and that the filter name(s) in the `conf` file match +one or more rows in the `csv` file. In this case, any incoming message with a hostname starting with `something-` or arriving from a netmask +of `192.168.100.1/24` will match the `f_test_test` filter, and the corresponding entries in the `csv` file will be checked for overrides. +In this case, the new index is `pciindex`, and an indexed field named `compliance` will be sent to Splunk, with it's value set to `pci`. +To add additional overrides, simply add another `filter foo_bar {};` stanza to the `conf` file, and add appropriate entries to the `csv` file +that match the filter name(s) to the overrides you deisre. + +* IMPORTANT: The files above are actual syslog-ng config file snippets that get parsed directly by the underlying syslog-ng +process. Take care that your syntax is correct; for more information on proper syslog-ng syntax, see the syslog-ng +[documentation](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.24/administration-guide/57#TOPIC-1298086). +A syntax error will cause the runtime process to abort in the "preflight" phase at startup. -* For the Docker/Podman runtimes, update the docker/podman run command in the systemd unit file or the docker-compose to -include volumes mapping the files above. -* In the Unit file, add the following lines to the `ExecStart` command prior to `$SC4SIMAGE` then restart using the command -``sudo systemctl daemon-reload; sudo systemctl restart sc4s`` - -`` -SC4S_UNIT_VP_CSV=-v /opt/sc4s/default/compliance_meta_by_source.csv:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.csv \ -SC4S_UNIT_VP_CONF=-v /opt/sc4s/default/compliance_meta_by_source.conf:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.conf \ -`` - -* For the Docker Swarm runtime, update the docker compose yml to add the following volume mounts to thee sc4s service and -redeploy the updated service using the command: -``docker stack deploy --compose-file docker-compose.yml sc4s`` - -`` - - /opt/sc4s/default/compliance_meta_by_source.csv:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.csv - - /opt/sc4s/default/compliance_meta_by_source.conf:/opt/syslog-ng/etc/context-local/compliance_meta_by_source.conf -`` +Finally, to update your changes for the systemd-based runtimes, restart SC4S using the commands: +``` +sudo systemctl daemon-reload +sudo systemctl restart sc4s +``` + +For the Docker Swarm runtime, redeploy the updated service using the command: +``` +docker stack deploy --compose-file docker-compose.yml sc4s +``` + ## Data Durability - Local Disk Buffer Configuration SC4S provides capability to minimize the number of lost events if the connection to all the Splunk Indexers goes down. This capability utilizes the disk buffering feature of Syslog-ng. SC4S receives a response from the Splunk HTTP Event Collector (HEC) when a message is received successfully. If a confirmation message from the HEC endpoint is not received (or a “server busy” reply, such as a “503” is sent), the load balancer will try the next HEC endpoint in the pool. If all pool members are exhausted (such as would occur if there were a full network outage to the HEC endpoints), events will queue to the local disk buffer on the SC4S Linux host. SC4S will continue attempting to send the failed events while it buffers all new incoming events to disk. If the disk space allocated to disk buffering fills up then SC4S will stop accepting new events and subsequent events will be lost. Once SC4S gets confirmation that events are again being received by one or more indexers, events will then stream from the buffer using FIFO queueing. The number of events in the disk buffer will reduce as long as the incoming event volume is less than the maximum SC4S (with the disk buffer in the path) can handle. When all events have been emptied from the disk buffer, SC4S will resume streaming events directly to Splunk. diff --git a/docs/gettingstarted/byoe-rhel7.md b/docs/gettingstarted/byoe-rhel7.md index d81c566..59baff5 100644 --- a/docs/gettingstarted/byoe-rhel7.md +++ b/docs/gettingstarted/byoe-rhel7.md @@ -127,6 +127,8 @@ mkdir -p /opt/syslog-ng/etc/conf.d/local/config/ cp --verbose -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context/ cp --verbose -R -n /opt/syslog-ng/etc/local_config/* /opt/syslog-ng/etc/conf.d/local/config/ mkdir -p /opt/syslog-ng/var/data/disk-buffer/ +mkdir -p /opt/syslog-ng/var/archive/ +mkdir -p /opt/syslog-ng/tls/ ``` * (Optional) Execute the preconfiguration shell script created above. You may also optionally execute it as part of the unit diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index dfa8b4d..44d2255 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -51,8 +51,10 @@ services: volumes: - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z +# Uncomment the following line if local disk archiving is desired +# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z # Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls +# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. @@ -76,7 +78,14 @@ of events in the event of network failure to the Splunk infrastructure. If you are sure, after stopping SC4S, that all data has been sent, these files can be removed. They will be created again upon restart. -* IMPORTANT: When creating the two directories above, ensure the directories created match the volume mounts specified in the +* Create the subdirectory ``/opt/sc4s/archive``. This will be used as a mount point for local storage of syslog events +(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration" +document for details on the directory structure the archive uses. + +* Create the subdirectory ``/opt/sc4s/tls``. This will be used as a mount point for custom TLS certificates +(if the optional mount is uncommented above). + +* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the `docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup. ## Configure the SC4S environment @@ -107,12 +116,15 @@ Log paths are preconfigured to utilize a convention of index destinations that a * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. ## Configure source filtering by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. @@ -122,8 +134,8 @@ apply to support such sources. To identify sources that require this step, refer In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in -the "Configuration" section. +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. ## Start/Restart SC4S diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index 7ed5afe..0af8b6b 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -59,8 +59,10 @@ services: volumes: - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z - /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z +# Uncomment the following line if local disk archiving is desired +# - /opt/sc4s/archive:/opt/syslog-ng/var/archive:z # Uncomment the following line if custom TLS certs are provided -# - /opt/sc4s/tls:/opt/syslog-ng/tls +# - /opt/sc4s/tls:/opt/syslog-ng/tls:z ``` * Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations. @@ -82,9 +84,16 @@ of events in the event of network failure to the Splunk infrastructure. * This directory will populate with the disk buffer files upon SC4S startup. If SC4S restarts for any reason, a new set of files will be created in addition to the original ones. _The original ones will not be removed_. If you are sure, after stopping SC4S, that all data has been sent, these files can be removed. They will be created -again upon restart. +again upon restart + +* Create the subdirectory ``/opt/sc4s/archive``. This will be used as a mount point for local storage of syslog events +(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration" +document for details on the directory structure the archive uses. + +* Create the subdirectory ``/opt/sc4s/tls``. This will be used as a mount point for custom TLS certificates +(if the optional mount is uncommented above). -* IMPORTANT: When creating the two directories above, ensure the directories created match the volume mounts specified in the +* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the `docker-compose.yml` file. Failure to do this will cause SC4S to abort at startup. ## Configure the SC4S environment @@ -116,6 +125,8 @@ Log paths are preconfigured to utilize a convention of index destinations that a * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. ## Configure source filtering by source IP or host name @@ -123,6 +134,7 @@ this table that pertain to the individual data source filters that are included Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. @@ -132,8 +144,8 @@ apply to support such sources. To identify sources that require this step, refer In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in -the "Configuration" section. +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. ## Start/Restart SC4S diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 838cdaa..7eae5b0 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -40,8 +40,10 @@ Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf. # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" +# Uncomment the following line if local disk archiving is desired +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" # Uncomment the following line if custom TLS certs are provided -# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" +# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z" TimeoutStartSec=0 Restart=always @@ -56,6 +58,8 @@ ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ + "$SC4S_LOCAL_ARCHIVE_MOUNT" \ + "$SC4S_TLS_DIR" \ --name SC4S --rm \ $SC4S_IMAGE ``` @@ -79,9 +83,16 @@ of events in the event of network failure to the Splunk infrastructure. * This directory will populate with the disk buffer files upon SC4S startup. If SC4S restarts for any reason, a new set of files will be created in addition to the original ones. _The original ones will not be removed_. If you are sure, after stopping SC4S, that all data has been sent, these files can be removed. They will be created -again upon restart. +again upon restart + +* Create the subdirectory ``/opt/sc4s/archive``. This will be used as a mount point for local storage of syslog events +(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration" +document for details on the directory structure the archive uses. + +* Create the subdirectory ``/opt/sc4s/tls``. This will be used as a mount point for custom TLS certificates +(if the optional mount is uncommented above). -* IMPORTANT: When creating the two directories above, ensure the directories created match the volume mounts specified in the +* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the unit file above. Failure to do this will cause SC4S to abort at startup. ## Configure the SC4S environment @@ -112,12 +123,15 @@ Log paths are preconfigured to utilize a convention of index destinations that a * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. ## Configure source filtering by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. @@ -127,8 +141,8 @@ apply to support such sources. To identify sources that require this step, refer In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in -the "Configuration" section. +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. ## Configure SC4S for systemd and start SC4S diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 1cec535..18f8f82 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -22,8 +22,10 @@ Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf. # Mount point for local disk buffer (required) Environment="SC4S_LOCAL_DISK_BUFFER_MOUNT=-v /opt/sc4s/disk-buffer:/opt/syslog-ng/var/data/disk-buffer:z" +# Uncomment the following line if local disk archiving is desired +# Environment="SC4S_LOCAL_ARCHIVE_MOUNT=-v /opt/sc4s/archive:/opt/syslog-ng/var/archive:z" # Uncomment the following line if custom TLS certs are provided -# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" +# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z" TimeoutStartSec=0 Restart=always @@ -38,6 +40,8 @@ ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp \ --env-file=/opt/sc4s/env_file \ "$SC4S_LOCAL_CONFIG_MOUNT" \ "$SC4S_LOCAL_DISK_BUFFER_MOUNT" \ + "$SC4S_LOCAL_ARCHIVE_MOUNT" \ + "$SC4S_TLS_DIR" \ --name SC4S --rm \ $SC4S_IMAGE ``` @@ -61,9 +65,16 @@ of events in the event of network failure to the Splunk infrastructure. * This directory will populate with the disk buffer files upon SC4S startup. If SC4S restarts for any reason, a new set of files will be created in addition to the original ones. _The original ones will not be removed_. If you are sure, after stopping SC4S, that all data has been sent, these files can be removed. They will be created -again upon restart. +again upon restart + +* Create the subdirectory ``/opt/sc4s/archive``. This will be used as a mount point for local storage of syslog events +(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration" +document for details on the directory structure the archive uses. + +* Create the subdirectory ``/opt/sc4s/tls``. This will be used as a mount point for custom TLS certificates +(if the optional mount is uncommented above). -* IMPORTANT: When creating the two directories above, ensure the directories created match the volume mounts specified in the +* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the unit file above. Failure to do this will cause SC4S to abort at startup. ## Configure the sc4s environment @@ -94,12 +105,15 @@ Log paths are preconfigured to utilize a convention of index destinations that a * Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in this table that pertain to the individual data source filters that are included with SC4S. +* Other Splunk metadata (e.g. source and sourcetype) can be overriden via this file as well. This is an advanced topic, and further +information is covered in the "Log Path overrides" section of the Configuration document. ## Configure source filtering by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +* If changes need to be made to source filtering, navigate to the ``/opt/sc4s/local/context`` directory to start. * Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. * Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. * The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. @@ -109,8 +123,8 @@ apply to support such sources. To identify sources that require this step, refer In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in -the "Configuration" section. +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is +covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. ## Configure SC4S for systemd and start SC4S