From d7f01617ac19039fd65b7a74e3131f4f087d4c66 Mon Sep 17 00:00:00 2001 From: l-kutch <55860089+l-kutch@users.noreply.github.com> Date: Fri, 4 Oct 2019 12:04:38 -0700 Subject: [PATCH 01/67] removing optional parts (#109) * Revising to make the optional parts mandatory for other getting started files --- docs/gettingstarted/docker-swarm-general.md | 40 ++++++++--------- docs/gettingstarted/docker-swarm-rhel7.md | 39 ++++++++-------- docs/gettingstarted/docker-systemd-general.md | 38 +++++++--------- docs/gettingstarted/podman-systemd-general.md | 45 +++++++++---------- 4 files changed, 74 insertions(+), 88 deletions(-) diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index 9fef61c..f9a6e12 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -35,21 +35,11 @@ services: - /opt/sc4s/tls:/opt/syslog-ng/tls ``` -* NOTE: While strictly optional, it is recommended that you create and/or download all files and directories referenced in the yml template -above (`volumes` declarations) according to the configuration steps that follow. The TLS options are described in the "Configuration" section. -Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors. +* NOTE: If you use the default `volumes` declarations as-is from the `docker-compose.yml` file template example, you must create and/or download all files and directories referenced in the file according to the configuration steps that follow. The TLS-specific options are described in the "Configure the sc4s Environment" section. Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors. ## Configure the SC4S environment -Create the following file ``/opt/sc4s/env_file`` and add the environment variables below: - -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment - -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. - -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event @@ -62,32 +52,38 @@ SPLUNK_METRICS_INDEX=em_metrics #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` +* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. + +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example above. + ## Configure index destinations for Splunk -Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow -customization of index destinations. +Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. -* Create a subdirectory called ``default`` in the directory (e.g. ``/opt/sc4s/``) created in the first step above. From this directory, +* Create a subdirectory called ``default`` in the directory that you created in the previous step (e.g. ``/opt/sc4s/``). Make sure the local directory volume references in the `yml` file match the directory you create here. From this directory, execute the command below to download the index context file: ```bash sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv ``` -* Edit splunk_index.csv review the index configuration and revise as required for sourcetypes utilized in your environment. +* Edit splunk_index.csv to review the index configuration and revise as required for the sourcetypes utilized in your environment. ## Configure sources by source IP or host name -Legacy sources and non-standard-compliant source require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation. +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* If not already done in the step immediately above, create a subdirectory called ``default`` in the directory (e.g. ``/opt/sc4s/``) -created in the first step above. From this directory, execute the commands below to download the vendor context files: +* If not already done, create a subdirectory called ``default`` in the ``/opt/sc4s/`` directory. Make sure the local directory volume references in the `yml` file match the directory you create here. From this directory, execute the following commands to download the vendor context files: ```bash sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv ``` -* Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax. +* If you have legacy sources and non-standard-compliant sources, edit the file to properly identify these products by host glob or network mask using syslog-ng filter syntax. ## Start/Restart SC4S @@ -224,4 +220,4 @@ Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', cl ``` If you see http server errors such as 4xx or 5xx responses from the http (HEC) endpoint, one or more of the items above are likely set incorrectly. If validating/fixing the configuration fails to correct the problem, proceed to the "Troubleshooting" section for more -information. \ No newline at end of file +information. diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index eba329c..008dd65 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -63,21 +63,12 @@ services: - /opt/sc4s/tls:/opt/syslog-ng/tls ``` -* NOTE: While strictly optional, it is recommended that you create and/or download all files and directories referenced in the yml template -above (`volumes` declarations) according to the configuration steps that follow. The TLS options are described in the "Configuration" section. -Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors. +* NOTE: If you use the default `volumes` declarations as-is from the `docker-compose.yml` file template example, do create and/or download all files and directories referenced in the file according to the configuration steps that follow. The TLS-specific options are described in the "Configure the sc4s environment" section. Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors. -## Configure the SC4S environment - -Create the following file ``/opt/sc4s/env_file`` and add the environment variables below: -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment - -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. +## Configure the SC4S environment -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +Create a file named ``/opt/sc4s/env_file`` and add the following environment variables: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event @@ -90,32 +81,40 @@ SPLUNK_METRICS_INDEX=em_metrics #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` +* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment. + +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example below. + + + ## Configure index destinations for Splunk -Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow -customization of index destinations. +Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. -* Create a subdirectory called ``default`` in the directory (e.g. ``/opt/sc4s/``) created in the first step above. From this directory, +* Create a subdirectory called ``default`` in the directory that you created in the previous step (e.g. ``/opt/sc4s/``). Make sure the local directory volume references in the `yml` file match the directory you create here. From this directory, execute the command below to download the index context file: ```bash sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv ``` -* Edit splunk_index.csv review the index configuration and revise as required for sourcertypes utilized in your environment. +* Edit splunk_index.csv to review the index configuration and revise as required for the sourcertypes utilized in your environment. ## Configure sources by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation. +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* If not already done in the step immediately above, create a subdirectory called ``default`` in the directory (e.g. ``/opt/sc4s/``) -created in the first step above. From this directory, execute the commands below to download the vendor context files: +* If not already done, create a subdirectory called ``default`` in the ``/opt/sc4s/`` directory. Make sure the local directory volume references in the `yml` file match the directory you create here. From this directory, execute the following commands to download the vendor context files: ```bash sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv ``` -* Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax. +* If you have legacy sources and non-standard-compliant sources, edit the file to properly identify these products by host glob or network mask using syslog-ng filter syntax. ## Start/Restart SC4S diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index d4cd5c4..0e1fc58 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -42,20 +42,11 @@ ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp \ $SC4S_IMAGE ``` -* NOTE: While strictly optional, it is recommended that you create and/or download all files and directories referenced in the unit -file above (the `Environment` assignments) according to the configuration steps that follow. The TLS options are described in the "Configuration" section. +* NOTE: If you use the default `Environment` assignments as-is from the `sc4s.service` unit file template example, do create and/or download all files and directories referenced in the file's Service stanza according to the configuration steps that follow. The TLS-specific options are described in the "Configure the sc4s environment" section. ## Configure the SC4S environment -Create the following file ``/opt/sc4s/default/env_file`` and add the environment variables below: - -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment - -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. - -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +Create a file named ``/opt/sc4s/default/env_file`` and add the following environment variables: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event @@ -68,33 +59,38 @@ SPLUNK_METRICS_INDEX=em_metrics #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` +* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment + +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example. + + ## Configure index destinations for Splunk -Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow -customization of index destinations. +Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. -* Create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the unit file above (the ``-v`` variables) -match the directory you created above. From this directory, execute the following to download the latest index context file: +* Create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the `sc4s.service` unit file match the directory you create here (the ``-v`` variables). From this directory, execute the following to download the latest index context file: ```bash sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv ``` -* Edit splunk_index.csv review the index configuration and revise as required for sourcertypes utilized in your environment. +* Edit splunk_index.csv to review the index configuration and revise as required for the sourcertypes utilized in your environment. ## Configure sources by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation. +apply to support such sources. To identify sources that require this step refer to the "sources" section of this documentation. -* If not already done in the step immediately above, create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory -references in the unit file above (the ``-v`` variables) match the directory you created above. From this directory, execute the following to -download the latest vendor context files: +* If not already done, create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the `sc4s.service` unit file match the directory you create here (the ``-v`` variables). From this directory, execute the following to download the latest vendor context files: ```bash sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv ``` -* Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax. +* If you have legacy sources and non-standard-compliant sources, edit the file to properly identify these products by host glob or network mask using syslog-ng filter syntax. ## Configure SC4S for systemd and start SC4S diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 74b6855..361ac12 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -20,8 +20,8 @@ Environment="SC4S_IMAGE=splunk/scs:latest" Environment="SC4S_UNIT_SPLUNK_INDEX=-v /opt/sc4s/default/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv" #Note Uncomment the following two lines for host and ip based source type mapping AND download the two file templates per getting started -#Environment="SC4S_UNIT_VP_CSV=-v /opt/sc4s/default/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv" -#Environment="SC4S_UNIT_VP_CONF=-v /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf" +Environment="SC4S_UNIT_VP_CSV=-v /opt/sc4s/default/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv" +Environment="SC4S_UNIT_VP_CONF=-v /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf" #Uncomment the following line if custom TLS certs are provided #Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls" @@ -42,20 +42,11 @@ ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp \ $SC4S_IMAGE ``` -* NOTE: While strictly optional, it is recommended that you create and/or download all files and directories referenced in the unit -file above (the `Environment` assignments) according to the configuration steps that follow. The TLS options are described in the "Configuration" section. +* NOTE: If you use the default `Environment` assignments as-is from the `sc4s.service` unit file template example, do create and/or download all files and directories referenced in the file's Service stanza according to the configuration steps that follow. The TLS-specific options are described in the "Configure the sc4s environment" section. ## Configure the sc4s environment -Create the following file ``/opt/sc4s/default/env_file`` and add the environment variables below: - -* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment - -* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, -match this value to the total number of indexers behind the load balancer. - -* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to -uncomment the last line in the example below. +Create a file named ``/opt/sc4s/default/env_file`` and add the following environment variables: ```dotenv SPLUNK_HEC_URL=https://splunk.smg.aws:8088/services/collector/event @@ -68,33 +59,37 @@ SPLUNK_METRICS_INDEX=em_metrics #SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no ``` +* Update ``SPLUNK_HEC_URL`` and ``SPLUNK_HEC_TOKEN`` to reflect the correct values for your environment + +* Set `SC4S_DEST_SPLUNK_HEC_WORKERS` to match the number of indexers and/or HWFs with HEC endpoints. If the endpoint is a VIP, +match this value to the total number of indexers behind the load balancer. + +* NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to +uncomment the last line in the example. + + ## Configure index destinations for Splunk -Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. This step is optional to allow -customization of index destinations. +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the unit file above (the ``-v`` variables) -match the directory you created above. From this directory, execute the following to download the latest index context file: +* Create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the `sc4s.service` unit file match the directory you create here (the ``-v`` variables). From this directory, execute the following to download the latest index context file: ```bash sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv ``` -* Edit splunk_index.csv review the index configuration and revise as required for sourcertypes utilized in your environment. +* Edit splunk_index.csv to review the index configuration and revise as required for the sourcertypes utilized in your environment. ## Configure sources by source IP or host name -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources which require this step refer to the "sources" section of this documentation. +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* If not already done in the step immediately above, create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory -references in the unit file above (the ``-v`` variables) match the directory you created above. From this directory, execute the following to -download the latest vendor context files: +* If not already done, create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the `sc4s.service` unit file match the directory you create here (the ``-v`` variables). From this directory, execute the following to download the latest vendor context files: ```bash sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv ``` -* Edit the file to identify appropriate vendor products by host glob or network mask using syslog-ng filter syntax. +* If you have legacy sources and non-standard-compliant sources, edit the file to properly identify these products by host glob or network mask using syslog-ng filter syntax. ## Configure SC4S for systemd and start SC4S @@ -235,4 +230,4 @@ Oct 1 05:29:55 77cd4776af41 syslog-ng[1]: Syslog connection closed; fd='49', cl ``` If you see http server errors such as 4xx or 5xx responses from the http (HEC) endpoint, one or more of the items above are likely set incorrectly. If validating/fixing the configuration fails to correct the problem, proceed to the "Troubleshooting" section for more -information. \ No newline at end of file +information. From 928350baa111f517b06d3b04442b0593018a15a7 Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Sat, 5 Oct 2019 22:35:27 -0700 Subject: [PATCH 02/67] update to single mount point --- package/Dockerfile | 4 +- .../conflib/_common/compliance_meta.conf | 4 +- .../vendor_product_by_source_context.conf | 4 +- .../conflib/_splunk/splunk_context.conf | 4 +- .../context}/microfocus_arcsight_source.csv | 0 .../etc/conf.d/local/destinations/README.md | 1 - package/etc/conf.d/local/filters/README.md | 1 - package/etc/conf.d/local/filters/example.conf | 4 - package/etc/conf.d/local/log_paths/README.md | 1 - .../conf.d/local/log_paths/example.conf.tmpl | 76 ------------------- package/etc/conf.d/local/sources/README.md | 1 - .../p_rfc3164_microfocus_arcsight.conf.tmpl | 2 +- .../compliance_meta_by_source.conf | 5 -- .../compliance_meta_by_source.csv | 2 - package/etc/context-local/splunk_index.csv | 40 ---------- .../vendor_product_by_source.conf | 37 --------- .../vendor_product_by_source.csv | 8 -- .../source_network.t | 0 package/etc/syslog-ng.conf | 9 +-- package/sbin/entrypoint.sh | 2 +- 20 files changed, 14 insertions(+), 191 deletions(-) rename package/etc/{context-local => conf.d/context}/microfocus_arcsight_source.csv (100%) delete mode 100644 package/etc/conf.d/local/destinations/README.md delete mode 100644 package/etc/conf.d/local/filters/README.md delete mode 100644 package/etc/conf.d/local/filters/example.conf delete mode 100644 package/etc/conf.d/local/log_paths/README.md delete mode 100644 package/etc/conf.d/local/log_paths/example.conf.tmpl delete mode 100644 package/etc/conf.d/local/sources/README.md delete mode 100644 package/etc/context-local/compliance_meta_by_source.conf delete mode 100644 package/etc/context-local/compliance_meta_by_source.csv delete mode 100644 package/etc/context-local/splunk_index.csv delete mode 100644 package/etc/context-local/vendor_product_by_source.conf delete mode 100644 package/etc/context-local/vendor_product_by_source.csv rename package/etc/{templates => go_templates}/source_network.t (100%) diff --git a/package/Dockerfile b/package/Dockerfile index a5f67cc..570a16a 100644 --- a/package/Dockerfile +++ b/package/Dockerfile @@ -86,8 +86,8 @@ RUN curl -o /usr/local/bin/gomplate -sSL https://github.com/hairyhenderson/gompl COPY etc/syslog-ng.conf /opt/syslog-ng/etc/syslog-ng.conf COPY etc/conf.d /opt/syslog-ng/etc/conf.d -COPY etc/templates /opt/syslog-ng/etc/templates -COPY etc/context-local /opt/syslog-ng/etc/context-local +COPY etc/go_templates /opt/syslog-ng/etc/go_templates +COPY etc/conf.d/local/context /opt/syslog-ng/etc/conf.d/local/context COPY sbin/entrypoint.sh / RUN mkdir -p /opt/syslog-ng/var/data/disk-buffer RUN source scl_source enable rh-python36 ;/opt/syslog-ng/sbin/syslog-ng -V diff --git a/package/etc/conf.d/conflib/_common/compliance_meta.conf b/package/etc/conf.d/conflib/_common/compliance_meta.conf index 3529ae5..75479c6 100644 --- a/package/etc/conf.d/conflib/_common/compliance_meta.conf +++ b/package/etc/conf.d/conflib/_common/compliance_meta.conf @@ -1,7 +1,7 @@ parser compliance_meta_by_source { add-contextual-data( - selector(filters("`syslog-ng-sysconfdir`/context-local/compliance_meta_by_source.conf")), - database("context-local/compliance_meta_by_source.csv") + selector(filters("`syslog-ng-sysconfdir`/conf.d/local/context/compliance_meta_by_source.conf")), + database("conf.d/local/context/compliance_meta_by_source.csv") ignore-case(yes) ); }; diff --git a/package/etc/conf.d/conflib/_common/vendor_product_by_source_context.conf b/package/etc/conf.d/conflib/_common/vendor_product_by_source_context.conf index f3789a7..44d8ff1 100644 --- a/package/etc/conf.d/conflib/_common/vendor_product_by_source_context.conf +++ b/package/etc/conf.d/conflib/_common/vendor_product_by_source_context.conf @@ -1,7 +1,7 @@ block parser vendor_product_by_source() { add-contextual-data( - selector(filters("`syslog-ng-sysconfdir`/context-local/vendor_product_by_source.conf")), - database("context-local/vendor_product_by_source.csv") + selector(filters("`syslog-ng-sysconfdir`/conf.d/local/context/vendor_product_by_source.conf")), + database("conf.d/local/context/vendor_product_by_source.csv") ignore-case(yes) prefix("fields.") ); diff --git a/package/etc/conf.d/conflib/_splunk/splunk_context.conf b/package/etc/conf.d/conflib/_splunk/splunk_context.conf index e2d1fbb..6fb181d 100644 --- a/package/etc/conf.d/conflib/_splunk/splunk_context.conf +++ b/package/etc/conf.d/conflib/_splunk/splunk_context.conf @@ -1,7 +1,7 @@ block parser p_add_context_splunk(key("syslogng-fallback")) { add-contextual-data( selector("`key`"), - database("context-local/splunk_index.csv"), + database("conf.d/local/context/splunk_index.csv"), prefix(".splunk.") ); -}; \ No newline at end of file +}; diff --git a/package/etc/context-local/microfocus_arcsight_source.csv b/package/etc/conf.d/context/microfocus_arcsight_source.csv similarity index 100% rename from package/etc/context-local/microfocus_arcsight_source.csv rename to package/etc/conf.d/context/microfocus_arcsight_source.csv diff --git a/package/etc/conf.d/local/destinations/README.md b/package/etc/conf.d/local/destinations/README.md deleted file mode 100644 index ee6571d..0000000 --- a/package/etc/conf.d/local/destinations/README.md +++ /dev/null @@ -1 +0,0 @@ -This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/conf.d/local/filters/README.md b/package/etc/conf.d/local/filters/README.md deleted file mode 100644 index ee6571d..0000000 --- a/package/etc/conf.d/local/filters/README.md +++ /dev/null @@ -1 +0,0 @@ -This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/conf.d/local/filters/example.conf b/package/etc/conf.d/local/filters/example.conf deleted file mode 100644 index 047fdc8..0000000 --- a/package/etc/conf.d/local/filters/example.conf +++ /dev/null @@ -1,4 +0,0 @@ - -filter f_local_example { - program(sc4splugin); -}; \ No newline at end of file diff --git a/package/etc/conf.d/local/log_paths/README.md b/package/etc/conf.d/local/log_paths/README.md deleted file mode 100644 index ee6571d..0000000 --- a/package/etc/conf.d/local/log_paths/README.md +++ /dev/null @@ -1 +0,0 @@ -This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/conf.d/local/log_paths/example.conf.tmpl b/package/etc/conf.d/local/log_paths/example.conf.tmpl deleted file mode 100644 index a8ac264..0000000 --- a/package/etc/conf.d/local/log_paths/example.conf.tmpl +++ /dev/null @@ -1,76 +0,0 @@ -# LOCAL_EXAMPLE - -# When creating a real plugin, replace the upper case text "LOCAL_EXAMPLE" throughout this file with a unique -# string to identify the vendor product. The string should be of the form "VENDOR_PRODUCT" to signify the -# manufacturer and product type, and must contain only characters matching this regex: [A-Z\_]+ - -# If any of the "LOCAL_EXAMPLE" variables passed into the environment are set (e.g. TLS, UDP, or TLS), -# the template generator will build a custom source based on the value of one or more of the set variables. - -{{- if (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_TCP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_UDP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_TLS_PORT") "no") "no") }} - -# "port_id" is used to generate the port variable to be used. It should match the "core" of the variable name -# set in the line above. For example, the "port_id" of "SC4S_LISTEN_LOCAL_EXAMPLE_TCP_PORT" is "LOCAL_EXAMPLE". -# "parser" can be customized on dedicated ports only -# "common" uses the same parser sequence as the default ports and is the most commonly used - -{{ $context := dict "port_id" "LOCAL_EXAMPLE" "parser" "common"}} - -# The following template execution creates a syslog-ng source with one or more dedicated ports for use with this log_path -# The ports used are based on the values of one or more of the environment variables set above. - -{{ tmpl.Exec "t/source_network.t" $context }} -{{- end -}} -{{ define "log_path" }} -log { - -# The first time this template is used the log_path will be linked to the default port - -{{- if eq (.) "yes"}} - source(s_default-ports); - -# Filters should be updated to use the simplest and most effecient logic possible to discard -# the message from this path - - filter(f_is_rfc3164); - filter(f_local_example); -{{- end}} -{{- if eq (.) "no"}} - -# In the second pass through the template a link to the dedicated port is used. This -# normally does not require additional filters - -source (s_dedicated_port_LOCAL_EXAMPLE); -{{- end}} - -#Set a default sourcetype and index - - rewrite { r_set_splunk_dest_default(sourcetype("sc4s:local_example"), index("main"), template("t_msg_only"))}; - -#using the key "local_example" find any cutomized index,source or sourcetype meta values - - parser {p_add_context_splunk(key("local_example")); }; - -# Any additional logic needed to process the event before sending to Splunk goes here - -# Send it to Splunk - - destination(d_hec); #--HEC-- - -# Note: We normally do not use the "final" flag; this will allow another plugin to be created that will -# forward events to another system - - flags(flow-control); - -}; -{{- end}} -{{- if (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_TCP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_UDP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TLS_PORT") "no") "no") }} - -# Listen on the specified dedicated port(s) for LOCAL_EXAMPLE traffic - - {{tmpl.Exec "log_path" "no" }} -{{- end}} - -# Listen on the default port (typically 514) for LOCAL_EXAMPLE traffic - -{{tmpl.Exec "log_path" "yes" }} diff --git a/package/etc/conf.d/local/sources/README.md b/package/etc/conf.d/local/sources/README.md deleted file mode 100644 index ee6571d..0000000 --- a/package/etc/conf.d/local/sources/README.md +++ /dev/null @@ -1 +0,0 @@ -This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/conf.d/log_paths/p_rfc3164_microfocus_arcsight.conf.tmpl b/package/etc/conf.d/log_paths/p_rfc3164_microfocus_arcsight.conf.tmpl index 6170b56..4c89731 100644 --- a/package/etc/conf.d/log_paths/p_rfc3164_microfocus_arcsight.conf.tmpl +++ b/package/etc/conf.d/log_paths/p_rfc3164_microfocus_arcsight.conf.tmpl @@ -22,7 +22,7 @@ parser p_microfocus_arcsight_ts_end { parser p_microfocus_arcsight_source { add-contextual-data( selector("${fields.cef_device_vendor}_${fields.cef_device_product}"), - database("context-local/microfocus_arcsight_source.csv") + database("conf.d/context/microfocus_arcsight_source.csv") ignore-case(yes) prefix(".splunk.") default-selector("unknown") diff --git a/package/etc/context-local/compliance_meta_by_source.conf b/package/etc/context-local/compliance_meta_by_source.conf deleted file mode 100644 index 1d5acae..0000000 --- a/package/etc/context-local/compliance_meta_by_source.conf +++ /dev/null @@ -1,5 +0,0 @@ -@version: 3.23 -filter f_test_test { - host("something-*" type(glob)) or - netmask(192.168.100.1/24) -}; diff --git a/package/etc/context-local/compliance_meta_by_source.csv b/package/etc/context-local/compliance_meta_by_source.csv deleted file mode 100644 index 6608db0..0000000 --- a/package/etc/context-local/compliance_meta_by_source.csv +++ /dev/null @@ -1,2 +0,0 @@ -#f_test_test,.splunk.index,"badindex" -#f_test_test,fields.compliance,"pci" diff --git a/package/etc/context-local/splunk_index.csv b/package/etc/context-local/splunk_index.csv deleted file mode 100644 index e93911a..0000000 --- a/package/etc/context-local/splunk_index.csv +++ /dev/null @@ -1,40 +0,0 @@ -#bluecoat_proxy,index,netproxy -#cef_ArcSight_ArcSight,index,netwaf -#cef_Incapsula_SIEMintegration,index,netwaf -#cef_Microsoft_Microsoft Windows,index,oswinsec -#cef_Microsoft_System or Application Event,index,oswin -#cisco_asa,index,netfw -#cisco_ios,index,netops -#cisco_nx_os,index,netops -#local_example,index,main -#fortinet_fortios_event,index,netops -#fortinet_fortios_log,index,netops -#fortinet_fortios_traffic,index,netfw -#fortinet_fortios_utm,index,netids -#juniper_idp,index,netids -#juniper_structured,index,netops -#juniper_idp_structured,index,netids -#juniper_junos_fw_structured,index,netfw -#juniper_junos_ids_structured,index,netids -#juniper_junos_utm_structured,index,netfw -#juniper_junos_fw,index,netfw -#juniper_junos_ids,index,netids -#juniper_junos_utm,index,netfw -#juniper_sslvpn,index,netfw -#juniper_netscreen,index,netfw -#juniper_nsm,index,netfw -#juniper_nsm_idp,index,netids -#juniper_legacy,index,netops -#pan_traffic,index,netfw -#pan_threat,index,netproxy -#pan_system,index,netops -#pan_config,index,netops -#pan_hipwatch,index,main -#pan_correlation,index,main -#pan_userid,index,netauth -#pan_unknown,index,netops -#proofpoint_pps_filter,index,email -#proofpoint_pps_sendmail,index,email -#sc4s_events,index,main -#sc4s_fallback,index,main -#sc4s_metrics,index,em_metrics diff --git a/package/etc/context-local/vendor_product_by_source.conf b/package/etc/context-local/vendor_product_by_source.conf deleted file mode 100644 index 37e3412..0000000 --- a/package/etc/context-local/vendor_product_by_source.conf +++ /dev/null @@ -1,37 +0,0 @@ -@version: 3.22 -#TODO: #60 The syntax below uses regex and an indirect reference to a variable due to a -#bug/limitation of selector files. The better syntax should be as follows -#filter {match("f5_test" template("$(env PRESUME_SYSLOG)")); }; - -filter f_test_test { - host("testvp-*" type(glob)) or - netmask(192.168.100.1/24) -}; -filter f_juniper_nsm { - host("jnpnsm-*" type(glob)) or - netmask(192.168.1.0/24) -}; -filter f_juniper_nsm_idp { - host("jnpnsmidp-*" type(glob)) or - netmask(192.168.2.0/24) -}; -filter f_juniper_idp { - host("jnpidp-*" type(glob)) or - netmask(192.168.3.0/24) -}; -filter f_juniper_netscreen { - host("jnpns-*" type(glob)) or - netmask(192.168.4.0/24) -}; -filter f_cisco_nx_os { - host("csconx-*" type(glob)) or - netmask(192.168.5.0/24) -}; -filter f_proofpoint_pps_sendmail { - host("pps-*" type(glob)) or - netmask(192.168.6.0/24) -}; -filter f_proofpoint_pps_filter { - host("pps-*" type(glob)) or - netmask(192.168.7.0/24) -}; \ No newline at end of file diff --git a/package/etc/context-local/vendor_product_by_source.csv b/package/etc/context-local/vendor_product_by_source.csv deleted file mode 100644 index 3f90603..0000000 --- a/package/etc/context-local/vendor_product_by_source.csv +++ /dev/null @@ -1,8 +0,0 @@ -f_test_test,sc4s_vendor_product,"test_test" -f_juniper_nsm,sc4s_vendor_product,"juniper_nsm" -f_juniper_nsm_idp,sc4s_vendor_product,"juniper_nsm_idp" -f_juniper_idp,sc4s_vendor_product,"juniper_idp" -f_juniper_netscreen,sc4s_vendor_product,"juniper_netscreen" -f_cisco_nx_os,sc4s_vendor_product,"cisco_nx_os" -f_proofpoint_pps_sendmail,sc4s_vendor_product,"proofpoint_pps_sendmail" -f_proofpoint_pps_filter,sc4s_vendor_product,"proofpoint_pps_filter" \ No newline at end of file diff --git a/package/etc/templates/source_network.t b/package/etc/go_templates/source_network.t similarity index 100% rename from package/etc/templates/source_network.t rename to package/etc/go_templates/source_network.t diff --git a/package/etc/syslog-ng.conf b/package/etc/syslog-ng.conf index e4a0a78..d0422fb 100644 --- a/package/etc/syslog-ng.conf +++ b/package/etc/syslog-ng.conf @@ -62,8 +62,7 @@ options { @include "conf.d/destinations/*.conf" @include "conf.d/log_paths/*.conf" -@include "conf.d/local/filters/*.conf" -@include "conf.d/local/filters/*/*.conf" -@include "conf.d/local/sources/*.conf" -@include "conf.d/local/destinations/*.conf" -@include "conf.d/local/log_paths/*.conf" +@include "conf.d/local/config/filters/*.conf" +@include "conf.d/local/config/sources/*.conf" +@include "conf.d/local/config/destinations/*.conf" +@include "conf.d/local/config/log_paths/*.conf" diff --git a/package/sbin/entrypoint.sh b/package/sbin/entrypoint.sh index d100fc0..d93862b 100755 --- a/package/sbin/entrypoint.sh +++ b/package/sbin/entrypoint.sh @@ -7,7 +7,7 @@ do echo Templating conf for $d gomplate \ --input-dir=$d \ - --template t=etc/templates/ \ + --template t=etc/go_templates/ \ --exclude=*.conf --exclude=*.csv --exclude=*.t --exclude=.*\ --output-map="$d/{{ .in | strings.ReplaceAll \".conf.tmpl\" \".conf\" }}" done From d926c345711a067803a753d6556c00c96b15cb04 Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Sun, 6 Oct 2019 08:42:51 -0700 Subject: [PATCH 03/67] Update .gitignore --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 3ac4e78..87b76b1 100644 --- a/.gitignore +++ b/.gitignore @@ -382,4 +382,4 @@ fabric.properties /.idea/ tests/test_plugin_*.py -package/etc/conf.d/local/ \ No newline at end of file +# package/etc/conf.d/local/ From 2f4b8ebc7356b462f051568ceaa60c401e3b1a0d Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Sun, 6 Oct 2019 10:59:39 -0700 Subject: [PATCH 04/67] Add local directory --- .../local/config/destinations/README.md | 1 + .../etc/conf.d/local/config/filters/README.md | 1 + .../conf.d/local/config/filters/example.conf | 4 + .../conf.d/local/config/log_paths/README.md | 1 + .../local/config/log_paths/example.conf.tmpl | 76 +++++++++++++++++++ .../etc/conf.d/local/config/sources/README.md | 1 + .../context/compliance_meta_by_source.conf | 5 ++ .../context/compliance_meta_by_source.csv | 2 + .../etc/conf.d/local/context/splunk_index.csv | 40 ++++++++++ .../context/vendor_product_by_source.conf | 37 +++++++++ .../context/vendor_product_by_source.csv | 8 ++ 11 files changed, 176 insertions(+) create mode 100644 package/etc/conf.d/local/config/destinations/README.md create mode 100644 package/etc/conf.d/local/config/filters/README.md create mode 100644 package/etc/conf.d/local/config/filters/example.conf create mode 100644 package/etc/conf.d/local/config/log_paths/README.md create mode 100644 package/etc/conf.d/local/config/log_paths/example.conf.tmpl create mode 100644 package/etc/conf.d/local/config/sources/README.md create mode 100644 package/etc/conf.d/local/context/compliance_meta_by_source.conf create mode 100644 package/etc/conf.d/local/context/compliance_meta_by_source.csv create mode 100644 package/etc/conf.d/local/context/splunk_index.csv create mode 100644 package/etc/conf.d/local/context/vendor_product_by_source.conf create mode 100644 package/etc/conf.d/local/context/vendor_product_by_source.csv diff --git a/package/etc/conf.d/local/config/destinations/README.md b/package/etc/conf.d/local/config/destinations/README.md new file mode 100644 index 0000000..ee6571d --- /dev/null +++ b/package/etc/conf.d/local/config/destinations/README.md @@ -0,0 +1 @@ +This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/conf.d/local/config/filters/README.md b/package/etc/conf.d/local/config/filters/README.md new file mode 100644 index 0000000..ee6571d --- /dev/null +++ b/package/etc/conf.d/local/config/filters/README.md @@ -0,0 +1 @@ +This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/conf.d/local/config/filters/example.conf b/package/etc/conf.d/local/config/filters/example.conf new file mode 100644 index 0000000..047fdc8 --- /dev/null +++ b/package/etc/conf.d/local/config/filters/example.conf @@ -0,0 +1,4 @@ + +filter f_local_example { + program(sc4splugin); +}; \ No newline at end of file diff --git a/package/etc/conf.d/local/config/log_paths/README.md b/package/etc/conf.d/local/config/log_paths/README.md new file mode 100644 index 0000000..ee6571d --- /dev/null +++ b/package/etc/conf.d/local/config/log_paths/README.md @@ -0,0 +1 @@ +This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/conf.d/local/config/log_paths/example.conf.tmpl b/package/etc/conf.d/local/config/log_paths/example.conf.tmpl new file mode 100644 index 0000000..a8ac264 --- /dev/null +++ b/package/etc/conf.d/local/config/log_paths/example.conf.tmpl @@ -0,0 +1,76 @@ +# LOCAL_EXAMPLE + +# When creating a real plugin, replace the upper case text "LOCAL_EXAMPLE" throughout this file with a unique +# string to identify the vendor product. The string should be of the form "VENDOR_PRODUCT" to signify the +# manufacturer and product type, and must contain only characters matching this regex: [A-Z\_]+ + +# If any of the "LOCAL_EXAMPLE" variables passed into the environment are set (e.g. TLS, UDP, or TLS), +# the template generator will build a custom source based on the value of one or more of the set variables. + +{{- if (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_TCP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_UDP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_TLS_PORT") "no") "no") }} + +# "port_id" is used to generate the port variable to be used. It should match the "core" of the variable name +# set in the line above. For example, the "port_id" of "SC4S_LISTEN_LOCAL_EXAMPLE_TCP_PORT" is "LOCAL_EXAMPLE". +# "parser" can be customized on dedicated ports only +# "common" uses the same parser sequence as the default ports and is the most commonly used + +{{ $context := dict "port_id" "LOCAL_EXAMPLE" "parser" "common"}} + +# The following template execution creates a syslog-ng source with one or more dedicated ports for use with this log_path +# The ports used are based on the values of one or more of the environment variables set above. + +{{ tmpl.Exec "t/source_network.t" $context }} +{{- end -}} +{{ define "log_path" }} +log { + +# The first time this template is used the log_path will be linked to the default port + +{{- if eq (.) "yes"}} + source(s_default-ports); + +# Filters should be updated to use the simplest and most effecient logic possible to discard +# the message from this path + + filter(f_is_rfc3164); + filter(f_local_example); +{{- end}} +{{- if eq (.) "no"}} + +# In the second pass through the template a link to the dedicated port is used. This +# normally does not require additional filters + +source (s_dedicated_port_LOCAL_EXAMPLE); +{{- end}} + +#Set a default sourcetype and index + + rewrite { r_set_splunk_dest_default(sourcetype("sc4s:local_example"), index("main"), template("t_msg_only"))}; + +#using the key "local_example" find any cutomized index,source or sourcetype meta values + + parser {p_add_context_splunk(key("local_example")); }; + +# Any additional logic needed to process the event before sending to Splunk goes here + +# Send it to Splunk + + destination(d_hec); #--HEC-- + +# Note: We normally do not use the "final" flag; this will allow another plugin to be created that will +# forward events to another system + + flags(flow-control); + +}; +{{- end}} +{{- if (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_TCP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_UDP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TLS_PORT") "no") "no") }} + +# Listen on the specified dedicated port(s) for LOCAL_EXAMPLE traffic + + {{tmpl.Exec "log_path" "no" }} +{{- end}} + +# Listen on the default port (typically 514) for LOCAL_EXAMPLE traffic + +{{tmpl.Exec "log_path" "yes" }} diff --git a/package/etc/conf.d/local/config/sources/README.md b/package/etc/conf.d/local/config/sources/README.md new file mode 100644 index 0000000..ee6571d --- /dev/null +++ b/package/etc/conf.d/local/config/sources/README.md @@ -0,0 +1 @@ +This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/conf.d/local/context/compliance_meta_by_source.conf b/package/etc/conf.d/local/context/compliance_meta_by_source.conf new file mode 100644 index 0000000..1d5acae --- /dev/null +++ b/package/etc/conf.d/local/context/compliance_meta_by_source.conf @@ -0,0 +1,5 @@ +@version: 3.23 +filter f_test_test { + host("something-*" type(glob)) or + netmask(192.168.100.1/24) +}; diff --git a/package/etc/conf.d/local/context/compliance_meta_by_source.csv b/package/etc/conf.d/local/context/compliance_meta_by_source.csv new file mode 100644 index 0000000..6608db0 --- /dev/null +++ b/package/etc/conf.d/local/context/compliance_meta_by_source.csv @@ -0,0 +1,2 @@ +#f_test_test,.splunk.index,"badindex" +#f_test_test,fields.compliance,"pci" diff --git a/package/etc/conf.d/local/context/splunk_index.csv b/package/etc/conf.d/local/context/splunk_index.csv new file mode 100644 index 0000000..e93911a --- /dev/null +++ b/package/etc/conf.d/local/context/splunk_index.csv @@ -0,0 +1,40 @@ +#bluecoat_proxy,index,netproxy +#cef_ArcSight_ArcSight,index,netwaf +#cef_Incapsula_SIEMintegration,index,netwaf +#cef_Microsoft_Microsoft Windows,index,oswinsec +#cef_Microsoft_System or Application Event,index,oswin +#cisco_asa,index,netfw +#cisco_ios,index,netops +#cisco_nx_os,index,netops +#local_example,index,main +#fortinet_fortios_event,index,netops +#fortinet_fortios_log,index,netops +#fortinet_fortios_traffic,index,netfw +#fortinet_fortios_utm,index,netids +#juniper_idp,index,netids +#juniper_structured,index,netops +#juniper_idp_structured,index,netids +#juniper_junos_fw_structured,index,netfw +#juniper_junos_ids_structured,index,netids +#juniper_junos_utm_structured,index,netfw +#juniper_junos_fw,index,netfw +#juniper_junos_ids,index,netids +#juniper_junos_utm,index,netfw +#juniper_sslvpn,index,netfw +#juniper_netscreen,index,netfw +#juniper_nsm,index,netfw +#juniper_nsm_idp,index,netids +#juniper_legacy,index,netops +#pan_traffic,index,netfw +#pan_threat,index,netproxy +#pan_system,index,netops +#pan_config,index,netops +#pan_hipwatch,index,main +#pan_correlation,index,main +#pan_userid,index,netauth +#pan_unknown,index,netops +#proofpoint_pps_filter,index,email +#proofpoint_pps_sendmail,index,email +#sc4s_events,index,main +#sc4s_fallback,index,main +#sc4s_metrics,index,em_metrics diff --git a/package/etc/conf.d/local/context/vendor_product_by_source.conf b/package/etc/conf.d/local/context/vendor_product_by_source.conf new file mode 100644 index 0000000..37e3412 --- /dev/null +++ b/package/etc/conf.d/local/context/vendor_product_by_source.conf @@ -0,0 +1,37 @@ +@version: 3.22 +#TODO: #60 The syntax below uses regex and an indirect reference to a variable due to a +#bug/limitation of selector files. The better syntax should be as follows +#filter {match("f5_test" template("$(env PRESUME_SYSLOG)")); }; + +filter f_test_test { + host("testvp-*" type(glob)) or + netmask(192.168.100.1/24) +}; +filter f_juniper_nsm { + host("jnpnsm-*" type(glob)) or + netmask(192.168.1.0/24) +}; +filter f_juniper_nsm_idp { + host("jnpnsmidp-*" type(glob)) or + netmask(192.168.2.0/24) +}; +filter f_juniper_idp { + host("jnpidp-*" type(glob)) or + netmask(192.168.3.0/24) +}; +filter f_juniper_netscreen { + host("jnpns-*" type(glob)) or + netmask(192.168.4.0/24) +}; +filter f_cisco_nx_os { + host("csconx-*" type(glob)) or + netmask(192.168.5.0/24) +}; +filter f_proofpoint_pps_sendmail { + host("pps-*" type(glob)) or + netmask(192.168.6.0/24) +}; +filter f_proofpoint_pps_filter { + host("pps-*" type(glob)) or + netmask(192.168.7.0/24) +}; \ No newline at end of file diff --git a/package/etc/conf.d/local/context/vendor_product_by_source.csv b/package/etc/conf.d/local/context/vendor_product_by_source.csv new file mode 100644 index 0000000..3f90603 --- /dev/null +++ b/package/etc/conf.d/local/context/vendor_product_by_source.csv @@ -0,0 +1,8 @@ +f_test_test,sc4s_vendor_product,"test_test" +f_juniper_nsm,sc4s_vendor_product,"juniper_nsm" +f_juniper_nsm_idp,sc4s_vendor_product,"juniper_nsm_idp" +f_juniper_idp,sc4s_vendor_product,"juniper_idp" +f_juniper_netscreen,sc4s_vendor_product,"juniper_netscreen" +f_cisco_nx_os,sc4s_vendor_product,"cisco_nx_os" +f_proofpoint_pps_sendmail,sc4s_vendor_product,"proofpoint_pps_sendmail" +f_proofpoint_pps_filter,sc4s_vendor_product,"proofpoint_pps_filter" \ No newline at end of file From 63d69a91af586823b3b0fedf160b9db62efdaee5 Mon Sep 17 00:00:00 2001 From: ryan faircloth Date: Sun, 6 Oct 2019 16:18:20 -0400 Subject: [PATCH 05/67] seed the context directory in local if it does not exist --- .env.template | 6 +++--- package/Dockerfile | 5 +++-- .../compliance_meta_by_source.conf | 0 .../compliance_meta_by_source.csv | 0 .../local/context => context_templates}/splunk_index.csv | 0 .../vendor_product_by_source.conf | 0 .../vendor_product_by_source.csv | 0 package/sbin/entrypoint.sh | 4 +++- 8 files changed, 9 insertions(+), 6 deletions(-) rename package/etc/{conf.d/local/context => context_templates}/compliance_meta_by_source.conf (100%) rename package/etc/{conf.d/local/context => context_templates}/compliance_meta_by_source.csv (100%) rename package/etc/{conf.d/local/context => context_templates}/splunk_index.csv (100%) rename package/etc/{conf.d/local/context => context_templates}/vendor_product_by_source.conf (100%) rename package/etc/{conf.d/local/context => context_templates}/vendor_product_by_source.csv (100%) diff --git a/.env.template b/.env.template index 8d0cafa..c1f7af2 100644 --- a/.env.template +++ b/.env.template @@ -16,6 +16,6 @@ SPLUNK_HEC_STATSURL=https://splunk:8088/services/collector/event SPLUNK_CONNECT_METHOD=hec SPLUNK_DEFAULT_INDEX=main SPLUNK_METRICS_INDEX=em_metrics -SPLUNK_APPS_URL=https://splunkbase.splunk.com/app/2757/release/6.1.1/download,https://splunkbase.splunk.com/app/3245/release/1.0/download,https://splunkbase.splunk.com/app/1620/release/3.4.0/download,https://splunkbase.splunk.com/app/1467/release/2.5.8/download,https://splunkbase.splunk.com/app/2846/release/1.6.0/download,https://splunkbase.splunk.com/app/2847/release/1.2.0/download -SPLUNKBASE_USERNAME=username -SPLUNKBASE_PASSWORD=password +#SPLUNK_APPS_URL=https://splunkbase.splunk.com/app/2757/release/6.1.1/download,https://splunkbase.splunk.com/app/3245/release/1.0/download,https://splunkbase.splunk.com/app/1620/release/3.4.0/download,https://splunkbase.splunk.com/app/1467/release/2.5.8/download,https://splunkbase.splunk.com/app/2846/release/1.6.0/download,https://splunkbase.splunk.com/app/2847/release/1.2.0/download +#SPLUNKBASE_USERNAME=username +#SPLUNKBASE_PASSWORD=password diff --git a/package/Dockerfile b/package/Dockerfile index 570a16a..a0c05f4 100644 --- a/package/Dockerfile +++ b/package/Dockerfile @@ -87,11 +87,12 @@ RUN curl -o /usr/local/bin/gomplate -sSL https://github.com/hairyhenderson/gompl COPY etc/syslog-ng.conf /opt/syslog-ng/etc/syslog-ng.conf COPY etc/conf.d /opt/syslog-ng/etc/conf.d COPY etc/go_templates /opt/syslog-ng/etc/go_templates +COPY etc/context_templates /opt/syslog-ng/etc/context_templates + COPY etc/conf.d/local/context /opt/syslog-ng/etc/conf.d/local/context COPY sbin/entrypoint.sh / RUN mkdir -p /opt/syslog-ng/var/data/disk-buffer RUN source scl_source enable rh-python36 ;/opt/syslog-ng/sbin/syslog-ng -V -#RUN source scl_source enable rh-python36 ;/opt/syslog-ng/sbin/syslog-ng -t EXPOSE 514 EXPOSE 601/tcp @@ -101,4 +102,4 @@ ENV SPLUNK_CONNECT_METHOD=UF ENTRYPOINT ["/entrypoint.sh", "-F"] -HEALTHCHECK --interval=1s --timeout=6s CMD source scl_source enable rh-python36 ;goss -g /etc/goss.yaml validate +HEALTHCHECK --interval=1s --timeout=6s CMD source scl_source enable rh-python36 ;goss -g /etc \ No newline at end of file diff --git a/package/etc/conf.d/local/context/compliance_meta_by_source.conf b/package/etc/context_templates/compliance_meta_by_source.conf similarity index 100% rename from package/etc/conf.d/local/context/compliance_meta_by_source.conf rename to package/etc/context_templates/compliance_meta_by_source.conf diff --git a/package/etc/conf.d/local/context/compliance_meta_by_source.csv b/package/etc/context_templates/compliance_meta_by_source.csv similarity index 100% rename from package/etc/conf.d/local/context/compliance_meta_by_source.csv rename to package/etc/context_templates/compliance_meta_by_source.csv diff --git a/package/etc/conf.d/local/context/splunk_index.csv b/package/etc/context_templates/splunk_index.csv similarity index 100% rename from package/etc/conf.d/local/context/splunk_index.csv rename to package/etc/context_templates/splunk_index.csv diff --git a/package/etc/conf.d/local/context/vendor_product_by_source.conf b/package/etc/context_templates/vendor_product_by_source.conf similarity index 100% rename from package/etc/conf.d/local/context/vendor_product_by_source.conf rename to package/etc/context_templates/vendor_product_by_source.conf diff --git a/package/etc/conf.d/local/context/vendor_product_by_source.csv b/package/etc/context_templates/vendor_product_by_source.csv similarity index 100% rename from package/etc/conf.d/local/context/vendor_product_by_source.csv rename to package/etc/context_templates/vendor_product_by_source.csv diff --git a/package/sbin/entrypoint.sh b/package/sbin/entrypoint.sh index d93862b..0af0742 100755 --- a/package/sbin/entrypoint.sh +++ b/package/sbin/entrypoint.sh @@ -12,5 +12,7 @@ do --output-map="$d/{{ .in | strings.ReplaceAll \".conf.tmpl\" \".conf\" }}" done -echo syslog-ng started +cp -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context + +echo syslog-ng starting exec /opt/syslog-ng/sbin/syslog-ng $@ \ No newline at end of file From 49c014a65efb11e76c0404dd67f692f036c2f432 Mon Sep 17 00:00:00 2001 From: ryan faircloth Date: Sun, 6 Oct 2019 16:38:49 -0400 Subject: [PATCH 06/67] Update Dockerfile --- package/Dockerfile | 1 - 1 file changed, 1 deletion(-) diff --git a/package/Dockerfile b/package/Dockerfile index a0c05f4..91da569 100644 --- a/package/Dockerfile +++ b/package/Dockerfile @@ -89,7 +89,6 @@ COPY etc/conf.d /opt/syslog-ng/etc/conf.d COPY etc/go_templates /opt/syslog-ng/etc/go_templates COPY etc/context_templates /opt/syslog-ng/etc/context_templates -COPY etc/conf.d/local/context /opt/syslog-ng/etc/conf.d/local/context COPY sbin/entrypoint.sh / RUN mkdir -p /opt/syslog-ng/var/data/disk-buffer RUN source scl_source enable rh-python36 ;/opt/syslog-ng/sbin/syslog-ng -V From db72d9f00475f1eddc04002376081fa90ec748fb Mon Sep 17 00:00:00 2001 From: ryan faircloth Date: Sun, 6 Oct 2019 16:47:01 -0400 Subject: [PATCH 07/67] Update entrypoint.sh --- package/sbin/entrypoint.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package/sbin/entrypoint.sh b/package/sbin/entrypoint.sh index 0af0742..5b7d6a1 100755 --- a/package/sbin/entrypoint.sh +++ b/package/sbin/entrypoint.sh @@ -12,7 +12,7 @@ do --output-map="$d/{{ .in | strings.ReplaceAll \".conf.tmpl\" \".conf\" }}" done -cp -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context +cp --verbose -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context/ echo syslog-ng starting exec /opt/syslog-ng/sbin/syslog-ng $@ \ No newline at end of file From 694ae594c05572efbd75a5763e4e85c0ded0b59d Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Sun, 6 Oct 2019 16:19:30 -0700 Subject: [PATCH 08/67] entrypoint.sh add mkdir Add mkdir for subdirectories of the local directory to prepare for copy. --- package/sbin/entrypoint.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/package/sbin/entrypoint.sh b/package/sbin/entrypoint.sh index 5b7d6a1..13a2464 100755 --- a/package/sbin/entrypoint.sh +++ b/package/sbin/entrypoint.sh @@ -12,6 +12,8 @@ do --output-map="$d/{{ .in | strings.ReplaceAll \".conf.tmpl\" \".conf\" }}" done +mkdir -p /opt/syslog-ng/etc/conf.d/local/context/ +mkdir -p /opt/syslog-ng/etc/conf.d/local/config/ cp --verbose -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context/ echo syslog-ng starting From 281266544d162a16d77daf2c86cf8b05dae1af4e Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Sun, 6 Oct 2019 17:22:01 -0700 Subject: [PATCH 09/67] Update entrypoint.sh to copy config local dir --- .../etc/local_config/destinations/README.md | 1 + package/etc/local_config/filters/README.md | 1 + package/etc/local_config/filters/example.conf | 4 + package/etc/local_config/log_paths/README.md | 1 + .../local_config/log_paths/example.conf.tmpl | 76 +++++++++++++++++++ package/etc/local_config/sources/README.md | 1 + package/sbin/entrypoint.sh | 1 + 7 files changed, 85 insertions(+) create mode 100644 package/etc/local_config/destinations/README.md create mode 100644 package/etc/local_config/filters/README.md create mode 100644 package/etc/local_config/filters/example.conf create mode 100644 package/etc/local_config/log_paths/README.md create mode 100644 package/etc/local_config/log_paths/example.conf.tmpl create mode 100644 package/etc/local_config/sources/README.md diff --git a/package/etc/local_config/destinations/README.md b/package/etc/local_config/destinations/README.md new file mode 100644 index 0000000..ee6571d --- /dev/null +++ b/package/etc/local_config/destinations/README.md @@ -0,0 +1 @@ +This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/local_config/filters/README.md b/package/etc/local_config/filters/README.md new file mode 100644 index 0000000..ee6571d --- /dev/null +++ b/package/etc/local_config/filters/README.md @@ -0,0 +1 @@ +This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/local_config/filters/example.conf b/package/etc/local_config/filters/example.conf new file mode 100644 index 0000000..047fdc8 --- /dev/null +++ b/package/etc/local_config/filters/example.conf @@ -0,0 +1,4 @@ + +filter f_local_example { + program(sc4splugin); +}; \ No newline at end of file diff --git a/package/etc/local_config/log_paths/README.md b/package/etc/local_config/log_paths/README.md new file mode 100644 index 0000000..ee6571d --- /dev/null +++ b/package/etc/local_config/log_paths/README.md @@ -0,0 +1 @@ +This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/etc/local_config/log_paths/example.conf.tmpl b/package/etc/local_config/log_paths/example.conf.tmpl new file mode 100644 index 0000000..a8ac264 --- /dev/null +++ b/package/etc/local_config/log_paths/example.conf.tmpl @@ -0,0 +1,76 @@ +# LOCAL_EXAMPLE + +# When creating a real plugin, replace the upper case text "LOCAL_EXAMPLE" throughout this file with a unique +# string to identify the vendor product. The string should be of the form "VENDOR_PRODUCT" to signify the +# manufacturer and product type, and must contain only characters matching this regex: [A-Z\_]+ + +# If any of the "LOCAL_EXAMPLE" variables passed into the environment are set (e.g. TLS, UDP, or TLS), +# the template generator will build a custom source based on the value of one or more of the set variables. + +{{- if (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_TCP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_UDP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_TLS_PORT") "no") "no") }} + +# "port_id" is used to generate the port variable to be used. It should match the "core" of the variable name +# set in the line above. For example, the "port_id" of "SC4S_LISTEN_LOCAL_EXAMPLE_TCP_PORT" is "LOCAL_EXAMPLE". +# "parser" can be customized on dedicated ports only +# "common" uses the same parser sequence as the default ports and is the most commonly used + +{{ $context := dict "port_id" "LOCAL_EXAMPLE" "parser" "common"}} + +# The following template execution creates a syslog-ng source with one or more dedicated ports for use with this log_path +# The ports used are based on the values of one or more of the environment variables set above. + +{{ tmpl.Exec "t/source_network.t" $context }} +{{- end -}} +{{ define "log_path" }} +log { + +# The first time this template is used the log_path will be linked to the default port + +{{- if eq (.) "yes"}} + source(s_default-ports); + +# Filters should be updated to use the simplest and most effecient logic possible to discard +# the message from this path + + filter(f_is_rfc3164); + filter(f_local_example); +{{- end}} +{{- if eq (.) "no"}} + +# In the second pass through the template a link to the dedicated port is used. This +# normally does not require additional filters + +source (s_dedicated_port_LOCAL_EXAMPLE); +{{- end}} + +#Set a default sourcetype and index + + rewrite { r_set_splunk_dest_default(sourcetype("sc4s:local_example"), index("main"), template("t_msg_only"))}; + +#using the key "local_example" find any cutomized index,source or sourcetype meta values + + parser {p_add_context_splunk(key("local_example")); }; + +# Any additional logic needed to process the event before sending to Splunk goes here + +# Send it to Splunk + + destination(d_hec); #--HEC-- + +# Note: We normally do not use the "final" flag; this will allow another plugin to be created that will +# forward events to another system + + flags(flow-control); + +}; +{{- end}} +{{- if (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_TCP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_LOCAL_EXAMPLE_UDP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_MICROFOCUS_ARCSIGHT_TLS_PORT") "no") "no") }} + +# Listen on the specified dedicated port(s) for LOCAL_EXAMPLE traffic + + {{tmpl.Exec "log_path" "no" }} +{{- end}} + +# Listen on the default port (typically 514) for LOCAL_EXAMPLE traffic + +{{tmpl.Exec "log_path" "yes" }} diff --git a/package/etc/local_config/sources/README.md b/package/etc/local_config/sources/README.md new file mode 100644 index 0000000..ee6571d --- /dev/null +++ b/package/etc/local_config/sources/README.md @@ -0,0 +1 @@ +This file exists to preserve the path for plugin use \ No newline at end of file diff --git a/package/sbin/entrypoint.sh b/package/sbin/entrypoint.sh index 13a2464..30537c7 100755 --- a/package/sbin/entrypoint.sh +++ b/package/sbin/entrypoint.sh @@ -15,6 +15,7 @@ done mkdir -p /opt/syslog-ng/etc/conf.d/local/context/ mkdir -p /opt/syslog-ng/etc/conf.d/local/config/ cp --verbose -n /opt/syslog-ng/etc/context_templates/* /opt/syslog-ng/etc/conf.d/local/context/ +cp --verbose -R -n /opt/syslog-ng/etc/local_config/* /opt/syslog-ng/etc/conf.d/local/config/ echo syslog-ng starting exec /opt/syslog-ng/sbin/syslog-ng $@ \ No newline at end of file From 2d5c178c6c19faf72106930019477425b139c105 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 20:33:59 -0400 Subject: [PATCH 10/67] Update Dockerfile --- package/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package/Dockerfile b/package/Dockerfile index 91da569..92dd2cf 100644 --- a/package/Dockerfile +++ b/package/Dockerfile @@ -101,4 +101,4 @@ ENV SPLUNK_CONNECT_METHOD=UF ENTRYPOINT ["/entrypoint.sh", "-F"] -HEALTHCHECK --interval=1s --timeout=6s CMD source scl_source enable rh-python36 ;goss -g /etc \ No newline at end of file +HEALTHCHECK --interval=1s --timeout=6s CMD source scl_source enable rh-python36 ;goss -g /etc/goss.yaml validate \ No newline at end of file From 67faacfee8c52a83df233a6b54f890478fa5e8a1 Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Sun, 6 Oct 2019 17:47:36 -0700 Subject: [PATCH 11/67] Update Dockerfile to include local_config tree --- package/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/package/Dockerfile b/package/Dockerfile index 91da569..74f7bb8 100644 --- a/package/Dockerfile +++ b/package/Dockerfile @@ -88,6 +88,7 @@ COPY etc/syslog-ng.conf /opt/syslog-ng/etc/syslog-ng.conf COPY etc/conf.d /opt/syslog-ng/etc/conf.d COPY etc/go_templates /opt/syslog-ng/etc/go_templates COPY etc/context_templates /opt/syslog-ng/etc/context_templates +COPY etc/local_config /opt/syslog-ng/etc/local_config COPY sbin/entrypoint.sh / RUN mkdir -p /opt/syslog-ng/var/data/disk-buffer From ed700e4bbf13529183f9df55f623eae2aede3981 Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Sun, 6 Oct 2019 18:06:17 -0700 Subject: [PATCH 12/67] gitignore -- ignore local mount directory --- .gitignore | 1 + .../context/compliance_meta_by_source.conf | 5 +++ .../context/compliance_meta_by_source.csv | 2 + .../etc/conf.d/local/context/splunk_index.csv | 40 +++++++++++++++++++ .../context/vendor_product_by_source.conf | 37 +++++++++++++++++ .../context/vendor_product_by_source.csv | 8 ++++ 6 files changed, 93 insertions(+) create mode 100644 package/etc/conf.d/local/context/compliance_meta_by_source.conf create mode 100644 package/etc/conf.d/local/context/compliance_meta_by_source.csv create mode 100644 package/etc/conf.d/local/context/splunk_index.csv create mode 100644 package/etc/conf.d/local/context/vendor_product_by_source.conf create mode 100644 package/etc/conf.d/local/context/vendor_product_by_source.csv diff --git a/.gitignore b/.gitignore index 87b76b1..4d40b52 100644 --- a/.gitignore +++ b/.gitignore @@ -383,3 +383,4 @@ fabric.properties tests/test_plugin_*.py # package/etc/conf.d/local/ +!package/etc/conf.d/local diff --git a/package/etc/conf.d/local/context/compliance_meta_by_source.conf b/package/etc/conf.d/local/context/compliance_meta_by_source.conf new file mode 100644 index 0000000..1d5acae --- /dev/null +++ b/package/etc/conf.d/local/context/compliance_meta_by_source.conf @@ -0,0 +1,5 @@ +@version: 3.23 +filter f_test_test { + host("something-*" type(glob)) or + netmask(192.168.100.1/24) +}; diff --git a/package/etc/conf.d/local/context/compliance_meta_by_source.csv b/package/etc/conf.d/local/context/compliance_meta_by_source.csv new file mode 100644 index 0000000..6608db0 --- /dev/null +++ b/package/etc/conf.d/local/context/compliance_meta_by_source.csv @@ -0,0 +1,2 @@ +#f_test_test,.splunk.index,"badindex" +#f_test_test,fields.compliance,"pci" diff --git a/package/etc/conf.d/local/context/splunk_index.csv b/package/etc/conf.d/local/context/splunk_index.csv new file mode 100644 index 0000000..e93911a --- /dev/null +++ b/package/etc/conf.d/local/context/splunk_index.csv @@ -0,0 +1,40 @@ +#bluecoat_proxy,index,netproxy +#cef_ArcSight_ArcSight,index,netwaf +#cef_Incapsula_SIEMintegration,index,netwaf +#cef_Microsoft_Microsoft Windows,index,oswinsec +#cef_Microsoft_System or Application Event,index,oswin +#cisco_asa,index,netfw +#cisco_ios,index,netops +#cisco_nx_os,index,netops +#local_example,index,main +#fortinet_fortios_event,index,netops +#fortinet_fortios_log,index,netops +#fortinet_fortios_traffic,index,netfw +#fortinet_fortios_utm,index,netids +#juniper_idp,index,netids +#juniper_structured,index,netops +#juniper_idp_structured,index,netids +#juniper_junos_fw_structured,index,netfw +#juniper_junos_ids_structured,index,netids +#juniper_junos_utm_structured,index,netfw +#juniper_junos_fw,index,netfw +#juniper_junos_ids,index,netids +#juniper_junos_utm,index,netfw +#juniper_sslvpn,index,netfw +#juniper_netscreen,index,netfw +#juniper_nsm,index,netfw +#juniper_nsm_idp,index,netids +#juniper_legacy,index,netops +#pan_traffic,index,netfw +#pan_threat,index,netproxy +#pan_system,index,netops +#pan_config,index,netops +#pan_hipwatch,index,main +#pan_correlation,index,main +#pan_userid,index,netauth +#pan_unknown,index,netops +#proofpoint_pps_filter,index,email +#proofpoint_pps_sendmail,index,email +#sc4s_events,index,main +#sc4s_fallback,index,main +#sc4s_metrics,index,em_metrics diff --git a/package/etc/conf.d/local/context/vendor_product_by_source.conf b/package/etc/conf.d/local/context/vendor_product_by_source.conf new file mode 100644 index 0000000..37e3412 --- /dev/null +++ b/package/etc/conf.d/local/context/vendor_product_by_source.conf @@ -0,0 +1,37 @@ +@version: 3.22 +#TODO: #60 The syntax below uses regex and an indirect reference to a variable due to a +#bug/limitation of selector files. The better syntax should be as follows +#filter {match("f5_test" template("$(env PRESUME_SYSLOG)")); }; + +filter f_test_test { + host("testvp-*" type(glob)) or + netmask(192.168.100.1/24) +}; +filter f_juniper_nsm { + host("jnpnsm-*" type(glob)) or + netmask(192.168.1.0/24) +}; +filter f_juniper_nsm_idp { + host("jnpnsmidp-*" type(glob)) or + netmask(192.168.2.0/24) +}; +filter f_juniper_idp { + host("jnpidp-*" type(glob)) or + netmask(192.168.3.0/24) +}; +filter f_juniper_netscreen { + host("jnpns-*" type(glob)) or + netmask(192.168.4.0/24) +}; +filter f_cisco_nx_os { + host("csconx-*" type(glob)) or + netmask(192.168.5.0/24) +}; +filter f_proofpoint_pps_sendmail { + host("pps-*" type(glob)) or + netmask(192.168.6.0/24) +}; +filter f_proofpoint_pps_filter { + host("pps-*" type(glob)) or + netmask(192.168.7.0/24) +}; \ No newline at end of file diff --git a/package/etc/conf.d/local/context/vendor_product_by_source.csv b/package/etc/conf.d/local/context/vendor_product_by_source.csv new file mode 100644 index 0000000..3f90603 --- /dev/null +++ b/package/etc/conf.d/local/context/vendor_product_by_source.csv @@ -0,0 +1,8 @@ +f_test_test,sc4s_vendor_product,"test_test" +f_juniper_nsm,sc4s_vendor_product,"juniper_nsm" +f_juniper_nsm_idp,sc4s_vendor_product,"juniper_nsm_idp" +f_juniper_idp,sc4s_vendor_product,"juniper_idp" +f_juniper_netscreen,sc4s_vendor_product,"juniper_netscreen" +f_cisco_nx_os,sc4s_vendor_product,"cisco_nx_os" +f_proofpoint_pps_sendmail,sc4s_vendor_product,"proofpoint_pps_sendmail" +f_proofpoint_pps_filter,sc4s_vendor_product,"proofpoint_pps_filter" \ No newline at end of file From 77f2eacbbb589820c646433bdb989ed37525a4b1 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 21:24:56 -0400 Subject: [PATCH 13/67] build on tag version --- .circleci/config.yml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 8d7ddba..f154295 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -295,10 +295,9 @@ workflows: - dgoss - test-unit filters: - branches: - only: - - master - - /^release\/.*/ + filters: + tags: + only: /([0-9]+\.[0-9]+\.[0-9]+) - publish-latest: requires: - dgoss From 667d8854f9eae64bad081c9b0085ffc0af3467ef Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 21:35:56 -0400 Subject: [PATCH 14/67] Update config.yml --- .circleci/config.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index f154295..3a72924 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -295,9 +295,8 @@ workflows: - dgoss - test-unit filters: - filters: - tags: - only: /([0-9]+\.[0-9]+\.[0-9]+) + tags: + only: /([0-9]+\.[0-9]+\.[0-9]+) - publish-latest: requires: - dgoss From 4833b8e4f409f8bb4774279cd179fe6067afe988 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 21:51:32 -0400 Subject: [PATCH 15/67] Update config.yml --- .circleci/config.yml | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index 3a72924..a519ef4 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -238,6 +238,38 @@ jobs: command: | SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) docker push splunk/scs:$SEMVER + release-version: + machine: + image: ubuntu-1604:201903-01 + docker_layer_caching: true # default - false + environment: + IMAGE_NAME: rfaircloth/scs + steps: + - checkout + + - run: + name: Docker Login + command: docker login -u $DOCKER_USER -p $DOCKER_PASS + - run: + name: Docker pull + command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 + - run: + name: Docker tag image + command: | + SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + docker tag $IMAGE_NAME:$CIRCLE_SHA1 splunk/scs:$SEMVER + - run: + name: Docker Save + command: | + mkdir artifacts + docker save splunk/scs:$SEMVER > artifacts/scs.tar + - run: + name: "Publish Release on GitHub" + command: | + go get github.com/tcnksm/ghr + VERSION=$(my-binary --version) + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${VERSION} ./artifacts/ + publish-latest: environment: IMAGE_NAME: rfaircloth/scs From 7189f1214a39296e991d34f939bce32a18e63f49 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 21:52:35 -0400 Subject: [PATCH 16/67] Update config.yml --- .circleci/config.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index a519ef4..d55244b 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -329,6 +329,9 @@ workflows: filters: tags: only: /([0-9]+\.[0-9]+\.[0-9]+) + - release-version: + requires: + - publish-version - publish-latest: requires: - dgoss From f475f5d14e9cd1288df19b874a93fdb9e2183dc5 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 22:17:01 -0400 Subject: [PATCH 17/67] Update config.yml --- .circleci/config.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index d55244b..e8b6596 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -262,7 +262,8 @@ jobs: name: Docker Save command: | mkdir artifacts - docker save splunk/scs:$SEMVER > artifacts/scs.tar + SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + docker save splunk/scs:$SEMVER | gzip -c > artifacts/sc4s_container.tar.gz - run: name: "Publish Release on GitHub" command: | From 0680e34890c88f8a34f7d7aa374e16597c6089da Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 22:32:56 -0400 Subject: [PATCH 18/67] Update config.yml --- .circleci/config.yml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index e8b6596..27fd314 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -264,12 +264,17 @@ jobs: mkdir artifacts SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) docker save splunk/scs:$SEMVER | gzip -c > artifacts/sc4s_container.tar.gz + - run: + name: BYOE Config + command: | + tar rvf artifacts/baremetal.tar -C package etc + tar rvf artifacts/baremetal.tar -C package/sbin entrypoint.sh - run: name: "Publish Release on GitHub" command: | go get github.com/tcnksm/ghr - VERSION=$(my-binary --version) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${VERSION} ./artifacts/ + SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${SEMVER} ./artifacts/ publish-latest: environment: From 1970b2add2a098f85ce2ac0cc0823d4d26ffdd74 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 22:46:40 -0400 Subject: [PATCH 19/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 27fd314..39bd584 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -274,7 +274,7 @@ jobs: command: | go get github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${SEMVER} ./artifacts/ + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete $SEMVER ./artifacts/ publish-latest: environment: From 0054914b0d88628aeb7ab162f3527d98b9d1b29a Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 23:03:26 -0400 Subject: [PATCH 20/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 39bd584..17cc979 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -274,7 +274,7 @@ jobs: command: | go get github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete $SEMVER ./artifacts/ + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} $SEMVER ./artifacts/ publish-latest: environment: From 2ec9e5e5e2b2c7b1660e1138af4b405346da5932 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 23:23:01 -0400 Subject: [PATCH 21/67] Update config.yml --- .circleci/config.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index 17cc979..0b5943c 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -274,6 +274,8 @@ jobs: command: | go get github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + echo version $SEMVER + ls -l ./artifacts ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} $SEMVER ./artifacts/ publish-latest: From 9930082666a41fc6943455a143852ecd49304b5c Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 23:34:57 -0400 Subject: [PATCH 22/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 0b5943c..77a7dcf 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -276,7 +276,7 @@ jobs: SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) echo version $SEMVER ls -l ./artifacts - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} $SEMVER ./artifacts/ + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} $SEMVER ./artifacts/ publish-latest: environment: From 5faaae44b037a662499e7e0d78f355fc8039d8ba Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Sun, 6 Oct 2019 23:47:02 -0400 Subject: [PATCH 23/67] Update config.yml --- .circleci/config.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 77a7dcf..10486e9 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -274,9 +274,7 @@ jobs: command: | go get github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - echo version $SEMVER - ls -l ./artifacts - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} $SEMVER ./artifacts/ + echo ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} $SEMVER ./artifacts/ publish-latest: environment: From db96e95c696b5e6214af9a02717ac499b059aa75 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:02:37 -0400 Subject: [PATCH 24/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 10486e9..44aa7d3 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -274,7 +274,7 @@ jobs: command: | go get github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - echo ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} $SEMVER ./artifacts/ + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} $SEMVER ./artifacts/ publish-latest: environment: From 422f6e316e410c30af6b7182716077d318b1a8f7 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:16:10 -0400 Subject: [PATCH 25/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 44aa7d3..46d15bb 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -274,7 +274,7 @@ jobs: command: | go get github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} $SEMVER ./artifacts/ + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} $SEMVER artifacts/ publish-latest: environment: From b574b323055fbaa045492b750b3ab828168481ae Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:17:48 -0400 Subject: [PATCH 26/67] Update config.yml --- .circleci/config.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 46d15bb..97ab4b0 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -329,9 +329,9 @@ workflows: only: - develop - publish-version: - requires: - - dgoss - - test-unit +# requires: +# - dgoss +# - test-unit filters: tags: only: /([0-9]+\.[0-9]+\.[0-9]+) From 39a33ace99797d5b087ea18e8260a22a4971b178 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:18:45 -0400 Subject: [PATCH 27/67] Update config.yml --- .circleci/config.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 97ab4b0..45ba336 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -329,7 +329,8 @@ workflows: only: - develop - publish-version: -# requires: + requires: + - build # - dgoss # - test-unit filters: From 756e0cb96bf758cbcce34e3f2162f2d91d73579e Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:22:34 -0400 Subject: [PATCH 28/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 45ba336..dd7b3a1 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -274,7 +274,7 @@ jobs: command: | go get github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} $SEMVER artifacts/ + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} 0.11.15 artifacts/ publish-latest: environment: From 079fcbe46823ae3c0b933e781dde3e573083aefb Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:25:01 -0400 Subject: [PATCH 29/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index dd7b3a1..7918ca7 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -274,7 +274,7 @@ jobs: command: | go get github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} 0.11.15 artifacts/ + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete v$SEMVER artifacts/ publish-latest: environment: From ad0177eae99151e76cde22aaacdadecfd44230fb Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:33:22 -0400 Subject: [PATCH 30/67] Update config.yml --- .circleci/config.yml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 7918ca7..f3c9084 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -272,9 +272,14 @@ jobs: - run: name: "Publish Release on GitHub" command: | - go get github.com/tcnksm/ghr + go get github.com/aktau/github-release SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete v$SEMVER artifacts/ + github-release upload \ + --user ${CIRCLE_PROJECT_USERNAME} \ + --repo ${CIRCLE_PROJECT_REPONAME} \ + --tag $SEMVER \ + --name "baremetal.tar" \ + --file artifacts/baremetal.tar publish-latest: environment: From 1d687de936dfb4881f146c0f986a1d15ee26acdc Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:38:43 -0400 Subject: [PATCH 31/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index f3c9084..53de381 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -275,7 +275,7 @@ jobs: go get github.com/aktau/github-release SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) github-release upload \ - --user ${CIRCLE_PROJECT_USERNAME} \ + --user ${GITHUB_USER} \ --repo ${CIRCLE_PROJECT_REPONAME} \ --tag $SEMVER \ --name "baremetal.tar" \ From 916da584705d04625b4b308b4ba3a22881a0a87f Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:42:02 -0400 Subject: [PATCH 32/67] Update config.yml --- .circleci/config.yml | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 53de381..d8111f6 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -274,12 +274,8 @@ jobs: command: | go get github.com/aktau/github-release SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - github-release upload \ - --user ${GITHUB_USER} \ - --repo ${CIRCLE_PROJECT_REPONAME} \ - --tag $SEMVER \ - --name "baremetal.tar" \ - --file artifacts/baremetal.tar + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${SEMVER} ./artifacts/ + publish-latest: environment: From 320088df16689db79e22159e72249f2073e1099e Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:47:22 -0400 Subject: [PATCH 33/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index d8111f6..5d90e6a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -272,7 +272,7 @@ jobs: - run: name: "Publish Release on GitHub" command: | - go get github.com/aktau/github-release + go get -u github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${SEMVER} ./artifacts/ From 9912b12551268a27ee32d0dfeef20f29ea19d8ee Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 00:51:41 -0400 Subject: [PATCH 34/67] Update config.yml --- .circleci/config.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 5d90e6a..5629455 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -263,7 +263,7 @@ jobs: command: | mkdir artifacts SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker save splunk/scs:$SEMVER | gzip -c > artifacts/sc4s_container.tar.gz + docker save splunk/scs:$SEMVER | gzip -c > artifacts/oci_container.tar.gz - run: name: BYOE Config command: | @@ -332,8 +332,8 @@ workflows: - publish-version: requires: - build -# - dgoss -# - test-unit + - dgoss + - test-unit filters: tags: only: /([0-9]+\.[0-9]+\.[0-9]+) From ae0e0bb70df014a186e21b01c8cfe4b3efbc76d2 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:23:37 -0400 Subject: [PATCH 35/67] Update config.yml --- .circleci/config.yml | 24 +++++++++--------------- 1 file changed, 9 insertions(+), 15 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 8d7ddba..a2957f7 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -9,7 +9,7 @@ version: 2.1 orbs: - clair_scanner: ovotech/clair-scanner@1 + snyk: snyk/snyk@0.0.8 jobs: build: @@ -149,23 +149,17 @@ jobs: - store_test_results: path: test-results test-scan_images: + docker: + - image: 'circleci/buildpack-deps:stretch' environment: IMAGE_NAME: rfaircloth/scs - executor: clair_scanner/default steps: - - clair_scanner/scan: - image: $IMAGE_NAME:$CIRCLE_SHA1 - whitelist: clair-whitelist.yml - - run: - command: | - mkdir -p /root/project/test-results - pip install -r requirements.txt - python clair_to_junit_parser.py "/clair-reports/$IMAGE_NAME:$CIRCLE_SHA1.json" --output test-results/results.xml - when: on_fail - - store_test_results: - path: test-results/results.xml - - store_artifacts: - path: /clair-reports + - checkout + - setup_remote_docker + - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 + - snyk/scan: + docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 + publish-common: machine: From b59f103bf373668307ec335b2e30c239bad60851 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:40:03 -0400 Subject: [PATCH 36/67] Update config.yml --- .circleci/config.yml | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index a2957f7..8c629ef 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -10,6 +10,7 @@ version: 2.1 orbs: snyk: snyk/snyk@0.0.8 + clair-scanner: ovotech/clair-scanner@1.5.0 jobs: build: @@ -148,7 +149,7 @@ jobs: when: always - store_test_results: path: test-results - test-scan_images: + test-scan-synk: docker: - image: 'circleci/buildpack-deps:stretch' environment: @@ -160,6 +161,25 @@ jobs: - snyk/scan: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 + test-scan-clair: + environment: + IMAGE_NAME: rfaircloth/scs + executor: clair_scanner/default + steps: + - clair_scanner/scan: + image: $IMAGE_NAME:$CIRCLE_SHA1 + whitelist: clair-whitelist.yml +# - run: +# command: | +# mkdir -p /root/project/test-results +# pip install -r requirements.txt +# python clair_to_junit_parser.py "/clair-reports/$IMAGE_NAME:$CIRCLE_SHA1.json" --output test-results/results.xml +# when: on_fail +# - store_test_results: +# path: test-results/results.xml +# - store_artifacts: +# path: /clair-reports + publish-common: machine: @@ -264,7 +284,7 @@ workflows: - test-unit: requires: - build - - test-scan_images: + - test-scan-synk: requires: - build - publish-common: From 081f2b6075ad857119f20f5cc0781d629b76a0b0 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:40:40 -0400 Subject: [PATCH 37/67] Update config.yml --- .circleci/config.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index 8c629ef..014f75f 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -287,6 +287,9 @@ workflows: - test-scan-synk: requires: - build + - test-scan-clair: + requires: + - build - publish-common: requires: - dgoss From 9a645dd6f4105c05180f1c157df77f83ff8bd625 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:43:04 -0400 Subject: [PATCH 38/67] Update config.yml --- .circleci/config.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 014f75f..db1c81a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -162,10 +162,13 @@ jobs: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 test-scan-clair: + docker: + - image: 'circleci/buildpack-deps:stretch' environment: IMAGE_NAME: rfaircloth/scs - executor: clair_scanner/default steps: + - setup_remote_docker + - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - clair_scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 whitelist: clair-whitelist.yml From c247216caee91c76680914da195447121b483e60 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:45:44 -0400 Subject: [PATCH 39/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index db1c81a..99a4aed 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -169,7 +169,7 @@ jobs: steps: - setup_remote_docker - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - clair_scanner/scan: + - clair-scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 whitelist: clair-whitelist.yml # - run: From e6d9298364e79ad1b0247ad4b38d769ecbd2b15a Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:47:36 -0400 Subject: [PATCH 40/67] Update config.yml --- .circleci/config.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 99a4aed..5e6b9f8 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -162,12 +162,10 @@ jobs: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 test-scan-clair: - docker: - - image: 'circleci/buildpack-deps:stretch' + executor: clair_scanner/default environment: IMAGE_NAME: rfaircloth/scs steps: - - setup_remote_docker - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - clair-scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 From 7ea2aab077b4380a0ae42ba6866228ff732565a2 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:50:20 -0400 Subject: [PATCH 41/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 5e6b9f8..9fb2dbe 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -162,7 +162,7 @@ jobs: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 test-scan-clair: - executor: clair_scanner/default + executor: clair-scanner/default environment: IMAGE_NAME: rfaircloth/scs steps: From 3428636c2de3250911291037ff186898e322958d Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:55:42 -0400 Subject: [PATCH 42/67] Update config.yml --- .circleci/config.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 9fb2dbe..52aabc7 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -162,10 +162,11 @@ jobs: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 test-scan-clair: - executor: clair-scanner/default + executor: clair_scanner/default environment: IMAGE_NAME: rfaircloth/scs steps: + - setup_remote_docker - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - clair-scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 From 882e5b32321842b106aa555b27abcb656227de0c Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:56:08 -0400 Subject: [PATCH 43/67] Update config.yml --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 52aabc7..2552e8f 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -162,7 +162,7 @@ jobs: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 test-scan-clair: - executor: clair_scanner/default + executor: clair-scanner/default environment: IMAGE_NAME: rfaircloth/scs steps: From 7a8c3b0970cebb9ad90d9d7bf3ddcca28d284303 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 08:59:12 -0400 Subject: [PATCH 44/67] Update config.yml --- .circleci/config.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 2552e8f..9fb2dbe 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -166,7 +166,6 @@ jobs: environment: IMAGE_NAME: rfaircloth/scs steps: - - setup_remote_docker - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - clair-scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 From 0000723a47b680692e6c8b4f67d053cbede8e0a4 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:02:49 -0400 Subject: [PATCH 45/67] Update config.yml --- .circleci/config.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 9fb2dbe..d7035d0 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -166,7 +166,6 @@ jobs: environment: IMAGE_NAME: rfaircloth/scs steps: - - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - clair-scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 whitelist: clair-whitelist.yml From 3600d1d6e68386ddbf293bafa0d8a4e2cd46b63b Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:11:08 -0400 Subject: [PATCH 46/67] Update config.yml --- .circleci/config.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index d7035d0..de0afc1 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -166,6 +166,8 @@ jobs: environment: IMAGE_NAME: rfaircloth/scs steps: + - setup_remote_docker: + docker_layer_caching: true - clair-scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 whitelist: clair-whitelist.yml From 2e33403ee99c1abb76473c6224c3d689cdcc88f1 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:17:15 -0400 Subject: [PATCH 47/67] Update config.yml --- .circleci/config.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index de0afc1..17fb7c9 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -162,12 +162,13 @@ jobs: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 test-scan-clair: - executor: clair-scanner/default + machine: + image: ubuntu-1604:201903-01 + docker_layer_caching: true # default - false environment: IMAGE_NAME: rfaircloth/scs steps: - - setup_remote_docker: - docker_layer_caching: true + - clair-scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 whitelist: clair-whitelist.yml From b80ab5b257ebc6074b762c3870b1918b00bf58a8 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:19:06 -0400 Subject: [PATCH 48/67] Update config.yml --- .circleci/config.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 17fb7c9..5dfe7cc 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -162,9 +162,6 @@ jobs: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 test-scan-clair: - machine: - image: ubuntu-1604:201903-01 - docker_layer_caching: true # default - false environment: IMAGE_NAME: rfaircloth/scs steps: From feff30cae99663c826c313c0c8951317dd5d245f Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:20:00 -0400 Subject: [PATCH 49/67] Update config.yml --- .circleci/config.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 5dfe7cc..c16809e 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -162,10 +162,13 @@ jobs: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 test-scan-clair: + docker: + - image: 'circleci/buildpack-deps:stretch' environment: IMAGE_NAME: rfaircloth/scs steps: - + - checkout + - setup_remote_docker - clair-scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 whitelist: clair-whitelist.yml From a7cd8535f3df45a923dcaeaf81a721d2e231472b Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:26:46 -0400 Subject: [PATCH 50/67] Update config.yml --- .circleci/config.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index c16809e..37e6a5e 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -167,8 +167,6 @@ jobs: environment: IMAGE_NAME: rfaircloth/scs steps: - - checkout - - setup_remote_docker - clair-scanner/scan: image: $IMAGE_NAME:$CIRCLE_SHA1 whitelist: clair-whitelist.yml From 277f856e25c9104852391fd98b8ca8f4fdc2ec04 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:30:28 -0400 Subject: [PATCH 51/67] Update config.yml --- .circleci/config.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index 37e6a5e..1ba4fe6 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -158,6 +158,7 @@ jobs: - checkout - setup_remote_docker - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 + - run: sudo mkdir /clair-reports - snyk/scan: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 From 311454f7986ae225c70d485f8e2e74a112db3ace Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:38:21 -0400 Subject: [PATCH 52/67] Update config.yml --- .circleci/config.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 1ba4fe6..7c24dd0 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -151,7 +151,7 @@ jobs: path: test-results test-scan-synk: docker: - - image: 'circleci/buildpack-deps:stretch' + - image: 'circleci/buildpack-deps:stable' environment: IMAGE_NAME: rfaircloth/scs steps: @@ -164,7 +164,7 @@ jobs: test-scan-clair: docker: - - image: 'circleci/buildpack-deps:stretch' + - image: 'docker:stable' environment: IMAGE_NAME: rfaircloth/scs steps: From 2673b71716d8e2fa1453de0624d72fe32395fe87 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:45:44 -0400 Subject: [PATCH 53/67] Update config.yml --- .circleci/config.yml | 78 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 64 insertions(+), 14 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 7c24dd0..c79348b 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -158,7 +158,6 @@ jobs: - checkout - setup_remote_docker - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: sudo mkdir /clair-reports - snyk/scan: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 @@ -168,19 +167,70 @@ jobs: environment: IMAGE_NAME: rfaircloth/scs steps: - - clair-scanner/scan: - image: $IMAGE_NAME:$CIRCLE_SHA1 - whitelist: clair-whitelist.yml -# - run: -# command: | -# mkdir -p /root/project/test-results -# pip install -r requirements.txt -# python clair_to_junit_parser.py "/clair-reports/$IMAGE_NAME:$CIRCLE_SHA1.json" --output test-results/results.xml -# when: on_fail -# - store_test_results: -# path: test-results/results.xml -# - store_artifacts: -# path: /clair-reports + - checkout + - setup_remote_docker: + - run: + name: "Vulnerability scan" + command: | + #!/usr/bin/env bash + + set -e + + REPORT_DIR=clair-reports + mkdir $REPORT_DIR + + DB=$(docker run -p 5432:5432 -d arminc/clair-db:latest) + CLAIR=$(docker run -p 6060:6060 --link "$DB":postgres -d arminc/clair-local-scan:latest) + CLAIR_SCANNER=$(docker run -v /var/run/docker.sock:/var/run/docker.sock -d ovotech/clair-scanner@sha256:53fe8e8ac63af330d2dfc63498d23d8825d07f916f7d230271176de06d12acd6 tail -f /dev/null) + + clair_ip=$(docker exec -it "$CLAIR" hostname -i | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+') + scanner_ip=$(docker exec -it "$CLAIR_SCANNER" hostname -i | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+') + + docker cp "clair-whitelist.yml" "$CLAIR_SCANNER:/whitelist.yml" + WHITELIST="-w /whitelist.yml" + + function scan() { + local image=$1 + # replace forward-slashes and colons with underscores + munged_image=$(echo "$image" | sed 's/\//_/g' | sed 's/:/_/g') + sanitised_image_filename="${munged_image}.json" + local ret=0 + local docker_cmd=(docker exec -it "$CLAIR_SCANNER" clair-scanner \ + --ip "$scanner_ip" \ + --clair=http://"$clair_ip":6060 \ + -t "high" \ + --report "/$sanitised_image_filename" \ + --log "/log.json" ${WHITELIST:+"-x"} + --reportAll=true \ + --exit-when-no-features=false \ + "$image") + + docker pull "$image" + + "${docker_cmd[@]}" 2>&1 || ret=$? + if [ $ret -eq 0 ]; then + echo "No unapproved vulnerabilities" + elif [ $ret -eq 1 ]; then + echo "Unapproved vulnerabilities found" + EXIT_STATUS=1 + elif [ $ret -eq 5 ]; then + echo "Image was not scanned, not supported." + EXIT_STATUS=1 + else + echo "Unknown clair-scanner return code $ret." + EXIT_STATUS=1 + fi + + docker cp "$CLAIR_SCANNER:/$sanitised_image_filename" "$REPORT_DIR/$sanitised_image_filename" || true + } + + EXIT_STATUS=0 + + scan "$IMAGE_NAME:$CIRCLE_SHA1" + + exit $EXIT_STATUS + - store_artifacts: + path: clair-reports publish-common: From ce0e85e1be4ccf0a232bc303e1d8bd5f85964404 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 09:47:34 -0400 Subject: [PATCH 54/67] Update config.yml --- .circleci/config.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index c79348b..3be25bd 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -156,7 +156,8 @@ jobs: IMAGE_NAME: rfaircloth/scs steps: - checkout - - setup_remote_docker + - setup_remote_docker: + docker_layer_caching: true - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - snyk/scan: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 @@ -169,6 +170,7 @@ jobs: steps: - checkout - setup_remote_docker: + docker_layer_caching: true - run: name: "Vulnerability scan" command: | From 3cecbca82c8411a022b2ef23f3d0d496b7fbe675 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 10:19:33 -0400 Subject: [PATCH 55/67] disable clair scanner --- .circleci/config.yml | 63 +++------------------------------------- clair-scan.sh | 68 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 59 deletions(-) create mode 100755 clair-scan.sh diff --git a/.circleci/config.yml b/.circleci/config.yml index 3be25bd..8dc6844 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -174,63 +174,7 @@ jobs: - run: name: "Vulnerability scan" command: | - #!/usr/bin/env bash - set -e - - REPORT_DIR=clair-reports - mkdir $REPORT_DIR - - DB=$(docker run -p 5432:5432 -d arminc/clair-db:latest) - CLAIR=$(docker run -p 6060:6060 --link "$DB":postgres -d arminc/clair-local-scan:latest) - CLAIR_SCANNER=$(docker run -v /var/run/docker.sock:/var/run/docker.sock -d ovotech/clair-scanner@sha256:53fe8e8ac63af330d2dfc63498d23d8825d07f916f7d230271176de06d12acd6 tail -f /dev/null) - - clair_ip=$(docker exec -it "$CLAIR" hostname -i | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+') - scanner_ip=$(docker exec -it "$CLAIR_SCANNER" hostname -i | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+') - - docker cp "clair-whitelist.yml" "$CLAIR_SCANNER:/whitelist.yml" - WHITELIST="-w /whitelist.yml" - - function scan() { - local image=$1 - # replace forward-slashes and colons with underscores - munged_image=$(echo "$image" | sed 's/\//_/g' | sed 's/:/_/g') - sanitised_image_filename="${munged_image}.json" - local ret=0 - local docker_cmd=(docker exec -it "$CLAIR_SCANNER" clair-scanner \ - --ip "$scanner_ip" \ - --clair=http://"$clair_ip":6060 \ - -t "high" \ - --report "/$sanitised_image_filename" \ - --log "/log.json" ${WHITELIST:+"-x"} - --reportAll=true \ - --exit-when-no-features=false \ - "$image") - - docker pull "$image" - - "${docker_cmd[@]}" 2>&1 || ret=$? - if [ $ret -eq 0 ]; then - echo "No unapproved vulnerabilities" - elif [ $ret -eq 1 ]; then - echo "Unapproved vulnerabilities found" - EXIT_STATUS=1 - elif [ $ret -eq 5 ]; then - echo "Image was not scanned, not supported." - EXIT_STATUS=1 - else - echo "Unknown clair-scanner return code $ret." - EXIT_STATUS=1 - fi - - docker cp "$CLAIR_SCANNER:/$sanitised_image_filename" "$REPORT_DIR/$sanitised_image_filename" || true - } - - EXIT_STATUS=0 - - scan "$IMAGE_NAME:$CIRCLE_SHA1" - - exit $EXIT_STATUS - store_artifacts: path: clair-reports @@ -341,9 +285,10 @@ workflows: - test-scan-synk: requires: - build - - test-scan-clair: - requires: - - build +#Clair scanner image is broken using synk for now +# - test-scan-clair: +# requires: +# - build - publish-common: requires: - dgoss diff --git a/clair-scan.sh b/clair-scan.sh new file mode 100755 index 0000000..e890ce3 --- /dev/null +++ b/clair-scan.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash + + set -e + + REPORT_DIR=clair-reports + mkdir $REPORT_DIR || true + + #DB=$(docker run -p 5432:5432 -d arminc/clair-db:latest) + docker run -p 5432:5432 -d --rm --name db arminc/clair-db:latest + #CLAIR=$(docker run -p 6060:6060 --link "$DB":postgres -d arminc/clair-local-scan:latest)' + sleep 30 + docker run -p 6060:6060 --link db:postgres -d --rm --name clair arminc/clair-local-scan:latest + #CLAIR_SCANNER=$(docker run -v /var/run/docker.sock:/var/run/docker.sock --link clair:clair --name clairscanner --rm -d ovotech/clair-scanner@sha256:53fe8e8ac63af330d2dfc63498d23d8825d07f916f7d230271176de06d12acd6 tail -f /dev/null) + + CLAIR_SCANNER=$(docker run --link clair:clair --name clairscanner --rm -d ovotech/clair-scanner@sha256:53fe8e8ac63af330d2dfc63498d23d8825d07f916f7d230271176de06d12acd6 tail -f /dev/null) + + #clair_ip=$(docker exec -it "$CLAIR" hostname -i | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+') + #scanner_ip=$(docker exec -it "$CLAIR_SCANNER" hostname -i | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+') + + docker cp "clair-whitelist.yml" "$CLAIR_SCANNER:/whitelist.yml" + WHITELIST="-w /whitelist.yml" + + function scan() { + echo Scanning $1 + local image=$1 + # replace forward-slashes and colons with underscores + munged_image=$(echo "$image" | sed 's/\//_/g' | sed 's/:/_/g') + sanitised_image_filename="${munged_image}.json" + local ret=0 + #--ip "$scanner_ip" \ + # + local docker_cmd=(docker exec -it "$CLAIR_SCANNER" clair-scanner \ + --clair=http://clair:6060 \ + -t "high" \ + --report "$REPORT_DIR/$sanitised_image_filename" \ + --log "$REPORT_DIR/log.json" --whitelist=${WHITELIST:+"-x"} + --reportAll=true \ + --exit-when-no-features=false \ + "$image") + + docker pull "$image" + + "${docker_cmd[@]}" 2>&1 || ret=$? + if [ $ret -eq 0 ]; then + echo "No unapproved vulnerabilities" + elif [ $ret -eq 1 ]; then + echo "Unapproved vulnerabilities found" + EXIT_STATUS=1 + elif [ $ret -eq 5 ]; then + echo "Image was not scanned, not supported." + EXIT_STATUS=1 + else + echo "Unknown clair-scanner return code $ret." + EXIT_STATUS=1 + fi + + docker cp "$CLAIR_SCANNER:/$sanitised_image_filename" "$REPORT_DIR/$sanitised_image_filename" || true + } + + EXIT_STATUS=0 + + scan "$IMAGE_NAME:$CIRCLE_SHA1" + + docker kill clairscanner + docker kill clair + docker kill db + + exit $EXIT_STATUS \ No newline at end of file From d8694d09b4fb1019559e8ac0b7ec8db966a72af0 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 10:25:41 -0400 Subject: [PATCH 56/67] Update config.yml --- .circleci/config.yml | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 5629455..564cb0c 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -191,6 +191,23 @@ jobs: command: | SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) docker push $IMAGE_NAME:$SEMVER + - run: + name: Docker Save + command: | + mkdir artifacts + SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + docker save splunk/scs:$SEMVER | gzip -c > artifacts/oci_container.tar.gz + - run: + name: BYOE Config + command: | + tar rvf artifacts/baremetal.tar -C package etc + tar rvf artifacts/baremetal.tar -C package/sbin entrypoint.sh + - run: + name: "Publish Release on GitHub" + command: | + go get -u github.com/tcnksm/ghr + SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${SEMVER} ./artifacts/ publish-edge: environment: @@ -316,11 +333,11 @@ workflows: requires: - dgoss - test-unit -# filters: -# branches: -# only: -# - master -# - develop + filters: + branches: + only: + - master + - develop - publish-edge: requires: - dgoss From d54bf2af14ec2526343b34cca23be2f9b237a3f3 Mon Sep 17 00:00:00 2001 From: Mark Bonsack Date: Mon, 7 Oct 2019 07:57:19 -0700 Subject: [PATCH 57/67] update vendor_product_by_source.conf Remove bogus comment and update version string --- package/etc/context_templates/vendor_product_by_source.conf | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/package/etc/context_templates/vendor_product_by_source.conf b/package/etc/context_templates/vendor_product_by_source.conf index 37e3412..342c9f4 100644 --- a/package/etc/context_templates/vendor_product_by_source.conf +++ b/package/etc/context_templates/vendor_product_by_source.conf @@ -1,7 +1,4 @@ -@version: 3.22 -#TODO: #60 The syntax below uses regex and an indirect reference to a variable due to a -#bug/limitation of selector files. The better syntax should be as follows -#filter {match("f5_test" template("$(env PRESUME_SYSLOG)")); }; +@version: 3.23 filter f_test_test { host("testvp-*" type(glob)) or From 666bbd246f541b658a5c6d84674a33d2edc5e1ee Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 14:18:14 -0400 Subject: [PATCH 58/67] Fixes #112 --- .../{SC4S deployment.png => SC4Sdeployment.png} | Bin docs/gettingstarted.md | 3 ++- docs/performance.md | 16 ++++++++++++++++ 3 files changed, 18 insertions(+), 1 deletion(-) rename docs/{SC4S deployment.png => SC4Sdeployment.png} (100%) diff --git a/docs/SC4S deployment.png b/docs/SC4Sdeployment.png similarity index 100% rename from docs/SC4S deployment.png rename to docs/SC4Sdeployment.png diff --git a/docs/gettingstarted.md b/docs/gettingstarted.md index 7a29b37..a43e906 100644 --- a/docs/gettingstarted.md +++ b/docs/gettingstarted.md @@ -21,6 +21,7 @@ instance in the same VLAN as the source device. environment. * Avoid TCP except where the source is unable to contain the event to a single UDP packet. * Avoid TLS except where the event may cross a untrusted network. +* Plan for appropriately sized hardware (see)[performance.md] # Implementation @@ -81,4 +82,4 @@ Splunk type. Additional hosts can be deployed for syslog collection from additional network zones and locations: -![SC4S deployment diagram](SC4S%20deployment.png) \ No newline at end of file +![SC4S deployment diagram](SC4Sdeployment.png) \ No newline at end of file diff --git a/docs/performance.md b/docs/performance.md index 966a318..53831b2 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -24,3 +24,19 @@ average rate = 9717.58 msg/sec, count=1749420, time=180.026, (average) msg size= Splunk Enterprise's implementation of the http event collection server responds to the client with a status code 200 and fails to commit the events to disk during a rolling restart. In our testing, 20-30 events per indexer are lost. +## Guidance on sizing hardware + +The following reference deployment hardware specifications are based on Splunk performance testing results in AWS +The overall load on your deployment HW will vary based on the percentage of events which are not handled by a filter or +custom filters added not accounted for here. While we consider the following conservative actual hardware performance will vary +due to network interface card, driver, kernel version, exact CPU even type of memory and configuration. SYSLOG is a fire +and forget protocol making it sensitive to performance. Given this it is highly recommended you should validate +performance with your hardware and production data samples. The syslog-ng loggen tool available in the SC4S container +and the commands above can be utilized in this effort. + +Deployment Size | Hardware Spec | Average EPS with average msg size 800 k +-- | -- | -- +Small | 2 X 3.1 ghz cores1 GB of memory | 2K msg/sec +Medium | 4 X 3.1 ghz cores2 GB of memory | 4.5K msg/sec +Large | 8 X 3.1 ghz cores4 GB of memory | 9K msg/sec +XL | 16 X 3.1 ghz cores8 GB of memory | 18K msg/sec From c8cab0e0f9785e3c87714486e0b86e1ab8b37bb2 Mon Sep 17 00:00:00 2001 From: rfaircloth-splunk Date: Mon, 7 Oct 2019 14:43:27 -0400 Subject: [PATCH 59/67] Update performance.md --- docs/performance.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/performance.md b/docs/performance.md index 53831b2..8093a6c 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -22,13 +22,14 @@ average rate = 9717.58 msg/sec, count=1749420, time=180.026, (average) msg size= ## Limitations -Splunk Enterprise's implementation of the http event collection server responds to the client with a status code 200 and fails to commit the events to disk during a rolling restart. In our testing, 20-30 events per indexer are lost. +Splunk Enterprise's implementation of the http event collection server responds to the client with a status code 200 and +fails to commit the events to disk during a rolling restart. In our testing, 20-30 events per indexer are lost. ## Guidance on sizing hardware The following reference deployment hardware specifications are based on Splunk performance testing results in AWS -The overall load on your deployment HW will vary based on the percentage of events which are not handled by a filter or -custom filters added not accounted for here. While we consider the following conservative actual hardware performance will vary +The overall load on your deployment hardware will vary based on the percentage of events not handled by a filter or use of +exceptionally complex regex in filters. While we consider the following conservative actual hardware performance will vary due to network interface card, driver, kernel version, exact CPU even type of memory and configuration. SYSLOG is a fire and forget protocol making it sensitive to performance. Given this it is highly recommended you should validate performance with your hardware and production data samples. The syslog-ng loggen tool available in the SC4S container From 27b552f74eeab4953c41c4eb1a2b0366f92d2d49 Mon Sep 17 00:00:00 2001 From: mkarlstrand-splunk <49571555+mkarlstrand-splunk@users.noreply.github.com> Date: Mon, 7 Oct 2019 12:06:03 -0700 Subject: [PATCH 60/67] Update performance.md --- docs/performance.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/performance.md b/docs/performance.md index 8093a6c..ea02c72 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -22,16 +22,15 @@ average rate = 9717.58 msg/sec, count=1749420, time=180.026, (average) msg size= ## Limitations -Splunk Enterprise's implementation of the http event collection server responds to the client with a status code 200 and -fails to commit the events to disk during a rolling restart. In our testing, 20-30 events per indexer are lost. +In our tests, if Splunk Enterprise’s implementation of the http event collection server responded to the client with a status code 200 and failed to commit the events to disk during a rolling restart, then 20-30 events per indexer were lost. ## Guidance on sizing hardware -The following reference deployment hardware specifications are based on Splunk performance testing results in AWS +The following reference deployment hardware specifications are based on Splunk performance testing results in Amazon Web Services. The overall load on your deployment hardware will vary based on the percentage of events not handled by a filter or use of -exceptionally complex regex in filters. While we consider the following conservative actual hardware performance will vary -due to network interface card, driver, kernel version, exact CPU even type of memory and configuration. SYSLOG is a fire -and forget protocol making it sensitive to performance. Given this it is highly recommended you should validate +exceptionally complex regex in filters. While we consider the following conservative, actual hardware performance will vary +due to network interface card, driver, kernel version, exact CPU, type of memory and configuration. SYSLOG is a fire +and forget protocol making it sensitive to performance. Given this it is highly recommended that you validate performance with your hardware and production data samples. The syslog-ng loggen tool available in the SC4S container and the commands above can be utilized in this effort. From 5d1a7a23b0b0d646076b770d12393480f32694e1 Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Mon, 7 Oct 2019 20:58:42 -0400 Subject: [PATCH 61/67] Update config.yml (#124) * CI CD Cleanup * Use GITHUB Registry for non public builds * Use docker hub for final builds --- .circleci/config.yml | 371 +++++++++++++++++++++++++++--------------- docker-compose-ci.yml | 2 +- 2 files changed, 245 insertions(+), 128 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index fb30dd4..8517e64 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -14,54 +14,55 @@ orbs: jobs: build: + machine: + image: ubuntu-1604:201903-01 + docker_layer_caching: true # default - false environment: - IMAGE_NAME: rfaircloth/scs - docker: - - image: circleci/buildpack-deps:stretch + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci +# docker: +# - image: circleci/buildpack-deps:stretch steps: +# - setup_remote_docker: +# docker_layer_caching: true - checkout - run: git submodule sync - run: git submodule update --init --recursive - - setup_remote_docker: - docker_layer_caching: true + - run: mkdir /tmp/artifacts + - run: docker pull gittools/gitversion:latest-linux-netcoreapp2.1 - run: name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS + command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - run: name: Build Docker image command: docker build -f package/Dockerfile --build-arg RH_ORG=$RH_ORG --build-arg RH_ACTIVATION=$RH_ACTIVATION -t $IMAGE_NAME:$CIRCLE_SHA1 package - run: - name: Push Docker image + name: Push Docker image by hash command: docker push $IMAGE_NAME:$CIRCLE_SHA1 - build-egb: - environment: - IMAGE_NAME: rfaircloth/scs - docker: - - image: circleci/buildpack-deps:stretch - steps: - - checkout - - run: git submodule sync - - run: git submodule update --init --recursive - - setup_remote_docker: - docker_layer_caching: true - run: - name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS - - run: - name: Build Docker image - command: docker build -f perftests/bundlesrv/Dockerfile -t $IMAGE_NAME:egb-$CIRCLE_SHA1 perftests/bundlesrv - - run: - name: Tag Docker image - command: docker tag $IMAGE_NAME:egb-$CIRCLE_SHA1 $IMAGE_NAME:egb-edge + name: Docker tag and push with version label + command: | + SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + echo image $IMAGE_NAME:$SEMVER + docker tag $IMAGE_NAME:$CIRCLE_SHA1 $IMAGE_NAME:$SEMVER + docker push $IMAGE_NAME:$SEMVER + - run: - name: Push Docker image - command: docker push $IMAGE_NAME:egb-$CIRCLE_SHA1 + name: Docker Save + command: | + SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + docker save $IMAGE_NAME:$SEMVER | gzip -c > /tmp/artifacts/oci_container.tar.gz - run: - name: Push Docker image - command: docker push $IMAGE_NAME:egb-edge + name: BYOE Config + command: | + tar rvf /tmp/artifacts/baremetal.tar -C package etc + tar rvf /tmp/artifacts/baremetal.tar -C package/sbin entrypoint.sh + + - store_artifacts: + path: /tmp/artifacts + dgoss: environment: - IMAGE_NAME: rfaircloth/scs + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci GOSS_VER: v0.3.7 GOSS_OPTS: "--max-concurrent=1 --format junit" GOSS_FILES_STRATEGY: cp @@ -74,7 +75,7 @@ jobs: docker_layer_caching: true - run: name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS + command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - run: name: Docker pull command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 @@ -97,6 +98,8 @@ jobs: - store_test_results: path: test-results test-unit: + environment: + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci docker: - image: circleci/buildpack-deps:stretch steps: @@ -107,7 +110,7 @@ jobs: - run: git submodule update --init --recursive - run: name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS + command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - run: name: Docker pull command: docker-compose pull @@ -153,20 +156,22 @@ jobs: docker: - image: 'circleci/buildpack-deps:stable' environment: - IMAGE_NAME: rfaircloth/scs + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci steps: - checkout - setup_remote_docker: docker_layer_caching: true + - run: + name: Docker Login + command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - snyk/scan: docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 - test-scan-clair: docker: - image: 'docker:stable' environment: - IMAGE_NAME: rfaircloth/scs + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci steps: - checkout - setup_remote_docker: @@ -179,159 +184,225 @@ jobs: path: clair-reports - publish-common: - machine: - image: ubuntu-1604:201903-01 - docker_layer_caching: true # default - false + publish-edge: environment: - IMAGE_NAME: rfaircloth/scs + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci + PUBLIC_IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/releases + docker: + - image: circleci/buildpack-deps:stretch steps: - - checkout + - setup_remote_docker: + docker_layer_caching: true + - run: + name: Create Directory + command: mkdir /tmp/artifacts - run: name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS + command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - run: name: Docker pull command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - run: name: Docker tag image - command: | - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker tag $IMAGE_NAME:$CIRCLE_SHA1 $IMAGE_NAME:$SEMVER + command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:edge - run: name: Docker push tag - command: | - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker push $IMAGE_NAME:$SEMVER + command: docker push $PUBLIC_IMAGE_NAME:edge + - run: name: Docker Save command: | - mkdir artifacts - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker save splunk/scs:$SEMVER | gzip -c > artifacts/oci_container.tar.gz + docker save $IMAGE_NAME:$CIRCLE_SHA1 | gzip -c > /tmp/artifacts/oci_container.tar.gz - run: name: BYOE Config command: | - tar rvf artifacts/baremetal.tar -C package etc - tar rvf artifacts/baremetal.tar -C package/sbin entrypoint.sh + tar rvf /tmp/artifacts/baremetal.tar -C package etc + tar rvf /tmp/artifacts/baremetal.tar -C package/sbin entrypoint.sh + - run: name: "Publish Release on GitHub" command: | go get -u github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${SEMVER} ./artifacts/ - - publish-edge: - environment: - IMAGE_NAME: rfaircloth/scs - docker: - - image: circleci/buildpack-deps:stretch - steps: - - setup_remote_docker: - docker_layer_caching: true - - run: - name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS - - run: - name: Docker pull - command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: - name: Docker tag image - command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 $IMAGE_NAME:edge - - run: - name: Docker push tag - command: docker push $IMAGE_NAME:edge + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete edge /tmp/artifacts/ publish-version: machine: image: ubuntu-1604:201903-01 docker_layer_caching: true # default - false environment: - IMAGE_NAME: rfaircloth/scs + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci + PUBLIC_IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/releases steps: + - checkout - run: - name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS - - run: - name: Docker pull - command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: - name: Docker tag image - command: | - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker tag $IMAGE_NAME:$CIRCLE_SHA1 splunk/scs:$SEMVER + name: Create Directory + command: mkdir /tmp/artifacts - run: - name: Docker push tag + name: Docker Login command: | - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker push splunk/scs:$SEMVER - release-version: - machine: - image: ubuntu-1604:201903-01 - docker_layer_caching: true # default - false - environment: - IMAGE_NAME: rfaircloth/scs - steps: - - checkout + docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN + docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: - name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS - - run: - name: Docker pull - command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - run: name: Docker tag image command: | SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker tag $IMAGE_NAME:$CIRCLE_SHA1 splunk/scs:$SEMVER + docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:$SEMVER + docker push $PUBLIC_IMAGE_NAME:$SEMVER + - run: name: Docker Save command: | - mkdir artifacts SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker save splunk/scs:$SEMVER | gzip -c > artifacts/oci_container.tar.gz + docker save $PUBLIC_IMAGE_NAME:$SEMVER | gzip -c > /tmp/artifacts/oci_container.tar.gz - run: name: BYOE Config command: | - tar rvf artifacts/baremetal.tar -C package etc - tar rvf artifacts/baremetal.tar -C package/sbin entrypoint.sh + tar rvf /tmp/artifacts/baremetal.tar -C package etc + tar rvf /tmp/artifacts/baremetal.tar -C package/sbin entrypoint.sh - run: name: "Publish Release on GitHub" command: | go get -u github.com/tcnksm/ghr SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${SEMVER} ./artifacts/ + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${SEMVER} /tmp/artifacts/ publish-latest: environment: - IMAGE_NAME: rfaircloth/scs + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci + PUBLIC_IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/releases docker: - image: circleci/buildpack-deps:stretch steps: - setup_remote_docker: docker_layer_caching: true + - run: + name: Create Directory + command: mkdir /tmp/artifacts - run: name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS + command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - run: name: Docker pull command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - run: name: Docker tag image - command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 splunk/scs:latest + command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:latest - run: name: Docker push tag - command: docker push splunk/scs:latest + command: docker push $PUBLIC_IMAGE_NAME:latest + + - run: + name: Docker Save + command: | + docker save $IMAGE_NAME:$CIRCLE_SHA1 | gzip -c > /tmp/artifacts/oci_container.tar.gz + - run: + name: BYOE Config + command: | + tar rvf /tmp/artifacts/baremetal.tar -C package etc + tar rvf /tmp/artifacts/baremetal.tar -C package/sbin entrypoint.sh + + - run: + name: "Publish Release on GitHub" + command: | + go get -u github.com/tcnksm/ghr + SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete latest /tmp/artifacts/ + + + dockerhub-version: + machine: + image: ubuntu-1604:201903-01 + docker_layer_caching: true # default - false + environment: + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci + PUBLIC_IMAGE_NAME: splunk/scs + steps: + + - checkout + + - run: + name: Create Directory + command: mkdir artifacts + - run: + name: Docker Login + command: | + docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN + docker pull $IMAGE_NAME:$CIRCLE_SHA1 + - run: + name: Docker tag image + command: | + SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) + docker login -u $DOCKER_USER -p $DOCKER_PASS + docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:$SEMVER + docker push $PUBLIC_IMAGE_NAME:$SEMVER + + dockerhub-edge: + environment: + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci + PUBLIC_IMAGE_NAME: splunk/scs + docker: + - image: circleci/buildpack-deps:stretch + steps: + - setup_remote_docker: + docker_layer_caching: true + - run: + name: Create Directory + command: mkdir /tmp/artifacts + - run: + name: Docker Login + command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN + - run: + name: Docker pull + command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 + - run: + name: Docker Login + command: docker login -u $DOCKER_USER -p $DOCKER_PASS + - run: + name: Docker tag image + command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:edge + - run: + name: Docker push tag + command: docker push $PUBLIC_IMAGE_NAME:edge + + dockerhub-latest: + environment: + IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci + PUBLIC_IMAGE_NAME: splunk/scs + docker: + - image: circleci/buildpack-deps:stretch + steps: + - setup_remote_docker: + docker_layer_caching: true + - run: + name: Create Directory + command: mkdir /tmp/artifacts + - run: + name: Docker Login + command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN + - run: + name: Docker pull + command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 + - run: + name: Docker Login + command: docker login -u $DOCKER_USER -p $DOCKER_PASS + - run: + name: Docker tag image + command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:latest + - run: + name: Docker push tag + command: docker push $PUBLIC_IMAGE_NAME:latest + workflows: - build-publish: + version: 2 + build-branches: jobs: - build - - build-egb - dgoss: requires: - build @@ -345,16 +416,23 @@ workflows: # - test-scan-clair: # requires: # - build - - publish-common: + - publish-edge: requires: - dgoss - test-unit filters: branches: only: - - master - develop - - publish-edge: + - publish-latest: + requires: + - dgoss + - test-unit + filters: + branches: + only: + - master + - dockerhub-edge: requires: - dgoss - test-unit @@ -362,22 +440,61 @@ workflows: branches: only: - develop - - publish-version: + - dockerhub-latest: requires: - - build - dgoss - test-unit + filters: + branches: + only: + - master + build-tags: + jobs: + - build: filters: tags: - only: /([0-9]+\.[0-9]+\.[0-9]+) - - release-version: + only: /^\d*\.\d*\.\d*.*$/ + branches: + ignore: /.*/ + - dgoss: + filters: + tags: + only: /^\d*\.\d*\.\d*.*$/ + branches: + ignore: /.*/ requires: - - publish-version - - publish-latest: + - build + - test-unit: + filters: + tags: + only: /^\d*\.\d*\.\d*.*$/ + branches: + ignore: /.*/ + requires: + - build + - test-scan-synk: + filters: + tags: + only: /^\d*\.\d*\.\d*.*$/ + branches: + ignore: /.*/ + requires: + - build + - publish-version: + filters: + tags: + only: /^\d*\.\d*\.\d*.*$/ + branches: + ignore: /.*/ requires: - dgoss - test-unit + - dockerhub-version: filters: + tags: + only: /^\d*\.\d*\.\d*.*$/ branches: - only: - - master + ignore: /.*/ + requires: + - dgoss + - test-unit \ No newline at end of file diff --git a/docker-compose-ci.yml b/docker-compose-ci.yml index c33122c..b4bb66d 100644 --- a/docker-compose-ci.yml +++ b/docker-compose-ci.yml @@ -19,7 +19,7 @@ services: - SPLUNK_PASSWORD=${SPLUNK_PASSWORD} sc4s: - image: rfaircloth/scs:${CIRCLE_SHA1} + image: ${IMAGE_NAME}:${CIRCLE_SHA1} hostname: sc4s ports: - "514" From 2d42b87028f4b1534973d3fb78280eaab832056a Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Mon, 7 Oct 2019 21:02:21 -0400 Subject: [PATCH 62/67] Update CONTRIBUTING.md (#121) * Update CONTRIBUTING.md * add link to wiki --- CONTRIBUTING.md | 150 ++++++++++++++++++------------------------------ 1 file changed, 56 insertions(+), 94 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 21e497a..e210a11 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,120 +1,82 @@ -# Contributing +#CONTRIBUTING -### Past / Active(marked as *) Contributors +##Prerequisites -Splunk Connect for Syslog is developed by Splunkers and the open-source community. +When contributing to this repository, please first discuss the change you wish to make via a GitHub issue or Slack message with the owners of this repository. -We want to give extra special thanks to Hurricane Labs our seed community contributor +##Setup Development Environment -We thank all of our contributors! +For a basic development environment docker and a bash shell is all you need. For a more complete IDE experience see our wiki (Setup PyCharm)[https://github.com/splunk/splunk-connect-for-syslog/wiki/SC4S-Development-Setup-Using-PyCharm] -[https://github.com/splunk/splunk-connect-for-syslog/graphs/contributors] +##Contribution Workflow -For the detailed history of contributions of a given file, try +SC4S is a community project so please consider contributing your efforts! For example, documentation can always use improvement. There's always code that can be clarified, functionality that can be extended, new data filters to develop. If you see something you think should be fixed or added, go for it. -git blame file -to see line-by-line credits and +#Feature Requests and Bug Reports -git log --follow file -to see the change log even across renames and rewrites. +Have ideas on improvements or found a problem? While the community encourages everyone to contribute code, it is also appreciated when someone reports an issue. Please report any issues or bugs you find through GitHub's issue tracker. +If you are reporting a bug, please include the following details: -## Code of conduct +* Your operating system name and version +* Any details about your local setup that might be helpful in troubleshooting (ex. container runtime you use, etc.) +* Detailed steps to reproduce the bug +* We want to hear about you enhancements as well. Feel free to submit them as issues: +* Explain in detail how they should work +* Keep the scope as narrow as possible. This will make it easier to implement +##Fixing Issues -### Our Pledge +Look through our issue tracker to find problems to fix! Feel free to comment and tag community members of this project with any questions or concerns. -In the interest of fostering an open and welcoming environment, we as -contributors and maintainers pledge to making participation in our project and -our community a harassment-free experience for everyone, regardless of age, body -size, disability, ethnicity, gender identity and expression, level of experience, -nationality, personal appearance, race, religion, or sexual identity and -orientation. +##Pull Requests -### Our Standards +What is a "pull request"? It informs the project's core developers about the changes you want to review and merge. Once you submit a pull request, it enters a stage of code review where you and others can discuss its potential modifications and even add more commits to it later on. -Examples of behavior that contributes to creating a positive environment -include: +If you want to learn more, please consult this tutorial on how pull requests work in the GitHub Help Center. -* Using welcoming and inclusive language -* Being respectful of differing viewpoints and experiences -* Gracefully accepting constructive criticism -* Focusing on what is best for the community -* Showing empathy towards other community members +Here's an overview of how you can make a pull request against this project: -Examples of unacceptable behavior by participants include: +* Fork the Splunk-connect-for-syslog GitHub repository +* Clone your fork using git and create a branch off develop +$ git clone git@github.com:YOUR_GITHUB_USERNAME/splunk-connect-for-syslog.git +$ cd splunk-connect-for-syslog +* This project uses 'develop' for all development activity, so create your branch off that +$ git checkout -b your-bugfix-branch-name develop +* Run all the tests to verify your environment +$ cd splunk-connect-for-syslog +$ ./test-with-compose.sh +* Make your changes, commit and push once your tests have passed +$ git commit -m "" +$ git push +* Submit a pull request through the GitHub website using the changes from your forked codebase -* The use of sexualized language or imagery and unwelcome sexual attention or -advances -* Trolling, insulting/derogatory comments, and personal or political attacks -* Public or private harassment -* Publishing others' private information, such as a physical or electronic - address, without explicit permission -* Other conduct which could reasonably be considered inappropriate in a - professional setting +##Code Review +There are two aspects of code review: giving and receiving. +To make it easier for your PR to receive reviews, consider the reviewers will need you to: -### Our Responsibilities +* Follow the project coding conventions +* Write good commit messages +* Break large changes into a logical series of smaller patches which individually make easily understandable changes, and in aggregate solve a broader issue +* Reviewers, the people giving the review, are highly encouraged to revisit the Code of Conduct and must go above and beyond to promote a collaborative, respectful community. +* When reviewing PRs from others The Gentle Art of Patch Review suggests an iterative series of focuses which is designed to lead new contributors to positive collaboration without inundating them initially with nuances: +* Is the idea behind the contribution sound? +* Is the contribution architected correctly? +* Is the contribution polished? +* For this project, we require that at least 2 approvals are given and a build from our continuous integration system is successful off of your branch. Please note that any new changes made with your existing pull request during review will automatically unapprove and retrigger another build/round of tests. -Project maintainers are responsible for clarifying the standards of acceptable -behavior and are expected to take appropriate and fair corrective action in -response to any instances of unacceptable behavior. +##Testing -Project maintainers have the right and responsibility to remove, edit, or -reject comments, commits, code, wiki edits, issues, and other contributions -that are not aligned to this Code of Conduct, or to ban temporarily or -permanently any contributor for other behaviors that they deem inappropriate, -threatening, offensive, or harmful. +Testing is the responsibility of all contributors. In general, we try to adhere to TDD, writing the test first. +There are multiple types of tests. The location of the test code varies with type, as do the specifics of the environment needed to successfully run the test. -### Scope +* Review existing tests in the tests folder of the repo -This Code of Conduct applies both within project spaces and in public spaces -when an individual is representing the project or its community. Examples of -representing a project or community include using an official project e-mail -address, posting via an official social media account, or acting as an appointed -representative at an online or offline event. Representation of a project may be -further defined and clarified by project maintainers. +We could always use improvements to our documentation! Anyone can contribute to these docs - whether you’re new to the project, you’ve been around a long time, and whether you self-identify as a developer, an end user, or someone who just can’t stand seeing typos. What exactly is needed? -### Enforcement - -Instances of abusive, harassing, or otherwise unacceptable behavior may be -reported by contacting the project team at tonyl@splunk.com. All -complaints will be reviewed and investigated and will result in a response that -is deemed necessary and appropriate to the circumstances. The project team is -obligated to maintain confidentiality with regard to the reporter of an incident. -Further details of specific enforcement policies may be posted separately. - -Project maintainers who do not follow or enforce the Code of Conduct in good -faith may face temporary or permanent repercussions as determined by other -members of the project's leadership. - -### Attribution - -This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, -available at [http://contributor-covenant.org/version/1/4][version] - -[homepage]: http://contributor-covenant.org -[version]: http://contributor-covenant.org/version/1/4/ - -## Filing issues - -Please file issues in this project with clear description of the problem. - -## Project Discussion Forums - -splunk-usergroups #splunk_connect_for_syslog - -## Contributing code or data - -Future Docs covering the contribution process and requirements - -By submitting a Contribution to this Work, You agree that Your Contribution is made subject to the license files applicable to this Work. Contributions comprising code shall be made pursuant the BSD2 license; Contributions consisting of configuration information or data shall be made pursuant to the CCO license. In addition, You represent that: You are the copyright owner of the Contribution or (ii) You have the requisite rights to make the Contribution. - -Definitions: - -“You” shall mean: yourself if you are making a Contribution on your own behalf; or (ii) your company, if you are making a Contribution on behalf of your company. If you are making a Contribution on behalf of your company, you represent that you have the requisite authority to do so. - -"Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You for inclusion in, or documentation of, this project/repository. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication submitted for inclusion in this project/repository, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the maintainers of the project/repository. - -“Work” shall mean the collective software, data, content, and documentation in this project/repository. +* More complementary documentation. Have you perhaps found something unclear? +* More examples or generic templates that others can use. +* Blog posts, articles and such – they’re all very appreciated. +* You can also edit documentation files directly in the GitHub web interface, without creating a local copy. This can be convenient for small typos or grammar fixes. \ No newline at end of file From e89f1058959ae8c519ac5d5bed09f49d4437db7e Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Mon, 7 Oct 2019 21:03:24 -0400 Subject: [PATCH 63/67] Feature/coc (#122) * Fixes (115) * fixes (114) --- CODE_OF_CONDUCT.md | 74 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 CODE_OF_CONDUCT.md diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..ebc927b --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,74 @@ +# Code of conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to making participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, gender identity and expression, level of experience, +nationality, personal appearance, race, religion, or sexual identity and +orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or +advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic + address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project e-mail +address, posting via an official social media account, or acting as an appointed +representative at an online or offline event. Representation of a project may be +further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at [support@splunk.com](mailto:support@splunk.com). All +complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, +available at [http://contributor-covenant.org/version/1/4][version] + +[homepage]: http://contributor-covenant.org +[version]: http://contributor-covenant.org/version/1/4/ \ No newline at end of file From 5e52cf4909c3ff90657bc5a7e088b82369bf219b Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Mon, 7 Oct 2019 21:23:31 -0400 Subject: [PATCH 64/67] Feature/don't upload p rs to releases (#125) * Add missing checkout --- .circleci/config.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index 8517e64..b2884a2 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -193,6 +193,7 @@ jobs: steps: - setup_remote_docker: docker_layer_caching: true + - checkout - run: name: Create Directory command: mkdir /tmp/artifacts @@ -280,6 +281,10 @@ jobs: steps: - setup_remote_docker: docker_layer_caching: true + - checkout + - run: + name: Create Directory + command: mkdir /tmp/artifacts - run: name: Create Directory command: mkdir /tmp/artifacts From 81efe1125cb4bb89ac3a5876919aa9f488ce96dc Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Mon, 7 Oct 2019 22:19:06 -0400 Subject: [PATCH 65/67] Feature/don't upload p rs to releases (#127) CI Work --- .circleci/config.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index b2884a2..65ac936 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -189,7 +189,9 @@ jobs: IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci PUBLIC_IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/releases docker: - - image: circleci/buildpack-deps:stretch + #- image: circleci/buildpack-deps:stretch + - image: circleci/golang:latest + steps: - setup_remote_docker: docker_layer_caching: true @@ -277,14 +279,12 @@ jobs: IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci PUBLIC_IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/releases docker: - - image: circleci/buildpack-deps:stretch + #- image: circleci/buildpack-deps:stretch + - image: circleci/golang:latest steps: - setup_remote_docker: docker_layer_caching: true - checkout - - run: - name: Create Directory - command: mkdir /tmp/artifacts - run: name: Create Directory command: mkdir /tmp/artifacts From 8f7ca3116c18a05cce2afa66a3c6ca0b5563b548 Mon Sep 17 00:00:00 2001 From: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com> Date: Tue, 8 Oct 2019 15:01:50 -0400 Subject: [PATCH 66/67] Feature/don't upload p rs to releases (#128) Restructure CI --- .circleci/config.yml | 584 ++++++++++++++++++------------------------ docker-compose-ci.yml | 2 +- 2 files changed, 251 insertions(+), 335 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 65ac936..fda0601 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -7,117 +7,102 @@ #You should have received a copy of the CC0 legalcode along with this #work. If not, see . version: 2.1 - orbs: + docker: circleci/docker@0.5.13 + go: circleci/go@0.2.0 snyk: snyk/snyk@0.0.8 - clair-scanner: ovotech/clair-scanner@1.5.0 - + versioning: kollex/versioning@1.0.0 jobs: build: - machine: - image: ubuntu-1604:201903-01 - docker_layer_caching: true # default - false + docker: + - image: circleci/buildpack-deps:18.04 environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci -# docker: -# - image: circleci/buildpack-deps:stretch + REGISTRY: docker.pkg.github.com + CI_IMAGE: splunk/splunk-connect-for-syslog/ci steps: -# - setup_remote_docker: -# docker_layer_caching: true + - setup_remote_docker: + docker_layer_caching: true - checkout - - run: git submodule sync - run: git submodule update --init --recursive - - run: mkdir /tmp/artifacts - - run: docker pull gittools/gitversion:latest-linux-netcoreapp2.1 - - run: - name: Docker Login - command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - - run: - name: Build Docker image - command: docker build -f package/Dockerfile --build-arg RH_ORG=$RH_ORG --build-arg RH_ACTIVATION=$RH_ACTIVATION -t $IMAGE_NAME:$CIRCLE_SHA1 package - - run: - name: Push Docker image by hash - command: docker push $IMAGE_NAME:$CIRCLE_SHA1 + - run: mkdir /tmp/workspace + + - versioning/define_version + - versioning/create_version_file: + version_file_path: /tmp/workspace/VERSION + + - docker/install-docker + - docker/check: + registry: $REGISTRY + docker-username: GITHUB_USER + docker-password: GITHUB_TOKEN + - docker/build: + image: $CI_IMAGE + registry: $REGISTRY + path: package + extra_build_args: --build-arg RH_ORG=$RH_ORG --build-arg RH_ACTIVATION=$RH_ACTIVATION + + - docker/install-goss - run: - name: Docker tag and push with version label + name: Test Docker command: | - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - echo image $IMAGE_NAME:$SEMVER - docker tag $IMAGE_NAME:$CIRCLE_SHA1 $IMAGE_NAME:$SEMVER - docker push $IMAGE_NAME:$SEMVER + mkdir -p /tmp/test-results + # Don't forget path! + export PATH=$PATH:~/bin + pushd package + GOSS_FMT_OPTIONS="perfdata verbose" GOSS_FILES_STRATEGY=cp GOSS_OPTS="--max-concurrent=1 --format junit" dgoss run -t $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 3>&1 1>&2 2>&3 > /tmp/goss.tmp || true ; cat /tmp/goss.tmp | grep -v "<" + popd + cat /tmp/goss.tmp | grep "[<\>]" > /tmp/test-results/goss.xml + + - docker/push: + registry: $REGISTRY + image: $CI_IMAGE + - run: | + CIRCLE_BRANCH_CLEAN=$(echo ${CIRCLE_BRANCH} | sed 's/[^a-zA-Z0-9\._-]//g') + if [ -n "${CIRCLE_TAG}" ]; then DOCKER_TAG=${CIRCLE_TAG}; else DOCKER_TAG="${CIRCLE_BRANCH_CLEAN}"; fi + if [ -n "${CIRCLE_TAG}" ]; then VERSION=${CIRCLE_TAG}; else VERSION="${CIRCLE_BRANCH_CLEAN}@${CIRCLE_SHA1:0:6}"; fi + docker tag $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 $REGISTRY/$CI_IMAGE:$DOCKER_TAG + docker push $REGISTRY/$CI_IMAGE:$DOCKER_TAG - run: name: Docker Save command: | - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker save $IMAGE_NAME:$SEMVER | gzip -c > /tmp/artifacts/oci_container.tar.gz + docker save $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 | gzip -c > /tmp/workspace/oci_container.tar.gz - run: name: BYOE Config command: | - tar rvf /tmp/artifacts/baremetal.tar -C package etc - tar rvf /tmp/artifacts/baremetal.tar -C package/sbin entrypoint.sh + tar rvf /tmp/workspace/baremetal.tar -C package etc + tar rvf /tmp/workspace/baremetal.tar -C package/sbin entrypoint.sh - - store_artifacts: - path: /tmp/artifacts + - store_test_results: + path: /tmp/test-results/goss.xml - dgoss: - environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci - GOSS_VER: v0.3.7 - GOSS_OPTS: "--max-concurrent=1 --format junit" - GOSS_FILES_STRATEGY: cp + - persist_to_workspace: + root: /tmp + paths: + - workspace - docker: - - image: circleci/buildpack-deps:stretch - steps: - - checkout - - setup_remote_docker: - docker_layer_caching: true - - run: - name: Docker Login - command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - - run: - name: Docker pull - command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: - name: Install goss - command: | - # rather than give internet scripts SU rights, we install to local user bin and add to path - curl -fsSL https://goss.rocks/install | GOSS_VER=v0.3.7 sudo sh - goss -version - - run: - name: Test Docker - command: | - mkdir -p ~/project/test-results - # Don't forget path! - export PATH=$PATH:~/bin - pushd package - GOSS_FMT_OPTIONS="perfdata verbose" GOSS_FILES_STRATEGY=cp GOSS_OPTS="--max-concurrent=1 --format junit" dgoss run -t $IMAGE_NAME:$CIRCLE_SHA1 3>&1 1>&2 2>&3 > /tmp/goss.tmp || true ; cat /tmp/goss.tmp | grep -v "<" - popd - cat /tmp/goss.tmp | grep "[<\>]" > ~/project/test-results/goss.xml - - store_test_results: - path: test-results test-unit: - environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci docker: - - image: circleci/buildpack-deps:stretch + - image: circleci/buildpack-deps:18.04 + environment: + REGISTRY: docker.pkg.github.com + CI_IMAGE: splunk/splunk-connect-for-syslog/ci steps: - setup_remote_docker: docker_layer_caching: true - checkout - run: git submodule sync - run: git submodule update --init --recursive - - run: - name: Docker Login - command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - - run: - name: Docker pull - command: docker-compose pull + + - docker/install-docker-tools + - docker/check: + registry: $REGISTRY + docker-username: GITHUB_USER + docker-password: GITHUB_TOKEN - run: name: Setup test volume command: | - mkdir test-results + mkdir /tmp/test-results docker volume create sc4s-tests docker volume create sc4s-results @@ -136,6 +121,7 @@ jobs: - run: name: Docker Compose up command: | + export IMAGE_NAME=$CIRCLE_PROJECT_REPONAME docker-compose -f ./docker-compose-ci.yml up -d splunk sleep 70 docker-compose -f ./docker-compose-ci.yml up --abort-on-container-exit @@ -147,307 +133,253 @@ jobs: -v sc4s-results:/work/test-results \ registry.access.redhat.com/ubi7/ubi - docker cp dummy:/work/test-results/functional test-results + docker cp dummy:/work/test-results/functional /tmp/test-results docker rm dummy when: always - store_test_results: - path: test-results + path: /tmp/test-results + test-scan-synk: docker: - - image: 'circleci/buildpack-deps:stable' + - image: circleci/buildpack-deps:18.04 environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci + REGISTRY: docker.pkg.github.com + CI_IMAGE: splunk/splunk-connect-for-syslog/ci steps: - - checkout - - setup_remote_docker: - docker_layer_caching: true - - run: - name: Docker Login - command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - - run: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - snyk/scan: - docker-image-name: $IMAGE_NAME:$CIRCLE_SHA1 - test-scan-clair: + - setup_remote_docker: + docker_layer_caching: true + - checkout + - run: git submodule sync + - run: git submodule update --init --recursive + - docker/check: + registry: $REGISTRY + docker-username: GITHUB_USER + docker-password: GITHUB_TOKEN + - run: docker pull $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 + - snyk/scan: + docker-image-name: $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 + target-file: package/Dockerfile + project: '${CIRCLE_PROJECT_REPONAME}/${CIRCLE_BRANCH}-container' + + test-scan-synk-nomonitor: docker: - - image: 'docker:stable' + - image: circleci/buildpack-deps:18.04 environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci + REGISTRY: docker.pkg.github.com + CI_IMAGE: splunk/splunk-connect-for-syslog/ci steps: - - checkout - setup_remote_docker: docker_layer_caching: true - - run: - name: "Vulnerability scan" - command: | - - - store_artifacts: - path: clair-reports - + - checkout + - docker/check: + registry: $REGISTRY + docker-username: GITHUB_USER + docker-password: GITHUB_TOKEN + - run: docker pull $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 + - run: git submodule sync + - run: git submodule update --init --recursive + - snyk/scan: + docker-image-name: $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 + target-file: package/Dockerfile + project: '${CIRCLE_PROJECT_REPONAME}/${CIRCLE_BRANCH}-container' + monitor-on-build: false - publish-edge: - environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci - PUBLIC_IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/releases + publish-branch: docker: - #- image: circleci/buildpack-deps:stretch - - image: circleci/golang:latest + - image: circleci/buildpack-deps:18.04 + environment: + REGISTRY: docker.pkg.github.com + CI_IMAGE: splunk/splunk-connect-for-syslog/ci + REGISTRY_PUBLIC: docker.pkg.github.com + PUBLIC_IMAGE: splunk/splunk-connect-for-syslog/releases steps: - setup_remote_docker: docker_layer_caching: true - checkout - - run: - name: Create Directory - command: mkdir /tmp/artifacts - - run: - name: Docker Login - command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - - run: - name: Docker pull - command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: - name: Docker tag image - command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:edge - - run: - name: Docker push tag - command: docker push $PUBLIC_IMAGE_NAME:edge + - docker/check: + registry: $REGISTRY + docker-username: GITHUB_USER + docker-password: GITHUB_TOKEN + - run: docker pull $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 + - run: | + CIRCLE_BRANCH_CLEAN=$(echo ${CIRCLE_BRANCH} | sed 's/[^a-zA-Z0-9\._-]//g') + if [ -n "${CIRCLE_TAG}" ]; then DOCKER_TAG=${CIRCLE_TAG}; else DOCKER_TAG="${CIRCLE_BRANCH_CLEAN}"; fi + if [ -n "${CIRCLE_TAG}" ]; then VERSION=${CIRCLE_TAG}; else VERSION="${CIRCLE_BRANCH_CLEAN}@${CIRCLE_SHA1:0:6}"; fi + echo docker tag $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 -t $REGISTRY_PUBLIC/$PUBLIC_IMAGE:$DOCKER_TAG + docker tag $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 $REGISTRY_PUBLIC/$PUBLIC_IMAGE:$DOCKER_TAG + docker push $REGISTRY_PUBLIC/$PUBLIC_IMAGE:$DOCKER_TAG - - run: - name: Docker Save - command: | - docker save $IMAGE_NAME:$CIRCLE_SHA1 | gzip -c > /tmp/artifacts/oci_container.tar.gz - - run: - name: BYOE Config - command: | - tar rvf /tmp/artifacts/baremetal.tar -C package etc - tar rvf /tmp/artifacts/baremetal.tar -C package/sbin entrypoint.sh - - - run: - name: "Publish Release on GitHub" - command: | - go get -u github.com/tcnksm/ghr - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete edge /tmp/artifacts/ - - publish-version: - machine: - image: ubuntu-1604:201903-01 - docker_layer_caching: true # default - false + publish-edge: + docker: + - image: circleci/buildpack-deps:18.04 environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci - PUBLIC_IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/releases + REGISTRY: docker.pkg.github.com + CI_IMAGE: splunk/splunk-connect-for-syslog/ci + REGISTRY_PUBLIC: docker.pkg.github.com + PUBLIC_IMAGE: splunk/splunk-connect-for-syslog/releases steps: - + - setup_remote_docker: + docker_layer_caching: true - checkout - - - run: - name: Create Directory - command: mkdir /tmp/artifacts - - run: - name: Docker Login - command: | - docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - - run: - name: Docker tag image - command: | - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:$SEMVER - docker push $PUBLIC_IMAGE_NAME:$SEMVER - - - run: - name: Docker Save - command: | - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker save $PUBLIC_IMAGE_NAME:$SEMVER | gzip -c > /tmp/artifacts/oci_container.tar.gz - - run: - name: BYOE Config - command: | - tar rvf /tmp/artifacts/baremetal.tar -C package etc - tar rvf /tmp/artifacts/baremetal.tar -C package/sbin entrypoint.sh - - run: - name: "Publish Release on GitHub" + - docker/check: + registry: $REGISTRY + docker-username: GITHUB_USER + docker-password: GITHUB_TOKEN + - run: docker pull $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 + - run: docker tag $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 $REGISTRY/$PUBLIC_IMAGE:edge + - docker/push: + registry: $REGISTRY + image: $PUBLIC_IMAGE + tag: edge + - docker/check: + docker-username: DOCKER_USER + docker-password: DOCKER_PASS + - run: docker tag $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 splunk/scs:edge + - docker/push: + image: splunk/scs + tag: edge + - go/install + - attach_workspace: + at: /tmp + - run: + name: "Publish edge on GitHub" command: | - go get -u github.com/tcnksm/ghr - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${SEMVER} /tmp/artifacts/ - + PATH=$PATH:/usr/local/go/bin + go get -v -u github.com/tcnksm/ghr + $HOME/go/bin/ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -prerelease -delete edge /tmp/workspace/ + - store_artifacts: + path: /tmp/workspace/ publish-latest: - environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci - PUBLIC_IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/releases docker: - #- image: circleci/buildpack-deps:stretch - - image: circleci/golang:latest + - image: circleci/buildpack-deps:18.04 + environment: + REGISTRY: docker.pkg.github.com + CI_IMAGE: splunk/splunk-connect-for-syslog/ci + REGISTRY_PUBLIC: docker.pkg.github.com + PUBLIC_IMAGE: splunk/splunk-connect-for-syslog/releases + steps: - setup_remote_docker: docker_layer_caching: true - checkout - - run: - name: Create Directory - command: mkdir /tmp/artifacts - - run: - name: Docker Login - command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - - run: - name: Docker pull - command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: - name: Docker tag image - command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:latest - - run: - name: Docker push tag - command: docker push $PUBLIC_IMAGE_NAME:latest + - docker/check: + registry: $REGISTRY + docker-username: GITHUB_USER + docker-password: GITHUB_TOKEN + - run: docker pull $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 + - run: docker tag $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 $REGISTRY/$PUBLIC_IMAGE:latest + - docker/push: + registry: $REGISTRY + image: $PUBLIC_IMAGE + tag: latest + - docker/check: + docker-username: DOCKER_USER + docker-password: DOCKER_PASS + - run: docker tag $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 splunk/scs:latest + - docker/push: + image: splunk/scs + tag: latest + - go/install + - attach_workspace: + at: /tmp - run: - name: Docker Save + name: "Publish edge on GitHub" command: | - docker save $IMAGE_NAME:$CIRCLE_SHA1 | gzip -c > /tmp/artifacts/oci_container.tar.gz - - run: - name: BYOE Config - command: | - tar rvf /tmp/artifacts/baremetal.tar -C package etc - tar rvf /tmp/artifacts/baremetal.tar -C package/sbin entrypoint.sh - - - run: - name: "Publish Release on GitHub" - command: | - go get -u github.com/tcnksm/ghr - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete latest /tmp/artifacts/ - - - dockerhub-version: - machine: - image: ubuntu-1604:201903-01 - docker_layer_caching: true # default - false - environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci - PUBLIC_IMAGE_NAME: splunk/scs - steps: - - - checkout - - - run: - name: Create Directory - command: mkdir artifacts - - run: - name: Docker Login - command: | - docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: - name: Docker tag image - command: | - SEMVER=$(docker run --rm -v "$(pwd):/repo" gittools/gitversion:latest-linux-netcoreapp2.1 /repo /showvariable SemVer /nofetch) - docker login -u $DOCKER_USER -p $DOCKER_PASS - docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:$SEMVER - docker push $PUBLIC_IMAGE_NAME:$SEMVER + PATH=$PATH:/usr/local/go/bin + go get -v -u github.com/tcnksm/ghr + $HOME/go/bin/ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete latest /tmp/workspace/ + - store_artifacts: + path: /tmp/workspace/ - dockerhub-edge: - environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci - PUBLIC_IMAGE_NAME: splunk/scs + publish-tag: docker: - - image: circleci/buildpack-deps:stretch - steps: - - setup_remote_docker: - docker_layer_caching: true - - run: - name: Create Directory - command: mkdir /tmp/artifacts - - run: - name: Docker Login - command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - - run: - name: Docker pull - command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: - name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS - - run: - name: Docker tag image - command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:edge - - run: - name: Docker push tag - command: docker push $PUBLIC_IMAGE_NAME:edge - - dockerhub-latest: + - image: circleci/buildpack-deps:18.04 environment: - IMAGE_NAME: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci - PUBLIC_IMAGE_NAME: splunk/scs - docker: - - image: circleci/buildpack-deps:stretch + REGISTRY: docker.pkg.github.com + CI_IMAGE: splunk/splunk-connect-for-syslog/ci + REGISTRY_PUBLIC: docker.pkg.github.com + PUBLIC_IMAGE: splunk/splunk-connect-for-syslog/releases + steps: - setup_remote_docker: docker_layer_caching: true - - run: - name: Create Directory - command: mkdir /tmp/artifacts - - run: - name: Docker Login - command: docker login docker.pkg.github.com --username $GITHUB_USER --password $GITHUB_TOKEN - - run: - name: Docker pull - command: docker pull $IMAGE_NAME:$CIRCLE_SHA1 - - run: - name: Docker Login - command: docker login -u $DOCKER_USER -p $DOCKER_PASS - - run: - name: Docker tag image - command: docker tag $IMAGE_NAME:$CIRCLE_SHA1 $PUBLIC_IMAGE_NAME:latest - - run: - name: Docker push tag - command: docker push $PUBLIC_IMAGE_NAME:latest + - checkout + - docker/check: + registry: $REGISTRY + docker-username: GITHUB_USER + docker-password: GITHUB_TOKEN + - run: docker pull $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 + - run: | + CIRCLE_BRANCH_CLEAN=$(echo ${CIRCLE_BRANCH} | sed 's/[^a-zA-Z0-9\._-]//g') + if [ -n "${CIRCLE_TAG}" ]; then DOCKER_TAG=${CIRCLE_TAG}; else DOCKER_TAG="${CIRCLE_BRANCH_CLEAN}"; fi + if [ -n "${CIRCLE_TAG}" ]; then VERSION=${CIRCLE_TAG}; else VERSION="${CIRCLE_BRANCH_CLEAN}@${CIRCLE_SHA1:0:6}"; fi + docker tag $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 $REGISTRY/$PUBLIC_IMAGE:$DOCKER_TAG + docker push $REGISTRY/$PUBLIC_IMAGE:$DOCKER_TAG + - docker/check: + docker-username: DOCKER_USER + docker-password: DOCKER_PASS + - run: | + CIRCLE_BRANCH_CLEAN=$(echo ${CIRCLE_BRANCH} | sed 's/[^a-zA-Z0-9\._-]//g') + if [ -n "${CIRCLE_TAG}" ]; then DOCKER_TAG=${CIRCLE_TAG}; else DOCKER_TAG="${CIRCLE_BRANCH_CLEAN}"; fi + if [ -n "${CIRCLE_TAG}" ]; then VERSION=${CIRCLE_TAG}; else VERSION="${CIRCLE_BRANCH_CLEAN}@${CIRCLE_SHA1:0:6}"; fi + docker tag $REGISTRY/$CI_IMAGE:$CIRCLE_SHA1 splunk/scs:$DOCKER_TAG + docker push splunk/scs:$DOCKER_TAG + + - go/install + - attach_workspace: + at: /tmp + - run: + name: "Publish edge on GitHub" + command: | + PATH=$PATH:/usr/local/go/bin + go get -v -u github.com/tcnksm/ghr + CIRCLE_BRANCH_CLEAN=$(echo ${CIRCLE_BRANCH} | sed 's/[^a-zA-Z0-9\._-]//g') + if [ -n "${CIRCLE_TAG}" ]; then DOCKER_TAG=${CIRCLE_TAG}; else DOCKER_TAG="${CIRCLE_BRANCH_CLEAN}"; fi + if [ -n "${CIRCLE_TAG}" ]; then VERSION=${CIRCLE_TAG}; else VERSION="${CIRCLE_BRANCH_CLEAN}@${CIRCLE_SHA1:0:6}"; fi + $HOME/go/bin/ghr -t ${GITHUB_TOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete $VERSION /tmp/workspace/ + - store_artifacts: + path: /tmp/workspace/ workflows: version: 2 build-branches: jobs: - build - - dgoss: - requires: - - build - test-unit: requires: - build - test-scan-synk: requires: - build -#Clair scanner image is broken using synk for now -# - test-scan-clair: -# requires: -# - build - - publish-edge: - requires: - - dgoss - - test-unit filters: branches: only: - - develop - - publish-latest: + - master + - test-scan-synk-nomonitor: requires: - - dgoss - - test-unit + - build filters: branches: - only: + ignore: - master - - dockerhub-edge: + - publish-branch: + requires: + - build + - publish-edge: requires: - - dgoss + - build - test-unit filters: branches: only: - develop - - dockerhub-latest: + - publish-latest: requires: - - dgoss + - build - test-unit filters: branches: @@ -461,14 +393,6 @@ workflows: only: /^\d*\.\d*\.\d*.*$/ branches: ignore: /.*/ - - dgoss: - filters: - tags: - only: /^\d*\.\d*\.\d*.*$/ - branches: - ignore: /.*/ - requires: - - build - test-unit: filters: tags: @@ -485,21 +409,13 @@ workflows: ignore: /.*/ requires: - build - - publish-version: - filters: - tags: - only: /^\d*\.\d*\.\d*.*$/ - branches: - ignore: /.*/ + - publish-tag: requires: - - dgoss + - build - test-unit - - dockerhub-version: + - test-scan-synk filters: tags: only: /^\d*\.\d*\.\d*.*$/ branches: - ignore: /.*/ - requires: - - dgoss - - test-unit \ No newline at end of file + ignore: /.*/ \ No newline at end of file diff --git a/docker-compose-ci.yml b/docker-compose-ci.yml index b4bb66d..6028e6e 100644 --- a/docker-compose-ci.yml +++ b/docker-compose-ci.yml @@ -19,7 +19,7 @@ services: - SPLUNK_PASSWORD=${SPLUNK_PASSWORD} sc4s: - image: ${IMAGE_NAME}:${CIRCLE_SHA1} + image: ${REGISTRY}/${CI_IMAGE}:${CIRCLE_SHA1} hostname: sc4s ports: - "514" From 158ede7ab6cb73fca0efa33722cc427ab6fb8625 Mon Sep 17 00:00:00 2001 From: mbonsack Date: Wed, 9 Oct 2019 16:04:01 -0700 Subject: [PATCH 67/67] Docs/mount point (#131) Update docs for single mountpoint --- docs/gettingstarted/docker-swarm-general.md | 52 +++++++++++-------- docs/gettingstarted/docker-swarm-rhel7.md | 45 ++++++++++------ docs/gettingstarted/docker-systemd-general.md | 48 ++++++++++------- docs/gettingstarted/podman-systemd-general.md | 45 ++++++++++------ 4 files changed, 118 insertions(+), 72 deletions(-) diff --git a/docs/gettingstarted/docker-swarm-general.md b/docs/gettingstarted/docker-swarm-general.md index f9a6e12..91d56e7 100644 --- a/docs/gettingstarted/docker-swarm-general.md +++ b/docs/gettingstarted/docker-swarm-general.md @@ -28,14 +28,23 @@ services: env_file: - /opt/sc4s/env_file volumes: - - /opt/sc4s/default/splunk_index.csv:/opt/syslog-ng/etc/context-local/splunk_index.csv - - /opt/sc4s/default/vendor_product_by_source.csv:/opt/syslog-ng/etc/context-local/vendor_product_by_source.csv - - /opt/sc4s/default/vendor_product_by_source.conf:/opt/syslog-ng/etc/context-local/vendor_product_by_source.conf + - /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local #Uncomment the following line if custom TLS certs are provided - - /opt/sc4s/tls:/opt/syslog-ng/tls +# - /opt/sc4s/tls:/opt/syslog-ng/tls ``` -* NOTE: If you use the default `volumes` declarations as-is from the `docker-compose.yml` file template example, you must create and/or download all files and directories referenced in the file according to the configuration steps that follow. The TLS-specific options are described in the "Configure the sc4s Environment" section. Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors. +* Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations (below). + +* NOTE: The empty ``local`` directory created above will populate with templates at the first invocation +of SC4S for local configurations and overrides. Changes made to these files will be preserved on subsequent +restarts (i.e. a "no-clobber" copy is performed for any missing files). _Do not_ change the directory structure of +the files that are laid down; change (or add) only individual files if desired. SC4S depends on the directory layout +to read the local configurations properly. + +* NOTE: You can back up the contents of this directory elsewhere and return the directory to an empty state +when a new version of SC4S is released to pick up any new changes provided by Splunk. Upon a restart, +the direcory will populate as it did when you first installed SC4S. Your previous changes can then +be merged back in and will take effect after another restart. ## Configure the SC4S environment @@ -60,30 +69,31 @@ match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example above. -## Configure index destinations for Splunk +## Modify index destinations for Splunk -Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Create a subdirectory called ``default`` in the directory that you created in the previous step (e.g. ``/opt/sc4s/``). Make sure the local directory volume references in the `yml` file match the directory you create here. From this directory, -execute the command below to download the index context file: +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. -```bash -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv -``` -* Edit splunk_index.csv to review the index configuration and revise as required for the sourcetypes utilized in your environment. - -## Configure sources by source IP or host name +## Configure source filtering by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* If not already done, create a subdirectory called ``default`` in the ``/opt/sc4s/`` directory. Make sure the local directory volume references in the `yml` file match the directory you create here. From this directory, execute the following commands to download the vendor context files: +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. -```bash -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv -``` -* If you have legacy sources and non-standard-compliant sources, edit the file to properly identify these products by host glob or network mask using syslog-ng filter syntax. +## Configure compliance index/metadata overrides + +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in +the "Configuration" section. ## Start/Restart SC4S diff --git a/docs/gettingstarted/docker-swarm-rhel7.md b/docs/gettingstarted/docker-swarm-rhel7.md index 008dd65..a9527b8 100644 --- a/docs/gettingstarted/docker-swarm-rhel7.md +++ b/docs/gettingstarted/docker-swarm-rhel7.md @@ -63,8 +63,18 @@ services: - /opt/sc4s/tls:/opt/syslog-ng/tls ``` -* NOTE: If you use the default `volumes` declarations as-is from the `docker-compose.yml` file template example, do create and/or download all files and directories referenced in the file according to the configuration steps that follow. The TLS-specific options are described in the "Configure the sc4s environment" section. Failure to match the volume specification in the `yml` file with what exists locally will result in startup errors. +* Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations (below). +* NOTE: The empty ``local`` directory created above will populate with templates at the first invocation +of SC4S for local configurations and overrides. Changes made to these files will be preserved on subsequent +restarts (i.e. a "no-clobber" copy is performed for any missing files). _Do not_ change the directory structure of +the files that are laid down; change (or add) only individual files if desired. SC4S depends on the directory layout +to read the local configurations properly. + +* NOTE: You can back up the contents of this directory elsewhere and return the directory to an empty state +when a new version of SC4S is released to pick up any new changes provided by Splunk. Upon a restart, +the direcory will populate as it did when you first installed SC4S. Your previous changes can then +be merged back in and will take effect after another restart. ## Configure the SC4S environment @@ -90,31 +100,32 @@ match this value to the total number of indexers behind the load balancer. uncomment the last line in the example below. +## Modify index destinations for Splunk -## Configure index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Create a subdirectory called ``default`` in the directory that you created in the previous step (e.g. ``/opt/sc4s/``). Make sure the local directory volume references in the `yml` file match the directory you create here. From this directory, -execute the command below to download the index context file: +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. -```bash -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv -``` -* Edit splunk_index.csv to review the index configuration and revise as required for the sourcertypes utilized in your environment. -## Configure sources by source IP or host name +## Configure source filtering by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* If not already done, create a subdirectory called ``default`` in the ``/opt/sc4s/`` directory. Make sure the local directory volume references in the `yml` file match the directory you create here. From this directory, execute the following commands to download the vendor context files: +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. -```bash -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv -``` -* If you have legacy sources and non-standard-compliant sources, edit the file to properly identify these products by host glob or network mask using syslog-ng filter syntax. +## Configure compliance index/metadata overrides + +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in +the "Configuration" section. ## Start/Restart SC4S diff --git a/docs/gettingstarted/docker-systemd-general.md b/docs/gettingstarted/docker-systemd-general.md index 0e1fc58..05cdf59 100644 --- a/docs/gettingstarted/docker-systemd-general.md +++ b/docs/gettingstarted/docker-systemd-general.md @@ -42,7 +42,18 @@ ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp \ $SC4S_IMAGE ``` -* NOTE: If you use the default `Environment` assignments as-is from the `sc4s.service` unit file template example, do create and/or download all files and directories referenced in the file's Service stanza according to the configuration steps that follow. The TLS-specific options are described in the "Configure the sc4s environment" section. +* Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations (below). + +* NOTE: The empty ``local`` directory created above will populate with templates at the first invocation +of SC4S for local configurations and overrides. Changes made to these files will be preserved on subsequent +restarts (i.e. a "no-clobber" copy is performed for any missing files). _Do not_ change the directory structure of +the files that are laid down; change (or add) only individual files if desired. SC4S depends on the directory layout +to read the local configurations properly. + +* NOTE: You can back up the contents of this directory elsewhere and return the directory to an empty state +when a new version of SC4S is released to pick up any new changes provided by Splunk. Upon a restart, +the direcory will populate as it did when you first installed SC4S. Your previous changes can then +be merged back in and will take effect after another restart. ## Configure the SC4S environment @@ -67,30 +78,31 @@ match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example. +## Modify index destinations for Splunk -## Configure index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that is suitable for most customers. +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the `sc4s.service` unit file match the directory you create here (the ``-v`` variables). From this directory, execute the following to download the latest index context file: +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. -```bash -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv -``` -* Edit splunk_index.csv to review the index configuration and revise as required for the sourcertypes utilized in your environment. - -## Configure sources by source IP or host name +## Configure source filtering by source IP or host name Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step refer to the "sources" section of this documentation. +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -* If not already done, create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the `sc4s.service` unit file match the directory you create here (the ``-v`` variables). From this directory, execute the following to download the latest vendor context files: +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. -```bash -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv -``` -* If you have legacy sources and non-standard-compliant sources, edit the file to properly identify these products by host glob or network mask using syslog-ng filter syntax. +## Configure compliance index/metadata overrides + +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in +the "Configuration" section. ## Configure SC4S for systemd and start SC4S diff --git a/docs/gettingstarted/podman-systemd-general.md b/docs/gettingstarted/podman-systemd-general.md index 361ac12..2eff77c 100644 --- a/docs/gettingstarted/podman-systemd-general.md +++ b/docs/gettingstarted/podman-systemd-general.md @@ -42,7 +42,18 @@ ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp \ $SC4S_IMAGE ``` -* NOTE: If you use the default `Environment` assignments as-is from the `sc4s.service` unit file template example, do create and/or download all files and directories referenced in the file's Service stanza according to the configuration steps that follow. The TLS-specific options are described in the "Configure the sc4s environment" section. +* Create the subdirectory ``/opt/sc4s/local``. This will be used as a mount point for local overrides and configurations (below). + +* NOTE: The empty ``local`` directory created above will populate with templates at the first invocation +of SC4S for local configurations and overrides. Changes made to these files will be preserved on subsequent +restarts (i.e. a "no-clobber" copy is performed for any missing files). _Do not_ change the directory structure of +the files that are laid down; change (or add) only individual files if desired. SC4S depends on the directory layout +to read the local configurations properly. + +* NOTE: You can back up the contents of this directory elsewhere and return the directory to an empty state +when a new version of SC4S is released to pick up any new changes provided by Splunk. Upon a restart, +the direcory will populate as it did when you first installed SC4S. Your previous changes can then +be merged back in and will take effect after another restart. ## Configure the sc4s environment @@ -67,29 +78,31 @@ match this value to the total number of indexers behind the load balancer. * NOTE: Splunk Connect for Syslog defaults to secure configurations. If you are not using trusted SSL certificates, be sure to uncomment the last line in the example. - -## Configure index destinations for Splunk +## Modify index destinations for Splunk Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. -* Create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the `sc4s.service` unit file match the directory you create here (the ``-v`` variables). From this directory, execute the following to download the latest index context file: +* If changes need to be made to index destinations, navigate to the ``/opt/sc4s/local/context`` directory to start. +* Edit `splunk_index.csv` to review or change the index configuration and revise as required for the data sources utilized in your +environment. Simply uncomment the relevant line and enter the desired index. The "Sources" document details the specific entries in +this table that pertain to the individual data source filters that are included with SC4S. -```bash -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/splunk_index.csv -``` -* Edit splunk_index.csv to review the index configuration and revise as required for the sourcertypes utilized in your environment. +## Configure source filtering by source IP or host name -## Configure sources by source IP or host name +Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps +apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. +* Navigate to `vendor_product_by_source.conf` and find the appropriate filter that matches your legacy device type. +* Edit the file to properly identify these products by hostname glob or network mask using syslog-ng filter syntax. Configuration by hostname or source IP is needed only for those devices that cannot be determined via normal syslog-ng parsing or message contents. +* The `vendor_product_by_source.csv` file should not need to be changed unless a local filter is created that is specific to the environment. In this case, a matching filter will also need to be provided in `vendor_product_by_source.conf`. -* If not already done, create a directory (e.g. ``/opt/sc4s/default/`` ). Make sure the local directory references in the `sc4s.service` unit file match the directory you create here (the ``-v`` variables). From this directory, execute the following to download the latest vendor context files: +## Configure compliance index/metadata overrides -```bash -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.conf -sudo wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context-local/vendor_product_by_source.csv -``` -* If you have legacy sources and non-standard-compliant sources, edit the file to properly identify these products by host glob or network mask using syslog-ng filter syntax. +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. +The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to +the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is in +the "Configuration" section. ## Configure SC4S for systemd and start SC4S