Skip to content

Commit

Permalink
Merge branch 'develop' into mcafee-extract-source
Browse files Browse the repository at this point in the history
  • Loading branch information
Ryan Faircloth authored and GitHub committed Jun 15, 2020
2 parents d002666 + dea2776 commit 365b583
Show file tree
Hide file tree
Showing 61 changed files with 369 additions and 285 deletions.
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ separately from that of the alternates below.
| Variable | Values | Description |
|----------|---------------|-------------|
| SC4S_DEST_GLOBAL_ALTERNATES | Comma or space-separated list of syslog-ng destinations | Send all sources to alternate destinations |
| SC4S_DEST_<SOURCE>_ALTERNATES | Comma or space-separated list of syslog-ng destiinations | Send specific sources to alternate syslog-ng destinations, e.g. SC4S_DEST_CISCO_ASA_ALTERNATES |
| SC4S_DEST_<VENDOR_PRODUCT>_ALTERNATES | Comma or space-separated list of syslog-ng destinations | Send specific sources to alternate syslog-ng destinations using the VENDOR_PRODUCT syntax, e.g. SC4S_DEST_CISCO_ASA_ALTERNATES |

## SC4S Disk Buffer Configuration

Expand Down
6 changes: 1 addition & 5 deletions docs/gettingstarted/docker-systemd-general.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,6 @@ Restart=always

ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE
ExecStartPre=/usr/bin/bash -c "/usr/bin/systemctl set-environment SC4SHOST=$(hostname -s)"
ExecStartPre=/usr/bin/docker run \
--env-file=/opt/sc4s/env_file \
"$SC4S_LOCAL_CONFIG_MOUNT" \
--name SC4S_preflight \
--rm $SC4S_IMAGE -s
ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 \
-e "SC4S_CONTAINER_HOST=${SC4SHOST}" \
--env-file=/opt/sc4s/env_file \
Expand All @@ -68,6 +63,7 @@ ExecStart=/usr/bin/docker run -p 514:514 -p 514:514/udp -p 6514:6514 \
"$SC4S_TLS_DIR" \
--name SC4S \
--rm $SC4S_IMAGE
Restart=on-success
```

* Execute the following command to create a local volume that will contain the disk buffer files in the event of a communication
Expand Down
14 changes: 7 additions & 7 deletions docs/gettingstarted/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ using the SC4S defaults. SC4S can be easily customized to use different indexes
* netipam
* oswinsec
* osnix
* em_metrics (ensure this is created as a metrics index)
* em_metrics (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index)

#### Install Related Splunk Apps

Expand Down Expand Up @@ -102,13 +102,13 @@ which has proven inadequate for many.

#### Select a Container Runtime and SC4S Configuration

| Container and Orchestration | Notes |
| Container Runtime and Orchestration | Operating Systems |
|-----------------------------|-------|
| [Podman + systemd](podman-systemd-general.md) | First choice for RedHat 8.x and CentOS, second choice for Debian and Ubuntu (packages provided via PPA). |
| [Docker CE + systemd](docker-systemd-general.md) | First choice for RHEL/CentOS 7.x, Debian and Ubuntu |
| [Docker CE + Swarm](docker-swarm-general.md) | Option for Debian, Ubuntu, CentOS, and Desktop Docker desiring Docker Compose or Swarm orchestration |
| [Docker CE + Swarm RHEL 7.7](docker-swarm-rhel7.md) | Option for RedHat 7.7 desiring Docker Compose or Swarm orchestration |
| [Bring your own Envionment](byoe-rhel7.md) | Option for RedHat 7.7 (centos 7) with SC4S configuration without containers |
| [Podman 1.7 & 1.9 + systemd](podman-systemd-general.md) | RHEL or CentOS 8.1 & 8.2 (best option), Debian or Ubuntu 18.04LTS |
| [Docker CE 18 & 19 + systemd](docker-systemd-general.md) | RHEL or CentOS 7.7 (best option), Debian or Ubuntu 18.04LTS |
| [Docker CE 18 & 19 + Swarm](docker-swarm-general.md) | CentOS 7.7 (best option), Debian or Ubuntu 18.04LTS |
| [Docker CE 18 & 19 + Swarm](docker-swarm-rhel7.md) | RHEL 7.7 |
| [Bring your own Envionment](byoe-rhel7.md) | RHEL or CentOS 8.1 & 8.2 (best option) |

### Offline Container Installation

Expand Down
6 changes: 1 addition & 5 deletions docs/gettingstarted/podman-systemd-general.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,6 @@ Restart=always

ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE
ExecStartPre=/usr/bin/bash -c "/usr/bin/systemctl set-environment SC4SHOST=$(hostname -s)"
ExecStartPre=/usr/bin/podman run \
--env-file=/opt/sc4s/env_file \
"$SC4S_LOCAL_CONFIG_MOUNT" \
--name SC4S_preflight \
--rm $SC4S_IMAGE -s
ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 \
-e "SC4S_CONTAINER_HOST=${SC4SHOST}" \
--env-file=/opt/sc4s/env_file \
Expand All @@ -87,6 +82,7 @@ ExecStart=/usr/bin/podman run -p 514:514 -p 514:514/udp -p 6514:6514 \
--name SC4S \
--rm $SC4S_IMAGE
ExecStartPost=sleep 2 ; conntrack -D -p udp
Restart=on-success
```

* Execute the following command to create a local volume that will contain the disk buffer files in the event of a communication
Expand Down
6 changes: 3 additions & 3 deletions docs/sources/Cisco/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -405,9 +405,9 @@ Verify timestamp, and host values match as expected

| key | sourcetype | index | notes |
|----------------|----------------|----------------|----------------|
| cisco_wsa_l4tm | cisco:wsa:l4tm | netops | None |
| cisco_wsa_squid | cisco:wsa:squid | netops | None |
| cisco_wsa_squid_new | cisco:wsa:squid:new | netops | None |
| cisco_wsa | cisco:wsa:l4tm | netproxy | None |
| cisco_wsa | cisco:wsa:squid | netproxy | None |
| cisco_wsa | cisco:wsa:squid:new | netproxy | None |

### Filter type

Expand Down
8 changes: 3 additions & 5 deletions docs/sources/Citrix/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,9 @@ MSG Parse: This filter parses message content

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CITRIX_NETSCALER_SPLUNK_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the port number defined |
| SC4S_LISTEN_CITRIX_NETSCALERSPLUNK_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the port number defined |
| SC4S_ARCHIVE_CITRIX_NETSCALER_SPLUNK | no | Enable archive to disk for this specific source |
| SC4S_DEST_CITRIX_NETSCALER_SPLUNK_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |
| SC4S_DEST_CITRIX_NETSCALER_SPLUNK_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |
| SC4S_LISTEN_CITRIX_NETSCALER_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the port number defined |
| SC4S_LISTEN_CITRIX_NETSCALER_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the port number defined |
| SC4S_DEST_CITRIX_NETSCALER_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |
| SC4S_SOURCE_CITRIX_NETSCALER_USEALT_DATE_FORMAT | no | Use "DDMMYYYY" format rather than "MMDDYYYY" |

### Verification
Expand Down
51 changes: 51 additions & 0 deletions docs/sources/Splunk/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Vendor - Splunk


## Product - Splunk Connect for Syslog (SC4S)

| Ref | Link |
|----------------|---------------------------------------------------------------------------------------------------------|
| Splunk Add-on | https://splunkbase.splunk.com/app/4740/ |
| Product Manual | https://splunk-connect-for-syslog.readthedocs.io/en/master/ |


### Sourcetypes

| sourcetype | notes |
|----------------|---------------------------------------------------------------------------------------------------------|
| sc4s:events | Internal events from the SC4S container and underlying syslog-ng process |
| sc4s:metrics | syslog-ng operational metrics that will be delivered directly to a metrics index in Splunk |

### Sourcetype and Index Configuration

| key | sourcetype | index | notes |
|----------------|----------------|----------------|----------------|
| sc4s_events | all | main | none |
| sc4s_metrics | all | em_metrics | none |

### Filter type

SC4S events and metrics are generated automatically and no specific ports or filters need to be configured for the collection of this data.

### Setup and Configuration

* No specific requirements are required for the collection of sc4s internal events.
* Metrics data is _not_ collected by default; it is an opt-in set by the variable `SC4S_DEST_SC4S_METRICS_HEC`. See the "Options"
section below for details.

### Options

| Variable | default | description |
|-----------------------------------|-----------|----------------|
| SC4S_DEST_SPLUNK_SC4S_EVENTS_HEC | no | When Splunk HEC is disabled globally set to "yes" to enable this specific source |
| SC4S_DEST_SPLUNK_SC4S_METRICS_HEC | no | Set to "yes" to send metrics via HEC to Splunk (opt-in). Metrics are _not_ enabled by default when HEC is enabled globally. |

### Verification

SC4S will generate versioning events at startup. These startup events can be used to validate HEC is set up properly on the Splunk side.

```
index=<asconfigured> sourcetype=sc4s:events | stats count by host
```
Metrics can be observed via the "Analytics-->Metrics" navigation in the Search and Reporting app in Splunk.
* NOTE: The presentation of metrics is undergoing active development; the delivery of metrics is currently considered an experimental feature.
17 changes: 5 additions & 12 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,19 +89,12 @@ and navigating the syslog-ng config filesystem directly. To do this, run
/usr/bin/podman exec -it SC4S /bin/bash
```
and navigate to `/opt/syslog-ng/etc/` to see the actual config files in use. If you are adept with container operations and syslog-ng
itself, you can also modify files directly and reload syslog-ng with the command `kill -1 1` in the container. This is an advanced topic
and futher help can be obtained via the github issue tracker and Slack channels.
itself, you can modify files directly and reload syslog-ng with the command `kill -1 1` in the container.
You can also run the `/entrypoint.sh` script by hand (or a subset of it, such as everything
but syslog-ng) and have complete control over the templating and underlying syslog-ng process.
This is an advanced topic and futher help can be obtained via the github issue tracker and Slack channels.

## Run the container with a null entrypoint (Advanced!)

You can run the container without the usual entrypoint shell script by executing this command (modified to suit your environment):

```bash
/usr/bin/podman run -p 514:514 -p 514:514/udp -p 5000-5020:5000-5020 -p 5000-5020:5000-5020/udp --entrypoint=tail --env-file=/opt/sc4s/env_file -v /opt/sc4s/local:/opt/syslog-ng/etc/conf.d/local:z --name SC4S --rm splunk/scs:latest -f /dev/null
```
From there, you can "exec" into the container (above) and run the `/entrypoint.sh` script by hand (or a subset of it, such as everything
but syslog-ng) and have complete control over the templating and underlying syslog-ng process. Again, this is an advanced topic but can be
very useful for low-level troubleshooting.
When debugging a configuration syntax issue at startup the container must remain running. This can be enabled by adding `SC4S_DEBUG_CONTAINER=yes` to the `env_file`.

## Dealing with non RFC-5424 compliant sources

Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ nav:
- "pfSense": sources/Pfsense/index.md
- Proofpoint: sources/Proofpoint/index.md
- Schneider: sources/Schneider/index.md
- Splunk: sources/Splunk/index.md
- Symantec: sources/Symantec/index.md
- Ubiquiti: sources/Ubiquiti/index.md
- VMware: sources/VMWare/index.md
Expand Down
4 changes: 1 addition & 3 deletions package/etc/conf.d/conflib/_splunk/splunkfields.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,13 @@ rewrite r_set_splunk_default {
};
{{- end}}
};
#used by each log-path to set index and sourcetype which may be
#used by each log-path to set source and sourcetype which may be
#overridden by user defined values
block rewrite r_set_splunk_dest_default(
index()
source("${.splunk.source}")
sourcetype()
template(`splunk-template`)
) {
set("`index`", value(".splunk.index"));
set("`source`", value(".splunk.source"));
set("`sourcetype`", value(".splunk.sourcetype"));
};
Expand Down
1 change: 1 addition & 0 deletions package/etc/conf.d/context/common_event_format_source.csv
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
ArcSight_ArcSight,source,ArcSight:ArcSight
ArcSight_ArcSight,index,main
Carbon Black_Protection,sourcetype,carbonblack:protection:cef
Carbon Black_Protection,index,cb:cef
Cyber-Ark_Vault,sourcetype,cyberark:epv:cef
Expand Down
2 changes: 1 addition & 1 deletion package/etc/conf.d/filters/cisco/cisco_syslog.conf
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ parser cisco-parser-ex{
} elif {
#Cisco IOS Other
filter {
message('^<\d*> ?(?:(\d+)\: )?(?:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9]): )?(?:(\d+): )?(?:(\d\d:\d\d:\d\d|\d{1,6} \d{1,2}))?(?:(\*|\.)?((?:\w\w\w {1,2}\d{1,2} (?:\d{2,4} )?\d\d:\d\d:\d\d)(?:\.\d{3,6})?( [AP]M)?)( [A-Za-z]{3,3} )?)? ?((?:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*(?:[A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9]))? ?: ((\%[^\: ]+)\:? ?.*)'
message('^<\d*> ?(?:(\d+)\: )?(?:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9]): )?(?:(\d+): )?(?:(\d\d:\d\d:\d\d|\d{1,6} \d{1,2}))?(?:(\*|\.)?(?:20\d\d )?((?:\w\w\w {1,2}\d{1,2} (?:\d{2,4} )?\d\d:\d\d:\d\d)(?:\.\d{3,6})?( [AP]M)?)( [A-Za-z]{3,3} )?)? ?((?:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*(?:[A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9]))? ?: (last message repeated \d* times|(\%[^\: ]+)\:? ?.*)'
flags(store-matches));
};

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ log {

rewrite {
set("local_example", value("fields.sc4s_vendor_product"));
r_set_splunk_dest_default(sourcetype("sc4s:local_example"), index("main"));
r_set_splunk_dest_default(sourcetype("sc4s:local_example"));
};

# using the key "local_example" find any customized index,source or sourcetype meta values
Expand Down
2 changes: 1 addition & 1 deletion package/etc/conf.d/log_paths/lp-bbb-ietf_syslog.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ log {
set("IETF_SYSLOG", value("fields.sc4s_vendor_product"));
};

rewrite { r_set_splunk_dest_default(sourcetype("ietf:syslog"), index("main"), source("${APP}:${PROGRAM}")) };
rewrite { r_set_splunk_dest_default(sourcetype("ietf:syslog"), source("${APP}:${PROGRAM}")) };
parser { p_add_context_splunk(key("IETF_SYSLOG")); };
parser (compliance_meta_by_source);
rewrite { set("$(template ${.splunk.sc4s_template} $(template t_JSON_5424))" value("MSG")); };
Expand Down
2 changes: 1 addition & 1 deletion package/etc/conf.d/log_paths/lp-brocade.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ log {
subst("^[^\t]+\t", "", value("MESSAGE"), flags("global"));
set("${PROGRAM}", value(".PROGRAM"));
subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM"));
r_set_splunk_dest_default(sourcetype("brocade:syslog"), index("netops"), source("program:${.PROGRAM}"))
r_set_splunk_dest_default(sourcetype("brocade:syslog"), source("program:${.PROGRAM}"))
};
parser { p_add_context_splunk(key("brocade_syslog")); };

Expand Down
20 changes: 10 additions & 10 deletions package/etc/conf.d/log_paths/lp-checkpoint_splunk.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ log {
set("${.kv.hostname}", value("HOST"));
set("${.kv.hostname}", value("fields.cp_lm"));
set("checkpoint_splunk", value("fields.sc4s_vendor_product"));
r_set_splunk_dest_default(sourcetype("cp_log"), index("netops"))
r_set_splunk_dest_default(sourcetype("cp_log"))
};

if {
Expand Down Expand Up @@ -89,31 +89,31 @@ log {

if {
filter(f_checkpoint_splunk_NetworkTraffic);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("firewall"), index("netfw"))};
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("firewall"))};
parser {p_add_context_splunk(key("checkpoint_splunk_firewall")); };
} elif {
filter(f_checkpoint_splunk_Web);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("web"), index("netproxy"))};
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("web"))};
parser {p_add_context_splunk(key("checkpoint_splunk_web")); };
} elif {
filter(f_checkpoint_splunk_NetworkSessions);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("sessions"), index("netops"))};
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("sessions"))};
parser {p_add_context_splunk(key("checkpoint_splunk_sessions")); };
} elif {
filter(f_checkpoint_splunk_IDS_Malware);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("ids_malware"), index("netids"))};
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("ids_malware"))};
parser {p_add_context_splunk(key("checkpoint_splunk_ids")); };
} elif {
filter(f_checkpoint_splunk_IDS);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("ids"), index("netids"))};
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("ids"))};
parser {p_add_context_splunk(key("checkpoint_splunk_ids")); };
} elif {
filter(f_checkpoint_splunk_email);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("email"), index("email"))};
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("email"))};
parser {p_add_context_splunk(key("checkpoint_splunk_email")); };
} elif {
filter(f_checkpoint_splunk_DLP);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("firewall"), index("netdlp"))};
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("firewall"))};
parser {p_add_context_splunk(key("checkpoint_splunk_dlp")); };
} elif {
filter(f_checkpoint_splunk_syslog);
Expand All @@ -130,7 +130,7 @@ log {
set("${PROGRAM}", value(".PROGRAM"));
subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM"));
};
rewrite { r_set_splunk_dest_default(sourcetype("nix:syslog"), index("netops"), source("program:${.PROGRAM}")) };
rewrite { r_set_splunk_dest_default(sourcetype("nix:syslog"), source("program:${.PROGRAM}")) };
parser { p_add_context_splunk(key("checkpoint_os")); };

};
Expand Down Expand Up @@ -163,7 +163,7 @@ log {
set("${PROGRAM}", value(".PROGRAM"));
subst('^\/(?:[^\/]+\/)+', "" , value(".PROGRAM"));
};
rewrite { r_set_splunk_dest_default(sourcetype("nix:syslog"), index("netops"), source("program:${.PROGRAM}")) };
rewrite { r_set_splunk_dest_default(sourcetype("nix:syslog"), source("program:${.PROGRAM}")) };
parser { p_add_context_splunk(key("checkpoint_os")); };

parser (compliance_meta_by_source);
Expand Down
2 changes: 1 addition & 1 deletion package/etc/conf.d/log_paths/lp-cisco_acs.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ log {
parser(acs_event_time);
rewrite {
set("cisco_acs", value("fields.sc4s_vendor_product"));
r_set_splunk_dest_default(sourcetype("cisco:acs"), index("netauth"))
r_set_splunk_dest_default(sourcetype("cisco:acs"))
};

parser {p_add_context_splunk(key("cisco_acs")); };
Expand Down
4 changes: 2 additions & 2 deletions package/etc/conf.d/log_paths/lp-cisco_apic.conf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,14 @@ log {
};
rewrite {
set("cisco_APIC_acl", value("fields.sc4s_vendor_product"));
r_set_splunk_dest_default(sourcetype("cisco:apic:acl"), index("netfw"), template("t_hdr_msg"))
r_set_splunk_dest_default(sourcetype("cisco:apic:acl"), template("t_hdr_msg"))
};
parser { p_add_context_splunk(key("cisco_apic_acl")); };

} elif {
rewrite {
set("cisco_APIC_events", value("fields.sc4s_vendor_product"));
r_set_splunk_dest_default(sourcetype("cisco:apic:events"), index("netops"), template("t_hdr_msg"))
r_set_splunk_dest_default(sourcetype("cisco:apic:events"), template("t_hdr_msg"))
};
parser { p_add_context_splunk(key("cisco_apic_events")); };
};
Expand Down
Loading

0 comments on commit 365b583

Please sign in to comment.