Skip to content

Commit

Permalink
[filter] Improve CEF parser configuration and simplify future enhance…
Browse files Browse the repository at this point in the history 
…ments

* Move CEF context files to local tree
* Remove vendor-specfic template logic from CEF log path and use `splunk_metadata.csv` to override
  • Loading branch information
mbonsack authored and GitHub committed Jul 27, 2020
1 parent 93c6843 commit 9b32101
Show file tree
Hide file tree
Showing 30 changed files with 146 additions and 169 deletions.
4 changes: 2 additions & 2 deletions docs/sources/Brocade/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ Device setup unknown

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_BROCADE_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_BROCADE_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_BROCADE_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_BROCADE_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_BROCADE | no | Enable archive to disk for this specific source |
| SC4S_DEST_BROCADE_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down
38 changes: 19 additions & 19 deletions docs/sources/Cisco/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ PATTERN MATCH

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CISCO_ACS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_ACS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_ACS_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_CISCO_ACS_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_CISCO_ACS | no | Enable archive to disk for this specific source |
| SC4S_DEST_CISCO_ACS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down Expand Up @@ -82,8 +82,8 @@ PATTERN MATCH

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CISCO_APIC_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_APIC_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_APIC_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_CISCO_APIC_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_CISCO_APIC | no | Enable archive to disk for this specific source |
| SC4S_DEST_CISCO_APIC_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down Expand Up @@ -140,12 +140,12 @@ MSG Parse: This filter parses message content

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CISCO_ASA_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_ASA_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_ASA_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_CISCO_ASA_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_CISCO_ASA | no | Enable archive to disk for this specific source |
| SC4S_DEST_CISCO_ASA_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |
| SC4S_LISTEN_CISCO_ASA_LEGACY_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC3164 format |
| SC4S_LISTEN_CISCO_ASA_LEGACY_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined expecting RFC3164 format |
| SC4S_LISTEN_CISCO_ASA_LEGACY_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers expecting RFC3164 format |
| SC4S_LISTEN_CISCO_ASA_LEGACY_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers expecting RFC3164 format |
| SC4S_ARCHIVE_CISCO_ASA_LEGACY | no | Enable archive to disk for this specific source |
| SC4S_DEST_CISCO_ASA_LEGACY_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down Expand Up @@ -227,8 +227,8 @@ Cisco Network Products of multiple types share common logging characteristics th

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CISCO_IOS_UDP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_IOS_TCP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_IOS_UDP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_CISCO_IOS_TCP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_CISCO_IOS | no | Enable archive to disk for this specific source |
| SC4S_DEST_CISCO_IOS_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down Expand Up @@ -273,8 +273,8 @@ PATTERN MATCH

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CISCO_ISE_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format |
| SC4S_LISTEN_CISCO_ISE_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined expecting RFC5424 format |
| SC4S_LISTEN_CISCO_ISE_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers expecting RFC5424 format |
| SC4S_LISTEN_CISCO_ISE_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers expecting RFC5424 format |
| SC4S_ARCHIVE_CISCO_ISE | no | Enable archive to disk for this specific source |
| SC4S_DEST_CISCO_ISE_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down Expand Up @@ -323,8 +323,8 @@ IP, Netmask, Host or Port

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CISCO_MERAKI_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined expecting RFC5424 format |
| SC4S_LISTEN_CISCO_MERAKI_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined expecting RFC5424 format |
| SC4S_LISTEN_CISCO_MERAKI_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers expecting RFC5424 format |
| SC4S_LISTEN_CISCO_MERAKI_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers expecting RFC5424 format |
| SC4S_ARCHIVE_CISCO_MERAKI | no | Enable archive to disk for this specific source |
| SC4S_DEST_CISCO_MERAKI_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down Expand Up @@ -371,8 +371,8 @@ PATTERN MATCH

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CISCO_UCM_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_UCM_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_UCM_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_CISCO_UCM_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_CISCO_UCM | no | Enable archive to disk for this specific source |
| SC4S_DEST_CISCO_UCM_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down Expand Up @@ -424,8 +424,8 @@ IP, Netmask or Host

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CISCO_WSA_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_WSA_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CISCO_WSA_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_CISCO_WSA_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_CISCO_WSA | no | Enable archive to disk for this specific source |
| SC4S_DEST_CISCO_WSA_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand All @@ -437,4 +437,4 @@ Use the following search to validate events are present
index=netops sourcetype=cisco:wsa:*
```

Verify timestamp, and host values match as expected
Verify timestamp, and host values match as expected
65 changes: 34 additions & 31 deletions docs/sources/CommonEventFormat/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,31 +10,8 @@ Imperva, and Cyberark. Therefore, the CEF environment variables for unique port
should be set only _once_.

If your deployment has multiple CEF devices that send to more than one port,
set the CEF unique port variable(s) to just one of the ports in use. Then, map the others with
container networking to the port chosen, similar to the way default ports are configured (see the
"Getting Started" runtime documents for more details).

Example: If you have three CEF devices,
sending on TCP ports 2000,2001, and 2002, set `SC4S_LISTEN_CEF_TCP_PORT=2000`. Then, change the
unit/compose files to route the three external ports to the single port 2000 on the container.
Here is the example for podman/systemd:

```
ExecStart=/usr/bin/podman -p 514:514 -p 514:514/udp -p 6514:6514 -p 2000-2002:2000 \
```

or this, for docker-compose/swarm installations:

```
# Comment the following line out if using docker-compose
mode: host
- target: 2000
published: 2000-2002
protocol: tcp
```

These changes will route all three ports to TCP port 2000 inside the container, and the single CEF log
path will properly process data from all three devices.
set the CEF unique port variable(s) as a comma-separated list. See [Unique Listening Ports](https://splunk-connect-for-syslog.readthedocs.io/en/develop/sources/#unique-listening-ports)
for details.

The source documentation included below is a reference baseline for any product that sends data
using the CEF log path.
Expand All @@ -46,19 +23,45 @@ using the CEF log path.
| Product Manual | https://docs.imperva.com/bundle/cloud-application-security/page/more/log-configuration.htm |


### Sourcetypes
### Splunk Metadata with CEF events

The keys (first column) in `splunk_metadata.csv` for CEF data sources have a slightly different meaning than those for non-CEF ones.
The typical `vendor_product` syntax is instead replaced by checks against specific columns of the CEF event -- namely the first,
second, and fourth columns following the leading `CEF:0` ("column 0"). These specific columns refer to the CEF `device_vendor`,
`device_product`, and `device_event_class`, respectively. The third column, `device_version`, is not used for metadata assignment.

SC4S sets metadata based on the first two columns, and (optionally) the fourth. While the key (first column) in the
`splunk_metadata` file for non-CEF sources uses a "vendor_product" syntax that is arbitrary, the syntax for this key for CEF
events is based on the actual contents of columns 1,2 and 4 from the CEF event, namely:

`device_vendor`\_`device_product`\_`device_class`

The final `device_class` portion is optional. Therefore, CEF entries in `splunk_metadata` can have a key representing the vendor and
product, and others representing a vendor and product coupled with one or more additional classes. This allows for more granular
metadata assignment (or overrides).

Here is a snippet of a sample Imperva CEF event that includes a CEF device class entry (which is "Firewall"):
```
Apr 19 10:29:53 3.3.3.3 CEF:0|Imperva Inc.|SecureSphere|12.0.0|Firewall|SSL Untraceable Connection|Medium|
```
and the corresponding match in `splunk_metadata.csv`:
```
Imperva Inc._SecureSphere_Firewall,sourcetype,imperva:waf:firewall:cef
```

### Default Sourcetype

| sourcetype | notes |
|----------------|---------------------------------------------------------------------------------------------------------|
| cef | Common sourcetype |

### Typical Source
### Default Source

| source | notes |
|----------------|---------------------------------------------------------------------------------------------------------|
| Varies | Varies |

### Typical Index Configuration
### Default Index Configuration

| key | source | index | notes |
|----------------|----------------|----------------|----------------|
Expand All @@ -72,9 +75,9 @@ MSG Parse: This filter parses message content

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CEF_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CEF_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CEF_TLS_PORT | empty string | Enable a TLS port for this specific vendor product using the number defined |
| SC4S_LISTEN_CEF_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_CEF_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_CEF_TLS_PORT | empty string | Enable a TLS port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_CEF | no | Enable archive to disk for this specific source |
| SC4S_DEST_CEF_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down
4 changes: 2 additions & 2 deletions docs/sources/CyberArk/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ MSG Parse: This filter parses message content

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CEF_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CEF_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |

* NOTE: Set only _one_ set of CEF variables for the entire SC4S deployment, regardless of how
many ports are in use by this CEF source (or any others). See the "Common Event Format" source
Expand Down Expand Up @@ -72,7 +72,7 @@ MSG Parse: This filter parses message content

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CEF_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_CEF_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |

* NOTE: Set only _one_ set of CEF variables for the entire SC4S deployment, regardless of how
many ports are in use by this CEF source (or any others). See the "Common Event Format" source
Expand Down
4 changes: 2 additions & 2 deletions docs/sources/Dell_RSA/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,8 @@ NOTE: Java trace and exception will default to sc4s:fallback if the host/ip filt

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_DELL_RSA_SECUREID_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_DELL_RSA_SECUREID_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_DELL_RSA_SECUREID_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_DELL_RSA_SECUREID_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_DELL_RSA_SECUREID | no | Enable archive to disk for this specific source |
| SC4S_DEST_DELL_RSA_SECUREID_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down
4 changes: 2 additions & 2 deletions docs/sources/F5/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ When F5 blades are identified as part of the host name the blade will be indicat

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_F5_BIGIP_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_F5_BIGIP_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_F5_BIGIP_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_F5_BIGIP_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_F5_BIGIP | no | Enable archive to disk for this specific source |
| SC4S_DEST_F5_BIGIP_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down
4 changes: 2 additions & 2 deletions docs/sources/Forcepoint/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ MSG Parse: This filter parses message content

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_FORCEPOINT_WEBPROTECT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the number defined |
| SC4S_LISTEN_FORCEPOINT_WEBPROTECT_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using the number defined |
| SC4S_LISTEN_FORCEPOINT_WEBPROTECT_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_LISTEN_FORCEPOINT_WEBPROTECT_UDP_PORT | empty string | Enable a UDP port for this specific vendor product using a comma-separated list of port numbers |
| SC4S_ARCHIVE_FORCEPOINT_WEBPROTECT | no | Enable archive to disk for this specific source |
| SC4S_DEST_FORCEPOINT_WEBPROTECT_HEC | no | When Splunk HEC is disabled globally set to yes to enable this specific source |

Expand Down
Binary file modified docs/sources/Fortinet/FortiGate_event.png
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/sources/Fortinet/FortiGate_traffic.png
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/sources/Fortinet/FortiGate_utm.png
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 9b32101

Please sign in to comment.