Skip to content

Commit

Permalink
Merge branch 'develop' into master
Browse files Browse the repository at this point in the history
  • Loading branch information
Ryan Faircloth authored and GitHub committed Oct 15, 2019
2 parents 39faf36 + 355180c commit a3ced18
Show file tree
Hide file tree
Showing 13 changed files with 391 additions and 41 deletions.
2 changes: 1 addition & 1 deletion .gitmodules
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[submodule "package/syslog-ng"]
path = package/syslog-ng
url = https://github.com/balabit/syslog-ng.git
branch = syslog-ng-3.23.1
branch = syslog-ng-3.24.1
#

73 changes: 37 additions & 36 deletions docs/gettingstarted/byoe-rhel7.md
Original file line number Diff line number Diff line change
@@ -1,55 +1,54 @@
#Warning
# SC4S "Bring Your Own Environment"

The "Bring Your Own Environment" instructions that follow allow administrators to utilize the SC4S syslog-ng
config files directly on the host OS running on a hardware server or virtual machine. Administrators must provide an
appropriate host OS as well as an up-to-date syslog-ng installation either built from source (not documented) or
appropriate host OS as well as an up-to-date syslog-ng installation either built from source (not documented here) or
installed from community-built RPMs. Modification of the base configuration will be required for most customer
environments due to enterprise infrastructure variations.

* NOTE: Installing or modifying system configurations can have unexpected consequences, and rudimentary linux system
administratrion and syslog-ng configuration experience is assumed.

* NOTE: Do _not_ depend on the distribution-supplied version of syslog-ng, as it will likely be far too old.
Read this [explanation](https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-ng-on-rhel-and-other-rpm-distributions)
on the reason syslog-ng builds are so dated in most RHEL/Debian distributions.
for the reason why syslog-ng builds are so dated in most RHEL/Debian distributions.

# BYOE Installation Instructions

* Install CentOS or RHEL 7.7
* Enable EPEL
* Centos 7

```bash
sudo yum install epel-release
```

* RHEL 7

```bash
cd /tmp
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install ./epel-release-latest-*.noarch.rpm -y
```

* Enable EPEL (Centos 7)

```bash
sudo yum install epel-release
```

* Enable the optional repo for RHEL 7 only
* Enable EPEL and optional repo (RHEL 7)

```bash
cd /tmp
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install ./epel-release-latest-*.noarch.rpm -y
sudo subscription-manager repos --enable rhel-7-server-optional-rpms
```

```bash
sudo subscription-manager repos --enable rhel-7-server-optional-rpms
```
* Enable the "stable" unoffical repo for syslog-ng
* Enable the "stable" unofficial repo for syslog-ng and install required packages

```bash
cd /etc/yum.repos.d/
sudo wget https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-stable/repo/epel-7/czanik-syslog-ng-stable-epel-7.repo
sudo yum install syslog-ng syslog-ng-http syslog-ng-python
```
```bash
cd /etc/yum.repos.d/
sudo wget https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-stable/repo/epel-7/czanik-syslog-ng-stable-epel-7.repo
sudo yum install syslog-ng syslog-ng-http syslog-ng-python
```

* Optional step: Disable the OOB syslog-ng unit file, as the syslog-ng process configured here will run as the `sc4s`
service. rsyslog will continue to be the system logger, and can be left enabled _only_ if it is configured to not
listen on the same ports as sc4s.
* Optional step: Disable the distro-supplied syslog-ng unit file, as the syslog-ng process configured here will run as the `sc4s`
service. rsyslog will continue to be the system logger, but should be left enabled _only_ if it is configured to not
listen on the same ports as sc4s. sc4s BYOE can be configured to provide local logging as well if desired.

```bash
systemctl stop syslog-ng
systemctl disable syslog-ng
sudo systemctl stop syslog-ng
sudo systemctl disable syslog-ng
```

* Download the latest bare_metal.tar from [releases](https://github.com/splunk/splunk-connect-for-syslog/releases) on github and untar the package

```bash
Expand All @@ -61,7 +60,7 @@ sudo mkdir -p /opt/syslog-ng/var
sudo cp -R etc/* /opt/syslog-ng/etc/
```

* Install and verify gomplate verify the output is 3.5.0 or newer
* Install gomplate and confirm that the version is 3.5.0 or newer

```bash
sudo curl -o /usr/local/bin/gomplate -sSL https://github.com/hairyhenderson/gomplate/releases/download/v3.5.0/gomplate_linux-amd64
Expand Down Expand Up @@ -118,9 +117,10 @@ cp --verbose -R -n /opt/syslog-ng/etc/local_config/* /opt/syslog-ng/etc/conf.d/l
mkdir -p /opt/syslog-ng/var/data/disk-buffer/
```

* set execute permissions on the file
```
sudo chmod 755 /opt/sc4s/bin/preconfig.sh
* Execute the preconfiguration file created above

```bash
sudo bash /opt/sc4s/bin/preconfig.sh
```

* Create the file ``/opt/sc4s/default/env_file`` and add the following environment variables:
Expand All @@ -141,5 +141,6 @@ SPLUNK_METRICS_INDEX=em_metrics

```bash
sudo systemctl daemon-reload
sudo systemctl enable sc4s
sudo systemctl start sc4s
```
65 changes: 65 additions & 0 deletions docs/sources.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,68 @@
# Vendor - Checkpoint

## Product - Log Exporter (Splunk)

| Ref | Link |
|----------------|---------------------------------------------------------------------------------------------------------|
| Splunk Add-on | https://splunkbase.splunk.com/app/4293/ |
| Product Manual | https://sc1.checkpoint.com/documents/App_for_Splunk/html_frameset.htm |


### Sourcetypes

| sourcetype | notes |
|----------------|---------------------------------------------------------------------------------------------------------|
| cp_log | None |

### Sourcetype and Index Configuration

| key | sourcetype | index | notes |
|----------------|----------------|----------------|----------------|
| checkpoint_splunk | cp_log | netfw | none |

### Source and Index Configuration

Checkpoint Software blades with CIM mapping have been sub-grouped into sources
to allow routing to appropriate indexes. All other source meta data is left at default

| key | source | index | notes |
|----------------|----------------|----------------|----------------|
| checkpoint_splunk_dlp | dlp | netdlp | none |
| checkpoint_splunk_email | email | email | none |
| checkpoint_splunk_firewall | firewall | netfw | none |
| checkpoint_splunk_sessions | sessions | netops | none |
| checkpoint_splunk_web | web | netproxy | none |

### Filter type

MSG Parse: This filter parses message content

### Setup and Configuration

* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer.
* Review and update the splunk_index.csv file and set the index and sourcetype as required for the data source.
* Follow vendor configuration steps per Product Manual above ensure:
* Log Level is 6 "Informational"
* Protocol is TCP/IP
* permit-hostdown is on
* device-id is hostname and included
* timestamp is included

### Options

| Variable | default | description |
|----------------|----------------|----------------|
| SC4S_LISTEN_CHECKPOINT_SPLUNK_TCP_PORT | empty string | Enable a TCP port for this specific vendor product using the port number defined |

### Verification

Use the following search to validate events are present

```
index=<asconfigured> sourcetype=cisco:asa
```

Verify timestamp, and host values match as expected
# Vendor - Cisco

## Product - ASA (Pre Firepower)
Expand Down
2 changes: 2 additions & 0 deletions package/etc/conf.d/conflib/_splunk/splunkfields.conf
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@ rewrite r_set_splunk_default {
#overridden by user defined values
block rewrite r_set_splunk_dest_default(
index()
source("${.splunk.source}")
sourcetype()
template(`splunk-template`)
) {
set("`index`", value(".splunk.index"));
set("`source`", value(".splunk.source"));
set("`sourcetype`", value(".splunk.sourcetype"));
set("`template`", value("fields.sc4s_template"));
};
Expand Down
57 changes: 57 additions & 0 deletions package/etc/conf.d/filters/checkpoint/splunk.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
filter f_checkpoint_splunk {
match('\|(?:origin_sic_name|originsicname)\=[cC][nN]|\|product\=SmartConsole\|' value("RAWMSG") type("pcre"));
};

filter f_checkpoint_splunk_alerts {
match('*IOS Profile*' value('.kv.product') type('glob'))
or match('*Device*' value('.kv.product') type('glob'))
};

filter f_checkpoint_splunk_Change {
match('*Application Control*' value('.kv.product') type('glob'))
};

filter f_checkpoint_splunk_DLP {
match('*DLP*' value('.kv.product') type('glob'))
};

filter f_checkpoint_splunk_email {
match('*MTA*' value('.kv.product') type('glob'))
or match('*Anti-Spam*' value('.kv.product') type('glob'))
or match('*Anti Spam*' value('.kv.product') type('glob'))
};

filter f_checkpoint_splunk_IDS {
match('*IPS*' value('.kv.product') type('glob'))
or match('*WIFI*' value('.kv.product') type('glob'))
or match('*Cellular*' value('.kv.product') type('glob'))
};

filter f_checkpoint_splunk_IDS_Malware {
match('*Threat Emulation*' value('.kv.product') type('glob'))
or match('*Anti-Virus*' value('.kv.product') type('glob'))
or match('*Anti-Bot*' value('.kv.product') type('glob'))
or match('*Threat Extraction*' value('.kv.product') type('glob'))
or match('*Anti-Ransomware*' value('.kv.product') type('glob'))
or match('*Anti-Exploit**' value('.kv.product') type('glob'))
or match('*Forensics*' value('.kv.product') type('glob'))
or match('*OS Exploit*' value('.kv.product') type('glob'))
or (match('*Application*' value('.kv.product') type('glob')) and not match('*Application Control*' value('.kv.product') type('glob')))
or match('*Text Message*' value('.kv.product') type('glob'))
or match('*Network Access*' value('.kv.product') type('glob'))
or match('*Zero Phishing*' value('.kv.product') type('glob'))
};

filter f_checkpoint_splunk_NetworkSessions {
match('*VPN*' value('.kv.product') type('glob'))
or match('*Mobile*' value('.kv.product') type('glob'))
or match('*VPN*' value('.kv.fw_subproduct') type('glob'))
};

filter f_checkpoint_splunk_NetworkTraffic {
match('*Firewall*' value('.kv.product') type('glob'))
and not match('*VPN*' value('.kv.fw_subproduct') type('glob'))
};
filter f_checkpoint_splunk_Web {
match('*Url Filtering*' value('.kv.product') type('glob'))
};
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@version: 3.23
@version: 3.24
filter f_test_test {
host("something-*" type(glob)) or
netmask(192.168.100.1/24)
Expand Down
72 changes: 72 additions & 0 deletions package/etc/conf.d/log_paths/p_rfc3164-checkpoint_splunk.conf.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Checkpoint Splunk format
{{- if (ne (getenv (print "SC4S_LISTEN_CHECKPOINT_SPLUNK_TCP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_CHECKPOINT_SPLUNK_UDP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_CHECKPOINT_SPLUNK_TLS_PORT") "no") "no") }}
{{ $context := dict "port_id" "CHECKPOINT_SPLUNK" "parser" "common" }}
{{ tmpl.Exec "t/source_network.t" $context }}
{{- end -}}
{{ define "log_path" }}
log {
{{- if eq (.) "yes"}}
source(s_default-ports);
filter(f_is_rfc3164);
filter(f_checkpoint_splunk);
{{- end}}
{{- if eq (.) "no"}}
source (s_dedicated_port_CHECKPOINT_SPLUNK);
{{- end}}

parser {
kv-parser(prefix(".kv.") pair-separator("|") template("${MSGHDR} ${MSG}"));
date-parser(format("%s") template("${.kv.time}") time-zone({{- getenv "SC4S_DEFAULT_TIMEZONE" "GMT"}}));

};

rewrite { set("${.kv.hostname}", value("HOST")); };

rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), index("netops"), template("t_hdr_msg"))};
parser {p_add_context_splunk(key("checkpoint_splunk")); };

if {
filter(f_checkpoint_splunk_NetworkTraffic);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("firewall"), index("netfw"), template("t_standard"))};
parser {p_add_context_splunk(key("checkpoint_splunk_firewall")); };
} elif {
filter(f_checkpoint_splunk_Web);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("web"), index("netproxy"), template("t_standard"))};
parser {p_add_context_splunk(key("checkpoint_splunk_web")); };
} elif {
filter(f_checkpoint_splunk_NetworkSessions);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("sessions"), index("netops"), template("t_standard"))};
parser {p_add_context_splunk(key("checkpoint_splunk_sessions")); };
} elif {
filter(f_checkpoint_splunk_IDS_Malware);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("ids_malware"), index("netids"), template("t_standard"))};
parser {p_add_context_splunk(key("checkpoint_splunk_ids")); };
} elif {
filter(f_checkpoint_splunk_IDS);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("ids"), index("netids"), template("t_standard"))};
parser {p_add_context_splunk(key("checkpoint_splunk_ids")); };
} elif {
filter(f_checkpoint_splunk_email);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("email"), index("email"), template("t_standard"))};
parser {p_add_context_splunk(key("checkpoint_splunk_email")); };
} elif {
filter(f_checkpoint_splunk_DLP);
rewrite { r_set_splunk_dest_default(sourcetype("cp_log"), source("firewall"), index("netdlp"), template("t_standard"))};
parser {p_add_context_splunk(key("checkpoint_splunk_dlp")); };
};


parser (compliance_meta_by_source);

destination(d_hec); #--HEC--

flags(flow-control);
};
{{- end}}
{{- if (ne (getenv (print "SC4S_LISTEN_CHECKPOINT_SPLUNK_TCP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_CHECKPOINT_SPLUNK_UDP_PORT") "no") "no") or (ne (getenv (print "SC4S_LISTEN_CHECKPOINT_SPLUNK_TLS_PORT") "no") "no") }}
# Listen on the specified dedicated port(s) for CHECKPOINT_SPLUNK traffic
{{ tmpl.Exec "log_path" "no" }}
{{- end}}

# Listen on the default port (typically 514) for CHECKPOINT_SPLUNK traffic
{{ tmpl.Exec "log_path" "yes" }}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@version: 3.23
@version: 3.24
filter f_test_test {
host("something-*" type(glob)) or
netmask(192.168.100.1/24)
Expand Down
8 changes: 8 additions & 0 deletions package/etc/context_templates/splunk_index.csv
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
#cef_Incapsula_SIEMintegration,index,netwaf
#cef_Microsoft_Microsoft Windows,index,oswinsec
#cef_Microsoft_System or Application Event,index,oswin
#checkpoint_splunk,index,netops
#checkpoint_splunk_dlp,index,netdlp
#checkpoint_splunk_email,index,email
#checkpoint_splunk_firewall,index,netfw
#checkpoint_splunk_sessions,index,netops
#checkpoint_splunk_web,index,netproxy
#checkpoint_splunk,index,netops
#checkpoint_splunk,index,netops
#cisco_asa,index,netfw
#cisco_ios,index,netops
#cisco_nx_os,index,netops
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@version: 3.23
@version: 3.24

filter f_test_test {
host("testvp-*" type(glob)) or
Expand Down
2 changes: 1 addition & 1 deletion package/etc/syslog-ng.conf
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@version:3.23
@version:3.24

# syslog-ng configuration file.
#
Expand Down
5 changes: 5 additions & 0 deletions splunk/etc/apps/SA-syslog-ng/default/indexes.conf
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ homePath = $SPLUNK_DB/oswinsec/db
coldPath = $SPLUNK_DB/oswinsec/colddb
thawedPath = $SPLUNK_DB/oswinsec/thaweddb

[netdlp]
homePath = $SPLUNK_DB/netdlp/db
coldPath = $SPLUNK_DB/netdlp/colddb
thawedPath = $SPLUNK_DB/netdlp/thaweddb

[netfw]
homePath = $SPLUNK_DB/netfw/db
coldPath = $SPLUNK_DB/netfw/colddb
Expand Down
Loading

0 comments on commit a3ced18

Please sign in to comment.