Skip to content

Commit

Permalink
Merge to master (#627)
Browse files Browse the repository at this point in the history
* [filterchange] Cisco Nexus (#594)

Adjust airOS and Nexsus to address event with no HOST but with TZ indicator

* [doc] Palo filter does not support UDP due to limitations of message size (#591)

* [filteradd] Carbon Black Protection CEF format (#590)

* [fix] noise in SC4S logs from goss (#595)

* Troubleshooting/problem >solution (#593) (#596)

* Troubleshooting/problem >solution (#593)

* Troubleshooting guide format problem solution

* Minor chnages

* minor changes

* Revert "Troubleshooting/problem >solution (#593)" (#597)

This reverts commit 20a1116.

Co-authored-by: nandinivij <61885842+nandinivij@users.noreply.github.com>

* [filter] improve f5 filter logic (#601)

* [filter] Additional severity levels for f5

* [filter] improve f5 matching

* Fix indents

* Fix indents

Co-authored-by: mbonsack <mbonsack@splunk.com>

* [doc] clarify SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE (#602)

* [doc] clarify SC4S_DEST_SPLUNK_HEC_TLS_CA_FILE

* Clarification edit

* Clarification edit

* Clarify edit 2

* Clarify edit 2

* Clarify edit 3

* Clarify edit 3

* Clarify edit 4

* Clarify edit 4

Co-authored-by: mbonsack <mbonsack@splunk.com>

* [doc] Update troubleshooting guide (#600)

* Update troubleshooting guide
Co-authored-by: mbonsack <mbonsack@splunk.com>

* Unit file updates/home page update (#605)

* Update unit files for proper restart behavior
* Update pathnames for `sleep` and `conntrack`
* Update SC4S docs home page to indicate fully supported status

* Update `entrypoint.sh` HEC check (#607)

* Update `entrypoint.sh` to honor value of `SC4S_DEST_SPLUNK_HEC_TLS_VERIFY` during HEC connectivity check

* [filtermod] Update Citrix to handle malformed AAA (#609)

* [filtermod] Checkpoint doesn't use the correct whitespace (#608)

* [filtermod] CEF time stamp (#612)

False error caused when itterating through multiple time stamps
add support for fractional seconds

* [fix] Incorrect host resolution (#610)

* [fix] Incorrect host resolution

When the log source includes an IP as host resolve using connection IP rather than field IP. When the host resolves to a single name rather than FQDN do not set the host value as this can't be trusted

* Update test_common.py

* Update fix_dns.conf

* [filtermod] Resolve time issue in acs (#613)

* [filtermod] Resolve time issue in acs

ACS more often than not does not send a better time than BSD time field so do not attempt to use it

* Update lp-cisco_acs.conf.tmpl

* [filtermod] cisco date parser issues (#611)

* [filtermod] cisco date parser issues

False error reported when cisco device sends uptime
device reported time errors will now use the indexed field cisco_time_error
Add micro seconds format without year

* Update cisco_syslog.conf

Use cisco time even when wrong

* [filtermod] Fix issue with cp nested syslog (#614)

* [fix] lost brace in merge (#619)

* [filtermod] cisco date parser issues

False error reported when cisco device sends uptime
device reported time errors will now use the indexed field cisco_time_error
Add micro seconds format without year

* Update cisco_syslog.conf

Use cisco time even when wrong

* Update cisco_syslog.conf

* RSA SecurID timestamps (#616)

* Fix time/date parsing in RSA SecurID
* Nit: Should fix all references to this source as `SecurID` (with no "e")

* [fix] revert acs changes (#620)

Revert

* [fix] lookup host by sourceip (#621)

* [fix] lookup host by sourceip

* Update test_common.py

* Update test_cisco_acs.py (#622)

Improve test to avoid flaky ness

* [fix] Fixes #604 (#615)

Do not check client cert in TLS connections as we do not support a reasonable means of authorization checking

* [filtermod] Add program 'iControlPortal.cgi' to f5 list Fixes #568 (#617)

* [doc] prevent conntrack from halting start (#618)

* CEF:  Imperva WAF timestamp parsing (#624)

* CEF:  Imperva WAF timestamp parsing fix

* Fixed Cisco WSA python test (#623)

* Fixed Cisco WSA python test

* Update local container guidance (#625)

* Add item to remove pull from unit file in local container guidance
* Clarify use of internal (syslog-ng based) load balancing vs. external LB

* [feature] Alpha Documentation and changes to support microk8s

This is alpha level support and rough docs for k8s as a runtime

* [filtermod] Resolve f5 UTF issue with glob (#626)

Co-authored-by: nandinivij <61885842+nandinivij@users.noreply.github.com>
Co-authored-by: mbonsack <mbonsack@splunk.com>
  • Loading branch information
3 people authored and GitHub committed Aug 10, 2020
1 parent 0d6fea3 commit 3fa55d0
Show file tree
Hide file tree
Showing 27 changed files with 865 additions and 292 deletions.
68 changes: 68 additions & 0 deletions deploy/k8s-microk8s/sc4s-ds.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: splunk-sc4s
labels:
app: sc4s
spec:
selector:
matchLabels:
name: splunk-sc4s
template:
metadata:
labels:
name: splunk-sc4s
spec:
tolerations:
# this toleration is to have the daemonset runnable on master nodes
# remove it if your masters can't run pods
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: sc4s
image: localhost:32000/scs:latest
ports:
- containerPort: 514
envFrom:
- configMapRef:
name: sc4s-env-file
env:
- name: SPLUNK_HEC_TOKEN
valueFrom:
secretKeyRef:
name: splunk-s1-standalone-secrets
key: hec_token
- name: SC4S_SNMP_TRAP_COLLECT
value: "no"
- name: SC4S_CONTAINER_HOST
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: SC4S_RUNTIME_ENV
value: "k8s"
livenessProbe:
httpGet:
path: /healthz
port: 8080
# initialDelaySeconds: 15
periodSeconds: 3
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 30
periodSeconds: 10
volumeMounts:
- name: syslog-var
mountPath: "/opt/syslog-ng/var"
- name: sc4s-context
mountPath: /opt/syslog-ng/etc/conf.d/configmap/context
terminationGracePeriodSeconds: 600
volumes:
- name: syslog-var
persistentVolumeClaim:
claimName: splunk-sc4s-pvc
- name: sc4s-context
configMap:
name: sc4s-context-config
267 changes: 267 additions & 0 deletions deploy/k8s-microk8s/sc4s-infra.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,267 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: sc4s-env-file
data:
SPLUNK_HEC_URL: https://splunk-s1-standalone-headless:8088
SC4S_DEST_SPLUNK_HEC_TLS_VERIFY: "yes"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: sc4s-context-config
data:
# example of a simple property defined using --from-literal
compliance_meta_by_source.conf: |-
filter f_test_test {
# host("something-*" type(glob)) or
# netmask(169.254.100.0/24)
host("cannot_ever_happen")
};
compliance_meta_by_source.csv: |-
f_test_test,.splunk.index,"will_never_happen_index"
f_test_test,fields.compliance,"pci"
host.csv: |-
169.254.0.2,HOST,foo.example
splunk_metadata.csv: |-
bluecoat_proxy,index,netproxy
brocade_syslog,index,netops
ArcSight_ArcSight,index,main
Cyber-Ark_Vault,index,netauth
CyberArk_PTA,index,main
Incapsula_SIEMintegration,index,netwaf
Microsoft_Microsoft Windows,index,oswinsec
Microsoft_System or Application Event,index,oswin
checkpoint_splunk,index,netops
checkpoint_splunk_dlp,index,netdlp
checkpoint_splunk_email,index,email
checkpoint_splunk_firewall,index,netfw
checkpoint_splunk_ids,index,netids
checkpoint_splunk_os,index,netops
checkpoint_splunk_sessions,index,netops
checkpoint_splunk_web,index,netproxy
checkpoint_splunk,index,netops
checkpoint_splunk,index,netops
cisco_apic_acl,index,netfw
cisco_apic_events,index,netops
cisco_acs,index,netauth
cisco_asa,index,netfw
cisco_ftd,index,netfw
cisco_ios,index,netops
cisco_ise,index,netauth
cisco_meraki,index,netfw
cisco_nx_os,index,netops
cisco_ucm,index,main
cisco_wsa,index,netproxy
dell_rsa_secureid,index,netauth
citrix_netscaler,index,netfw
local_example,index,main
forcepoint_webprotect,index,netproxy
f5_bigip,index,netops
f5_bigip_access_json,index,netops
f5_bigip_irule,index,netops
f5_bigip_asm,index,netwaf
f5_bigip_nix,index,netops
fortinet_fortios_event,index,netops
fortinet_fortios_log,index,netops
fortinet_fortios_traffic,index,netfw
fortinet_fortios_utm,index,netids
fortinet_fortiweb_attack,index,netids
fortinet_fortiweb_event,index,netops
fortinet_fortiweb_log,index,netops
fortinet_fortiweb_traffic,index,netfw
infoblox_dns,index,netdns
infoblox_dhcp,index,netipam
infoblox_threat,index,netids
juniper_idp,index,netids
juniper_structured,index,netops
juniper_idp_structured,index,netids
juniper_junos_fw_structured,index,netfw
juniper_junos_ids_structured,index,netids
juniper_junos_utm_structured,index,netfw
juniper_junos_aamw_structured,index,netfw
juniper_junos_secintel_structured,index,netfw
juniper_junos_fw,index,netfw
juniper_junos_ids,index,netids
juniper_junos_utm,index,netfw
juniper_netscreen,index,netfw
juniper_legacy,index,netops
mcafee_epo,index,epav
nix_syslog,index,osnix
pan_traffic,index,netfw
pan_threat,index,netproxy
pan_system,index,netops
pan_config,index,netops
pan_hipmatch,index,main
pan_correlation,index,main
pan_userid,index,netauth
pan_unknown,index,netops
pfsense,index,netops
pfsense_filterlog,index,netfw
proofpoint_pps_filter,index,email
proofpoint_pps_sendmail,index,email
sc4s_events,index,main
sc4s_fallback,index,main
sc4s_metrics,index,em_metrics
symantec_ep,index,epav
symantec_brightmail,index,email
ubiquiti_unifi,index,netops
ubiquiti_unifi_fw,index,netfw
ubiquiti_unifi_link,index,netops
ubiquiti_unifi_sudo,index,netops
ubiquiti_unifi_switch,index,netops
ubiquiti_unifi_threat,index,netids
ubiquiti_unifi_wireless,index,netops
vmware_esx,index,main
vmware_horizon,index,main
vmware_nsx,index,main
vmware_vcenter,index,main
zscaler_alerts,index,netops
zscaler_dns,index,netdns
zscaler_fw,index,netfw
zscaler_web,index,netproxy
zscaler_zia_audit,index,netops
zscaler_zia_sandbox,index,main
zscaler_lss,index,netproxy
vendor_product_by_source.conf: |-
filter f_test_test {
host("testvp-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_null_queue {
netmask(169.254.100.0/24)
};
filter f_brocade_syslog {
host("test_brocade-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_citrix_netscaler {
host("test_ctitrixns-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_dell_rsa_secureid {
host("test_rsasecureid*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_juniper_netscreen {
host("jnpns-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_cisco_meraki {
host("testcm-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_cisco_wsa{
host("cisco_wsa" type(glob))
};
filter f_cisco_wsa11_7{
host("cisco_wsa11_7" type(glob))
};
filter f_cisco_nx_os {
host("csconx-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_f5_bigip {
host("test_f5-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_infoblox {
host("vib-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_pfsense {
host("pfsense-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_proofpoint_pps_filter {
host("pps-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_proofpoint_pps_sendmail {
host("pps-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_schneider_apc {
host("test_apc-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_ubiquiti_unifi_fw {
host("usg-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_tzfixhst {
host("tzfhst-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_tzfixny {
host("tzfny-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
vendor_product_by_source.csv: |-
f_test_test,sc4s_vendor_product,"test_test"
f_brocade_syslog,sc4s_vendor_product,"brocade_syslog"
f_null_queue,sc4s_vendor_product,"null_queue"
f_cisco_meraki,sc4s_vendor_product,"cisco_meraki"
f_cisco_wsa,sc4s_vendor_product,"cisco_wsa"
f_cisco_wsa11_7,sc4s_vendor_product,"cisco_wsa11_7"
f_citrix_netscaler,sc4s_vendor_product,"citrix_netscaler"
f_dell_rsa_secureid,sc4s_vendor_product,"dell_rsa_secureid"
f_f5_bigip,sc4s_vendor_product,"f5_bigip"
f_infoblox,sc4s_vendor_product,"infoblox"
f_juniper_netscreen,sc4s_vendor_product,"juniper_netscreen"
f_cisco_nx_os,sc4s_vendor_product,"cisco_nx_os"
f_pfsense,sc4s_vendor_product,"pfsense"
f_proofpoint_pps_sendmail,sc4s_vendor_product,"proofpoint_pps_sendmail"
f_proofpoint_pps_filter,sc4s_vendor_product,"proofpoint_pps_filter"
f_schneider_apc,sc4s_vendor_product,"schneider_apc"
f_ubiquiti_unifi_fw,sc4s_vendor_product,"ubiquiti_unifi_fw"
f_tzfixhst,sc4s_time_zone,"Pacific/Honolulu"
f_tzfixny,sc4s_time_zone,"America/New_York"
---

---
apiVersion: v1
kind: Service
metadata:
name: sc4s-ext-tcp
annotations:
metallb.universe.tf/allow-shared-ip: sc4s
spec:
ports:
- port: 514
targetPort: 514
protocol: TCP
selector:
app: sc4s
type: LoadBalancer
externalTrafficPolicy: Local
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: splunk-sc4s-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500M
---
apiVersion: v1
kind: Service
metadata:
name: sc4s-ext-udp
annotations:
metallb.universe.tf/allow-shared-ip: sc4s
spec:
ports:
- port: 514
targetPort: 514
protocol: UDP
selector:
app: sc4s
type: LoadBalancer
externalTrafficPolicy: Local
---

1 change: 0 additions & 1 deletion docs/gettingstarted/docker-systemd-general.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ Environment="SC4S_LOCAL_CONFIG_MOUNT=-v /opt/sc4s/local:/opt/syslog-ng/etc/conf.
# Environment="SC4S_TLS_DIR=-v /opt/sc4s/tls:/opt/syslog-ng/tls:z"

TimeoutStartSec=0
Restart=always

ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE
ExecStartPre=/usr/bin/bash -c "/usr/bin/systemctl set-environment SC4SHOST=$(hostname -s)"
Expand Down
21 changes: 16 additions & 5 deletions docs/gettingstarted/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,13 @@ Install the following:
#### Configure the Splunk HTTP Event Collector

- Set up the Splunk HTTP Event Collector with the HEC endpoints behind a load balancer (VIP) configured for https round robin *WITHOUT* sticky
session. Alternatively, a list of HEC endpoint URLs can be configured in SC4S (native Syslog-ng load balancing) if no load balancer is in place. In either case, it is
recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of HWFs. Deployments with 10 or fewer Indexers and where HEC is used exclusively for syslog, the recommendation is to use the native load balancing. In all other scenarios the recommendation is to use an external load balacer. If utilizing the native load balancing, be sure to update the configuration when the number and/or names of the indexers change.
session. Alternatively, a list of HEC endpoint URLs can be configured in SC4S (native syslog-ng load balancing) if no load balancer is in
place. In most scenarios the recommendation is to use an external load balancer, as that makes longer term
maintenance simpler by eliminating the need to manually keep the list of HEC URLs specified in sc4s current. However, if a LB is not
available, native load balancing can be used with 10 or fewer Indexers where HEC is used exclusively for syslog.

In either case, it is _strongly_ recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than
an intermediate tier of HWFs.
- Create a HEC token that will be used by SC4S and ensure the token has access to place events in main, em_metrics, and all indexes used as
event destinations.

Expand All @@ -83,7 +88,8 @@ Splunk type.
#### Prerequisites

* Linux host with Docker (CE 19.x or greater with Docker Swarm) or Podman enabled, depending on runtime choice (below).
* A network load balancer (NLB) configured for round robin. Note: Special consideration may be required when more advanced products are used. The optimal configuration of the load balancer will round robin each http POST request (not each connection).
* A network load balancer (NLB) configured for round robin. Note: Special consideration may be required when more advanced products are used.
The optimal configuration of the load balancer will round robin each http POST request (not each connection).
* The host linux OS receive buffer size should be tuned to match the sc4s default to avoid dropping events (packets) at the network level.
The default receive buffer for sc4s is set to 16 MB for UDP traffic, which should be OK for most environments. To set the host OS kernel to
match this, edit `/etc/sysctl.conf` using the following whole-byte values corresponding to 16 MB:
Expand Down Expand Up @@ -135,7 +141,8 @@ net.ipv4.ip_forward=1
Follow these instructions to "stage" SC4S by downloading the container so that it can be loaded "out of band" on a
host machine, such as an airgapped system, without internet connectivity.

* Download container image "oci_container.tgz" from our [Github Page](https://github.com/splunk/splunk-connect-for-syslog/releases). The following example downloads v1.12; replace the URL with the latest release or pre-release version as desired.
* Download container image "oci_container.tgz" from our [Github Page](https://github.com/splunk/splunk-connect-for-syslog/releases).
The following example downloads v1.12; replace the URL with the latest release or pre-release version as desired.

```
sudo wget https://github.com/splunk/splunk-connect-for-syslog/releases/download/v1.12.0/oci_container.tar.gz
Expand Down Expand Up @@ -167,4 +174,8 @@ attempt to obtain the container image via the internet.
```
Environment="SC4S_IMAGE=sc4slocal:latest"
```

* Remove the entry
```
ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE
```
from the relevant unit file when using systemd, as an external connection to pull the container is no longer needed (or available).
Loading

0 comments on commit 3fa55d0

Please sign in to comment.