diff --git a/README.md b/README.md index 02b59dd..e0d3259 100644 --- a/README.md +++ b/README.md @@ -41,8 +41,6 @@ index = main # License - - Configuration and documentation licensed subject to [CC0](LICENSE-CC0) Code and scripts licensed subject to [BSD-2-Clause](LICENSE-BSD2) \ No newline at end of file diff --git a/docs/FortiGate_event.png b/docs/FortiGate_event.png new file mode 100644 index 0000000..9a0113d Binary files /dev/null and b/docs/FortiGate_event.png differ diff --git a/docs/FortiGate_traffic.png b/docs/FortiGate_traffic.png new file mode 100644 index 0000000..5bf2971 Binary files /dev/null and b/docs/FortiGate_traffic.png differ diff --git a/docs/FortiGate_utm.png b/docs/FortiGate_utm.png new file mode 100644 index 0000000..dc36625 Binary files /dev/null and b/docs/FortiGate_utm.png differ diff --git a/docs/gettingstarted.md b/docs/gettingstarted.md new file mode 100644 index 0000000..2d7c971 --- /dev/null +++ b/docs/gettingstarted.md @@ -0,0 +1,58 @@ + +# Pre-req + +* Linux host with Docker 19.x or newer with Docker Swarm enabled + * [Getting Started](https://docs.docker.com/get-started/) +* A Splunk index for metrics typically "em_metrics" +* One or more Splunk indexes for events collected by SC4S +* Splunk HTTP event collector enabled with a token dedicated for SC4S + * [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) + * [Splunk Enterprise Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud) +* A network load balancer (NLB) configured for round robin. Note: Special consideration may be required when more advanced products are used. The optimal configuration of the load balancer will round robin each http POST request (not each connection) + +# Setup + +* Create a directory on the server for configuration +* Create a docker-compose.yml file based on the following template + +```yaml +version: "3" +services: + sc4s: + image: rfaircloth/scs:latest + hostname: sc4s + ports: + - "514" + - "601" + - "514/udp" + - "5514" + - "5514/udp" + stdin_open: true + tty: true + environment: + - SPLUNK_HEC_URL=https://foo:8088/services/collector/event + - SPLUNK_HEC_TOKEN= + - SPLUNK_CONNECT_METHOD=hec + - SPLUNK_DEFAULT_INDEX= + - SPLUNK_METRICS_INDEX=em_metrics + volumes: + - splunk_index.csv:/opt/syslog-ng/etc/context/splunk_index.csv +``` + +* Download the latest context.csv file to the current directory +```bash +wget https://raw.githubusercontent.com/splunk/splunk-connect-for-syslog/master/package/etc/context/splunk_index.csv +``` + +* Edit splunk_index.csv review the index configuration and revise as required for sourcertypes utilized in your environment. + +* Start SC4S + +```bash +docker stack deploy --compose-file docker-compose.yml sc4s +``` + + +## Scale out + +Additional hosts can be deployed for syslog collection from additional network zones and locations diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..19dfcad --- /dev/null +++ b/docs/index.md @@ -0,0 +1,24 @@ +# Welcome to Splunk Connect for Syslog + +Splunk Connect for Syslog is an open source packaged solution for +getting data into Splunk using syslog-ng Open Source Edition (Syslog-NG OSE) and the Splunk +HTTP event Collector. + +## Project Goals + +* Bring a tested configuration and build of syslog-ng OSE to the market that will function consistently regardless of the underlying host's linux distribution +* Provide a container with the tested configuration for Docker/K8s that can be more easily deployed than upstream packages directly on a customer OS +* Provide validated (testable and tested) implementations of filter and parse functions for common vendor products +* Reduce latency and improve scale by balancing event distribution across Splunk Indexers + + + +## License + +* Configuration and documentation licensed subject to [CC0](LICENSE-CC0) + +* Code and scripts licensed subject to [BSD-2-Clause](LICENSE-BSD2) + +* Third Party Red Hat Universal Base Image see [License](https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf) + +* Third Party Syslog-NG (OSE) [License](https://github.com/balabit/syslog-ng) diff --git a/docs/performance.md b/docs/performance.md new file mode 100644 index 0000000..3eba88d --- /dev/null +++ b/docs/performance.md @@ -0,0 +1,27 @@ +# Performance + +## Tested Configuration + +* SC4S instance requesting 8 cores and 4 GB of memory with K8S scheduler. +* 6 Splunk Indexers clustered in Single site +* 1 loggen test client using the following command +* AWS instance type c5n.4xlarge + +```bash +/opt/syslog-ng/bin/loggen -i --rate=1000 --interval=180 -P -F --sdata="[test name=\"stress17\"]" -s 800 --active-connections=10 sc4s 514 +``` + +## Result + +The single syslog-ng container in this test is able to provided effective balancing and routing of events equivalent 632 GB per day + +``` +average rate = 9717.58 msg/sec, count=1749420, time=180.026, (average) msg size=800, bandwidth=7591.86 kB/sec + +``` + + +## Limitations + +* Splunk Enterprise's implementation of the http event collection server will respond to the client with a status code 200 and fail to commit the events to disk during a rolling restart in our testing 20-30 events per indexer may be lost + diff --git a/docs/sources.md b/docs/sources.md new file mode 100644 index 0000000..3fd78da --- /dev/null +++ b/docs/sources.md @@ -0,0 +1,170 @@ +# Vendor - Cisco + +## Product - ASA (Pre Firepower) + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/1620/ | +| Product Manual | https://www.cisco.com/c/en/us/td/docs/security/asa/asa82/configuration/guide/config/monitor_syslog.html | + + +## Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| cisco:asa | None | +| cisco:pix | Not supported | +| cisco:fwsm | Not supported | + +## Filter type + +MSG Parse: This filter parses message content + +## Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index as required. +* Follow vendor configuration steps per Product Manual above ensure: + * Log Level is 6 "Informational" + * Protocol is TCP/IP + * permit-hostdown is on + * device-id is hostname and included + * timestamp is included + +## Verification + +Use the following search to validate events are present + +``` +index= sourcetype=cisco:asa +``` + +Verify timestamp, and host values match as expected + +# Vendor - Fortinet + +## Product - Fortigate + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2846/ | +| Product Manual | https://docs.fortinet.com/product/fortigate/6.2 | + + +## Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| fgt_log | The catch all sourcetype is not used | +| fgt_traffic | None | +| fgt_utm | None | +| fgt_event | None + +## Filter type + +MSG Parse: This filter parses message content + +## Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index as required. +* Refer to the admin manual for specific details of configuration to send Reliable syslog using RFC 3195 format, a typical logging configuration will include the following features. + +``` +config log memory filter + +set forward-traffic enable + +set local-traffic enable + +set sniffer-traffic disable + +set anomaly enable + +set voip disable + +set multicast-traffic enable + +set dns enable + +end + +config system global + +set cli-audit-log enable + +end + +config log setting + +set neighbor-event enable + +end + +``` + +## Verification + +An active firewall will generate frequent events, in addition fortigate has the ability to test logging functionality using a build in command + +``` +diag log test +``` + +Verify timestamp, and host values match as expected + +``` +index= (sourcetype=fgt_log OR sourcetype=fgt_traffic OR sourcetype=fgt_utm) +``` + +###UTM Message type + +![FortiGate UTM message](FortiGate_utm.png) + +### Traffic Message Type + +![FortiGate Traffic message](FortiGate_traffic.png) + +###Event Message Type +![FortiGate Event message](FortiGate_event.png) + +Verify timestamp, and host values match as expected + +# Vendor - PaloAlto + +## Product - NGFW + +| Ref | Link | +|----------------|---------------------------------------------------------------------------------------------------------| +| Splunk Add-on | https://splunkbase.splunk.com/app/2757/ | +| Product Manual | https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/configure-syslog-monitoring.html | + + +## Sourcetypes + +| sourcetype | notes | +|----------------|---------------------------------------------------------------------------------------------------------| +| pan:log | None | +| pan:traffic | None | +| pan:threat | None | + +## Filter type + +MSG Parse: This filter parses message content + +## Setup and Configuration + +* Install the Splunk Add-on on the search head(s) for the user communities interested in this data source. If SC4S is exclusively used the addon is not required on the indexer. +* Review and update the splunk_index.csv file and set the index as required. +* Refer to the admin manual for specific details of configuration + * Select TCP or SSL transport option + * Select IETF Format + * Ensure the format of the event is not customized + +## Verification + +An active firewall will generate frequent events use the following search to validate events are present per source device + +``` +index= sourcetype=pan:*| stats count by host +``` \ No newline at end of file diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..ff2258a --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,9 @@ +#Troubleshooting + +## Syslog-ng Metrics + +## Syslog-NG Events + +## Container Events + +# Monitoring diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..38dddfb --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,8 @@ +site_name: Splunk Connect for Syslog +theme: readthedocs +nav: + - Home: 'index.md' + - Performance: 'performance.md' + - Getting Started: 'gettingstarted.md' + - Sources: 'sources.md' + - Troubleshooting: 'troubleshooting.md' \ No newline at end of file diff --git a/package/etc/conf.d/destinations/d_destinations.conf b/package/etc/conf.d/destinations/d_destinations.conf index 3f08461..5c59aee 100644 --- a/package/etc/conf.d/destinations/d_destinations.conf +++ b/package/etc/conf.d/destinations/d_destinations.conf @@ -17,15 +17,16 @@ destination d_hec { method("POST") log-fifo-size(`splunk-log-fifo-size`) workers(`SYSLOGNG_HEC_WORKERS`) - batch-lines(500) - batch-bytes(512Kb) + batch-lines(1000) + batch-bytes(2048Kb) batch-timeout(3) timeout(15) user_agent("syslog-ng User Agent") user("syslog-ng") + headers("Connection: close") password("`SPLUNK_HEC_TOKEN`") persist-name("splunk") - disk-buffer(mem-buf-length(500) + disk-buffer(mem-buf-length(15000) disk-buf-size(20000) reliable(no) dir("/opt/syslog-ng/var/data/disk-buffer/"))