I love Splunk. I run it at work and I run it at home. For the most part, Splunk just works. This post will serve as a warning for others looking to deploy Splunk professionally.
I was initially running it at home prior to running it at work. Offering 500MB / day for home users is awesome. Between free Splunk, free Nessus, Bro, and DD-WRT, I feel like I’ve got a pretty nice setup at home. This blog post will continue to be updated as my Splunk instances are updated.
Issues / Observations
- The Splunkbase Apps are mostly garbage. I use the Technology-Addons a bunch, but most of the dashboards provide me little value. I was warned that I’ll likely be building all my own Dashboards and Views, but I didn’t believe anyone. I should have listened.
- If you don’t already have logs going to a syslog server or some sort of log aggregation system, you are doomed. No one seems to be able to estimate their log volume, myself included.
- Some systems have really terrible logs. Cisco’s logs are all over the place. Each module in the ASA has a wildly different logging format. You’d think Cisco would try to implement uniform logging across their platform. Microsoft DNS logs are another example.
- If you are integrating multiple instances of a system, but each owned by someone else, expect pain. I’ve got some admins that prefer an agent while others prefer WMI. I’ve got some Linux admins that would prefer to FTP me logs while others like syslog. I’ve got some admins that prefer log format A while others prefer format B.