How to onboard log file with two different line-breakers


Recently, a client asked for help onboarding this Mulesoft/Sitecore log file onto Splunk. This application log file appears quite complex at first glance.

Sample data

ManagedPoolThread #0 05:27:44 WARN  Failed to create counter 'Sitecore.System\Events | Events Raised / sec'. Sitecore has no necessary permissions for reading/creating counters.

Splunk docker container with Trunk, built with Packer and Terraform


I don't work at Splunk anymore, but I still do some freelance consulting, and so I thought I would share one of my old side projects, Trunk.

I am providing a couple of Github repos so that you can create and run your own Splunk instance with Trunk preconfigured to index Trello webhooks in your own AWS account. When the container starts, it subscribes to open Trello boards that are accessible with the Trello credential you provide.

The Trunk app, which provides status report dashboards.

Find size of lookup files in Splunk Web


I needed to list the lookup tables and their sizes using Splunk Web. Here's a query to do just that!

| rest splunk_server=local /services/data/lookup-table-files/
| rename eai:acl.app as app 
| table app title 
| search NOT title IN (*.kmz) 
| map maxsearches=990 search="| inputlookup $title$ 
| eval size=0
| foreach * [ eval size=size+coalesce(len('<<FIELD>>'),0), app=\"$app$\", title=$title$ | fields app title size]" 
| stats sum(size) by app title
| sort - sum(size)

Splunk foreach command examples


The Splunk foreach SPL command is pretty useful for building powerful queries. Here are some examples that I've created as a reference for how to use this powerful command.

Splunk macro to remove identical fields



Suppose that you have a Splunk query that returns a result set with some duplicate fields. Would you like to remove the duplicate fields so that you can quickly identify the differenes between each result? Here's a macro to do it!

Splunk macro to remove empty fields


If you have a Splunk query that returns empty fields, you can use this query to programatically remove these blanks.

Splunk Use Case Implementation Process



How are use cases developed and implemented? What does it mean to tune a correlation search? This post explains a disciplined process that focuses on the early and continuous delivery of valuable use cases. You'll learn time-saving shortcuts and a systematic approach that eliminates frustration and churning.


Identifying Splunk forwarders that phone home too frequently


As I've worked on many large-scale Splunk environments, a common problem I've noticed is that Splunk forwarders phone home to the deployment server (DS) too frequently. When a forwarder phones home to the DS more often than necessary, it wastes resources on the DS, and can prevent the DS from deploying apps to forwarders correctly.

Splunk saved search (and correlation search!) explorer


As a Splunk administrator, have you ever needed to list out your saved searches in some way? Perhaps you need to know which searches might be accelerated, scheduled, or even real-time scheduled? Here's a quick dashboard to show this information.

The dashboard is available in this GitHub repo.

saved search explorer screenshot


Quick tip when working with stats command


When you're working with the stats command, it's often nice to rename the fields to drop the aggregatation type:

| makeresults count=3 
| streamstats count
| eval foo="bar"
| stats latest(foo) as foo latest(count) as count

This gets annoying because of all the extra typing involved, and it violates the DRY principal.

Instead, try this next time:


Subscribe to RSS - splunk