Splunk Request Logging

SCENARIO

Walking through troubleshooting since the virtual server will show up, the pool will show up but going to the URL the application doesn't come up as long as on the logging profile you have Respond on error enabled and your logging pool has no available members.

BELOW IS AN EXAMPLE

VIRTUAL SERVER

ltm virtual /Integration/vs.sim1.102799.qa.enterpriseremarketing.int.thezah.com.443 {
    destination /Integration/10.46.65.206:443
    ip-protocol tcp
    last-modified-time 2021-11-15:10:18:54
    mask 255.255.255.255
    partition Integration
    persist {
        cookie {
            default yes
        }
    }
    pool /Integration/pool.sim1.102799.qa.enterpriseremarketing.int.thezah.com.443
    profiles {
        http { }
        logprofile { }
        oneconnect { }
        serverssl {
            context serverside
        }
        ssl.client.qa.enterpriseremarketing.int.thezah.com {
            context clientside
        }
        tcp-lan-optimized { }
    }
    rules {
        /Integration/irule.qa.enterpriseremarketing.int.thezah.com.content.redirect
    }
    serverssl-use-sni disabled
    source 0.0.0.0/0
    source-address-translation {
        pool MOD_SNAT_POOL
        type snat
    }
    translate-address enabled
    translate-port enabled
    vs-index 1492
}

Run a tcpdump against the IP address of the Virtual Server to see what's going on since Virtual Server shows up, Pool shows up

tcpdump -s0 -nni 0.0:nnnp -vvv -w /var/tmp/qa.enterprise_20200312.pcap host 10.46.65.206

In order to utilize the Wireshark F5 plugin, you need to flag the tcpdump command appropriately with -s0 and setting the level of noise by flagging the interface with a colon followed by a single, double, or triple n for, respectively, low, medium, and high details.

In Wireshark use the display filter: f5ethtrailer.rstcausetxt in order to get the same screen as show above

So from this you can see the F5 is RESETTING the connection (F5RST) and come to find out its because of the below setting in an attached logprofile

Now the when a virtual server has the logprofile attached it sends an extreme amount of data to Splunk so it only gets turned on critical applications or if you are troubleshooting an application so you can get more log data to help find out whats going on exactly.

With Respond on Error = Enabled it means when the Splunk Servers go down then the application also goes down.  With Respond on Error = Disabled (which is the default setting) then when the logging server goes down, the applications will continue to function.

Easy CLI command to change this to disable is

tmsh modify ltm profile request-log logprofile proxy-respond-on-logging-error no

Hope you find this helpful.  Also note that the wireshark capture above to get the F5RST to be displayed you have to either install the wireshark plugin on Wireshark 2.5 and older and on Wireshark 2.6 and newer you just need to enable it via Analyze - Enabled Protocols - F5 Ethernet trailer - f5ethtrailer

Now you can