High CPU
Following are some of the functionality and high CPU related debugging issues encoutered and the best practices to follow when working with Web App Firewall:
Check Policy hits, Bindings, Network configuration, Web App Firewall configuration:
-
Identify misconfiguration
-
Identify vserver that is serving the affected traffic
Inspect logs in the following log files for security violations and recent configuration changes:
-
/var/log/ns.log -
/var/nslog/import.log -
/var/nslog/aslearn.log -
tail -f /var/log/ns.log | grep APPFW_SIGNATURE_MATCH
Example:
Jun 13 01:11:09 <local0.info> 10.217.31.98 CEF:0|Citrix|NetScaler|NS11.0|APPFW| APPFW_SIGNATURE_MATCH|6|src=10.217.253.62 spt=61141 method=GET request= http://aaron.stratum8.net/FFC/wwwboard/passwd.txt msg=Signature violation rule ID 807: web-cgi /wwwboard/passwd.txt access cn1=140 cn2=841 cs1=pr_ffc cs2=PPE0 cs3=OyTgjbXBqcpBFeENKDlde3OkMQ00001 cs4=ALERT cs5=2015 cs6=web-cgi act=not blocked
Isolate the traffic that is effected:
-
Isolate the profile
-
Isolate the security check
-
Isolate the URL, virtual server and traffic parameters
Conditional profile level trace helps identify the traffic and violation records:
-
set appfw profile <profile> -trace ON -
start nstrace -mode APPFW -size 0 -
stop nstrace
Note: Ensure that the trace is collected with -size 0 option.
Check appfw, dht, IP reputation activity counters:
-
nsconmsg -g as_ -g appfwreq_ -g iprep -d current
Monitor window size for resets in connection:
Appfw sets the window size to 9845 when NetScaler resets the connection due to an invalid http message.
Examples:
-
Malformed request received - connection reset
-
High CPU related issues
-
Check data sheets for system limits
-
Inspect for cpu usage, appfw, DHT and memory related activity. Monitor appfw sessions
-
nsconmsg -g cc_cpu_use -g appfwreq -g as -g dht -g mem_AS_OBJ -g mem_AS_COMPONENT -d current
Monitor memory allocated and freed from Web App Firewall components and objects during the target time period. It helps in isolating the protection leading to high CPU usage.
-
Profiler output
-
Observe logs
Isolate appfw check leading to high CPU:
-
startURLClosure
-
Formfiledconsistency
-
CSRF
-
Cookie protections
-
Referer header check
Ascertain that autoupdate of signatures is not leading to high CPU (Disable to confirm).