今早splunk搜索服务器出现了异常，本来转发器有4个，上班发现只能搜索到2个，很奇怪。后来别人说转发器可能有限制，但经过分析，转发器是没有限制的，找到两个问题点
1:查找splunk日志，发现有吞吐量限制，默认为256
cat /opt/splunkforwarder/var/log/splunk/splunkd.log |grep limits
09-03-2014 10:59:48.466 +0800 WARN FileTracker - migrating maxDataSize value=500 from _thefishbucket in indexes.conf to limits.conf stanza=inputproc setting=file_tracking_db_threshold_mb
09-03-2014 11:05:30.726 +0800 INFO ThruputProcessor - Current data throughput (258 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
09-03-2014 11:10:30.735 +0800 INFO ThruputProcessor - Current data throughput (261 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
09-03-2014 11:15:30.980 +0800 INFO ThruputProcessor - Current data throughput (284 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
09-03-2014 11:20:31.230 +0800 INFO ThruputProcessor - Current data throughput (258 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
vim /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf
# Version 6.1.3
[thruput]
maxKBps = 600
之后修改了下吞吐量为600Kb/s，问题依旧
2：经查看，是我的日志程序进程死掉，导致日志文件0KB，splunk搜索不到是正常的啊（solunk默认是会忽略0kb的文件哈），我改了下日志文件，这时候就可以正常写入日志了，splunk会立马搜索到数据的，呵呵
