写在前面:kestrel当前发展很快,官方文档更新也不及时,比如这个:https://opencybersecurityalliance.org/posts/kestrel-2021-07-26/,巨坑!用最新版本的kestrel,里面的语法都出现解析错误,你说尴尬不。。。没办法,自己看源码搞吧,源码测试里仅仅有单元测试的,没有端到端的测试,只能去看底层源码揣摩使用方法。。。总之,我最终修改了elastic_ecs模块下stix_transmission/api_client.py的源码才搞定。
本文目标:kestrel对接elasticsearch,最终能够使用kestrel查询出ES的数据来。最终效果图:
kestrel查询脚本:
browsers = GET process FROM stixshifter://host110 WHERE [process:name IN ('firefox.exe', 'chrome.exe')] START t'2021-01-01T00:00:00Z' STOP t'2021-12-31T00:00:00Z'
DISP browsers ATTR name, pid
输出:
name pid
chrome.exe 12132112121
firefox.exe 121321
firefox.exe 9121321
ES数据源,主要写入了几条:
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/1 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/2 -d '{
"process": {
"name": "chrome.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "12132112121"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/3 -d '{
"process": {
"name": "twitter.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "1213211242123"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/4 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "9121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
在已经安装好了kestrel的前提下,操作如下:
第一步,设置ES的api_key我的ES配置:
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node-1"]
xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12
xpack.security.http.ssl.truststore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12
http.cors.enabled: true
http.cors.allow-origin: "*"
然后就是生成api_key了。
第二步:安装stix_shifter好以后,设置connector环境变量。
pip install stix-shifter-modules-elastic_ecs
export STIXSHIFTER_HOST110_CONFIG='{"auth":{"id":"ZT3QznwBhSK3ri59dnDv", "api_key":"oY8lmKTpTOOXxNqcwJuiqA"}}'
export STIXSHIFTER_HOST110_CONNECTION='{"host":"localhost", "port":9200, "indices":"host110"}'
export STIXSHIFTER_HOST110_CONNECTOR=elastic_ecs
补充说下stix_shifter的用途,本质上是将kestrel lang对ES数据的查询语句转换为ES的查询语法。
例如,我的脚本中:
GET process FROM stixshifter://host110 WHERE [process:name IN ('firefox.exe', 'chrome.exe')] START t'2021-01-01T00:00:00Z' STOP t'2021-12-31T00:00:00Z'
这条语句会被stix_shifter转换为如下ES查询:
{'Content-Type': 'application/json'} search data==> {'_source': {'includes': ['@timestamp', 'source.*', 'destination.*', 'event.*', 'client.*', 'server.*', 'host.*', 'network.*', 'process.*', 'user.*', 'file.*', 'url.*', 'registry.*', 'dns.*']}, 'query': {'query_string': {'query': '(process.pid : ("12132112121" OR "9121321" OR "121321") OR process.ppid : ("12132112121" OR "9121321" OR "121321") OR process.parent.pid : ("12132112121" OR "9121321" OR "121321") OR process.parent.ppid : ("12132112121" OR "9121321" OR "121321")) AND (@timestamp:["2021-01-01T00:00:00.000Z" TO "2021-12-31T00:00:00.000Z"])'}}}
当然要通过追踪kestrel源码追踪分析才知道。
第三步:kestrel在对接ES自签名证书的时候,有bug会一致报这样错误,真是蛋疼啊,我是通过源码分析才发现的。
错误如下:
File "/root/bone/huntingspace/lib/python3.8/site-packages/kestrel_datasource_stixshifter/interface.py", line 151, in query
raise DataSourceError(
kestrel.exceptions.DataSourceError: [ERROR] DataSourceError: data source internal error: STIX-shifter transmission.results() failed. please test data source manually.
然后你深入源码分析才知道是这个错误:
"Wrong certificate: HTTPSConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /_cluster/health?pretty (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1131)')))", 'code': 'authentication_fail'}
就是因为自签名证书的问题,解决方式如下:
修改源码:site-packages/stix_shifter_modules/elastic_ecs/stix_transmission/api_client.py
42 self.client = RestApiClient(connection.get('host'),
43 connection.get('port'),
44 headers,
45 url_modifier_function=url_modifier_function,
46 cert_verify= connection.get('selfSignedCert', True),
47 sni=connection.get('sni', None)
48 )
将46行的True修改为False就好了!
第四步,ES写入mapping和数据,一个bash脚本:
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110?pretty
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110?pretty -d '{"mappings": {
"properties": {
"process": {
"type": "object",
"properties": {
"@timestamp": {"type": "date"},
"name": {"type": "text"},
"pid": {"type": "text"},
"content": {"type": "text"}
}
}}}}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/1 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/2 -d '{
"process": {
"name": "chrome.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "12132112121"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/3 -d '{
"process": {
"name": "twitter.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "1213211242123"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/4 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "9121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
第5步: 使用最初的kestrel脚本运行即可出现目标结果了。