UFCFFY-15-M Cyber Security Analytics¶
Practical Lab 7: Working with Splunk and Python¶
In this lab, we will explore how we can integrate Splunk and Python. Whilst Splunk provides sophisticated means of analysis through the Splunk Query Language, the ability to integrate Python allows for even greater control of how we analyse and process our data. We can then make use of the extensive data science libraries in Python to perform further analytics.
Useful links:
- https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/
- https://github.com/splunk/splunk-sdk-python
- https://docs.splunk.com/DocumentationStatic/PythonSDK/1.6.13/client.html
General Guidance to Setup with DetectionLab instance
You can use the following examples to link up your UWEcyber VM with your DetectionLab logger machine that has an instance of Splunk running on it. This may be useful if you wish to obtain data from your DetectionLab to explore further in Python and Pandas.
- First, make sure you have the Splunk SDK installed in your UWEcyber VM:
python3 -m pip install splunk-sdk - Assuming that your DetectionLab is installed and running, you should be able to navigate to Splunk using a web browser - check that you can access Splunk at https://192.168.56.105:8000
- From the DetectionLab/Vagrant directory, type:
vagrant ssh loggerto log into the logger machine via ssh. - Navigate to /opt/splunk/etc/system/local and type
sudo nano server.conf - Add the line
allowRemoteLogin = alwaysunder the [general] section of the configuration file. Save the file with Ctrl+S and exit with Ctrl+X. - Navigate to /opt/splunk/bin and type
sudo ./splunk restartto restart Splunk. If asked for a password for user vagrant, typevagrant. - Type
exitto exit the ssh session.
Basics of cURL usage for Splunk
https://docs.splunk.com/Documentation/SplunkCloud/8.2.2112/RESTTUT/RESTsearches
- Try the following command
curl -u admin:changeme -k https://192.168.56.105:8089/services/search/jobs -d search="search *". If the above is configured correctly, then your default credentials will allow you to get a XML response that contains a search ID. You can then follow this up for more information.
import splunklib.client as client
HOST = "192.168.56.105"
PORT = 8089
USERNAME = "admin"
PASSWORD = "changeme"
# Create a Service instance and log in
service = client.connect(
host=HOST,
port=PORT,
username=USERNAME,
password=PASSWORD)
# Print installed apps to the console to verify login
for app in service.apps:
print (app.name)
import pandas as pd
kwargs_oneshot = {"earliest_time": "2021-10-1T12:00:00.00", "latest_time": "2022-10-1T12:00:00.00"}
searchquery_oneshot = "| tstats count WHERE index=* by index"
oneshotsearch_results = service.jobs.oneshot(searchquery_oneshot, **kwargs_oneshot)
def splunk_to_pandas(reader):
d = []
for item in reader:
entry = {}
for i in item:
entry[i] = item[i]
d.append(entry)
d = pd.DataFrame(d)
return d
reader = results.ResultsReader(oneshotsearch_results)
d = splunk_to_pandas(reader)
d
import splunklib.results as results
kwargs_oneshot = {"earliest_time": "2021-10-1T12:00:00.00", "latest_time": "2022-10-1T12:00:00.00"}
searchquery_oneshot = "search index=suricata | head 10"
oneshotsearch_results = service.jobs.oneshot(searchquery_oneshot, **kwargs_oneshot)
reader = results.ResultsReader(oneshotsearch_results)
d = splunk_to_pandas(reader)
d
import splunklib.results as results
kwargs_oneshot = {"earliest_time": "2021-10-1T12:00:00.00", "latest_time": "2022-10-1T12:00:00.00"}
searchquery_oneshot = "search index=zeek | head 10"
oneshotsearch_results = service.jobs.oneshot(searchquery_oneshot, **kwargs_oneshot)
reader = results.ResultsReader(oneshotsearch_results)
d = splunk_to_pandas(reader)
d