UFCFFY-15-M Cyber Security Analytics

Practical Lab 9: Insider Threat Detection


In this lab, we will explore an insider threat detection case. From the data provided, can you identify a suspicious user within this organisation?

Load in the data

In [4]:
import random
import string
import pandas as pd
import matplotlib.pyplot as plt
import seaborn
import datetime

login_data = pd.read_csv('http://www.cems.uwe.ac.uk/~pa-legg/sdav/data/example/login_data.csv')
web_data = pd.read_csv('http://www.cems.uwe.ac.uk/~pa-legg/sdav/data/example/web_data.csv')
file_data = pd.read_csv('http://www.cems.uwe.ac.uk/~pa-legg/sdav/data/example/file_data.csv')
employee_data = pd.read_csv('http://www.cems.uwe.ac.uk/~pa-legg/sdav/data/example/employee_data.csv')

#email_data = pd.read_csv('http://www.cems.uwe.ac.uk/~pa-legg/sdav/data/example/email_data.csv')
#usb_data = pd.read_csv('http://www.cems.uwe.ac.uk/~pa-legg/sdav/data/example/usb_data.csv')
In [5]:
login_data['datetime'] = pd.to_datetime(login_data['datetime'])
web_data['datetime'] = pd.to_datetime(web_data['datetime'])
file_data['datetime'] = pd.to_datetime(file_data['datetime'])

# email_data['datetime'] = pd.to_datetime(email_data['datetime'])
# usb_data['datetime'] = pd.to_datetime(usb_data['datetime'])

Explore the data

In [6]:
display(employee_data)
Unnamed: 0 user role email pc
0 0 usr-uda Security usr-uda@lockdown-lockups.com pc0
1 1 usr-hhe Security usr-hhe@lockdown-lockups.com pc1
2 2 usr-vxr Finance usr-vxr@lockdown-lockups.com pc2
3 3 usr-nba Finance usr-nba@lockdown-lockups.com pc3
4 4 usr-hqt Finance usr-hqt@lockdown-lockups.com pc4
... ... ... ... ... ...
244 244 usr-jwo Finance usr-jwo@lockdown-lockups.com pc244
245 245 usr-hiz Security usr-hiz@lockdown-lockups.com pc245
246 246 usr-svz Services usr-svz@lockdown-lockups.com pc246
247 247 usr-ndr HR usr-ndr@lockdown-lockups.com pc247
248 248 usr-eie Finance usr-eie@lockdown-lockups.com pc248

249 rows × 5 columns

In [7]:
display(login_data)
Unnamed: 0 datetime user action pc
0 0 2020-01-01 00:21:33 usr-hyo login pc205
1 1 2020-01-01 00:21:39 usr-ipd login pc230
2 2 2020-01-01 00:34:25 usr-nrx login pc169
3 3 2020-01-01 00:35:10 usr-hfz login pc111
4 4 2020-01-01 00:39:04 usr-hhe login pc1
... ... ... ... ... ...
166825 166825 2020-11-30 23:42:54 usr-ays logoff pc167
166826 166826 2020-11-30 23:44:18 usr-alj logoff pc168
166827 166827 2020-11-30 23:51:29 usr-nic logoff pc180
166828 166828 2020-11-30 23:56:34 usr-vul logoff pc54
166829 166829 2020-11-30 23:57:32 usr-lnn logoff pc64

166830 rows × 5 columns

In [8]:
display(file_data)
Unnamed: 0 datetime user filename
0 0 2020-01-01 00:42:25.544227 usr-ipd /policy
1 1 2020-01-01 00:50:48.627385 usr-hyo /do_not_delete
2 2 2020-01-01 01:01:38.409035 usr-hyo /newsletter
3 3 2020-01-01 01:14:49.310254 usr-hyo /tech
4 4 2020-01-01 01:17:22.914953 usr-hyo /newsletter
... ... ... ... ...
3799265 3799265 2020-11-30 23:40:54.695141 usr-lnn /newsletter/general
3799266 3799266 2020-11-30 23:48:35.828385 usr-nic /secret
3799267 3799267 2020-11-30 23:49:02.955959 usr-vul /company_profile
3799268 3799268 2020-11-30 23:49:28.216624 usr-nic /private/staffreview
3799269 3799269 2020-11-30 23:49:33.944279 usr-nic /private/staffreview

3799270 rows × 4 columns

Preparing our data

We know that users are assigned a role. We make the assumption that users in the same job role will share some commonalities in how they work and what resources they use. Therefore, we can compare users against peers in the same job role, as well as against their own former activity.

In [9]:
# Example as given to students for getting role/user hierarchy
user_set = {}
user_set_emails = {}
all_roles = employee_data['role'].unique()
for role in all_roles:
    user_set[role] = list(employee_data[employee_data['role'] == role]['user'].values)
    user_set_emails[role] = list(employee_data[employee_data['role'] == role]['email'].values)
user_set_emails
Out[9]:
{'Security': ['usr-uda@lockdown-lockups.com',
  'usr-hhe@lockdown-lockups.com',
  'usr-zay@lockdown-lockups.com',
  'usr-mdl@lockdown-lockups.com',
  'usr-yjc@lockdown-lockups.com',
  'usr-hvk@lockdown-lockups.com',
  'usr-ybh@lockdown-lockups.com',
  'usr-ryi@lockdown-lockups.com',
  'usr-duj@lockdown-lockups.com',
  'usr-scc@lockdown-lockups.com',
  'usr-cyr@lockdown-lockups.com',
  'usr-ngr@lockdown-lockups.com',
  'usr-hzp@lockdown-lockups.com',
  'usr-rnt@lockdown-lockups.com',
  'usr-hfz@lockdown-lockups.com',
  'usr-naf@lockdown-lockups.com',
  'usr-kzn@lockdown-lockups.com',
  'usr-olj@lockdown-lockups.com',
  'usr-jmr@lockdown-lockups.com',
  'usr-pzz@lockdown-lockups.com',
  'usr-hpt@lockdown-lockups.com',
  'usr-hui@lockdown-lockups.com',
  'usr-nbg@lockdown-lockups.com',
  'usr-alj@lockdown-lockups.com',
  'usr-nrx@lockdown-lockups.com',
  'usr-kvv@lockdown-lockups.com',
  'usr-xwd@lockdown-lockups.com',
  'usr-wyj@lockdown-lockups.com',
  'usr-tzd@lockdown-lockups.com',
  'usr-uka@lockdown-lockups.com',
  'usr-npr@lockdown-lockups.com',
  'usr-ipd@lockdown-lockups.com',
  'usr-hiz@lockdown-lockups.com'],
 'Finance': ['usr-vxr@lockdown-lockups.com',
  'usr-nba@lockdown-lockups.com',
  'usr-hqt@lockdown-lockups.com',
  'usr-gyk@lockdown-lockups.com',
  'usr-tiz@lockdown-lockups.com',
  'usr-eqp@lockdown-lockups.com',
  'usr-avx@lockdown-lockups.com',
  'usr-zjh@lockdown-lockups.com',
  'usr-hsh@lockdown-lockups.com',
  'usr-gro@lockdown-lockups.com',
  'usr-xkb@lockdown-lockups.com',
  'usr-qcf@lockdown-lockups.com',
  'usr-zuq@lockdown-lockups.com',
  'usr-rjv@lockdown-lockups.com',
  'usr-nra@lockdown-lockups.com',
  'usr-wer@lockdown-lockups.com',
  'usr-sgi@lockdown-lockups.com',
  'usr-utk@lockdown-lockups.com',
  'usr-zge@lockdown-lockups.com',
  'usr-inp@lockdown-lockups.com',
  'usr-ssv@lockdown-lockups.com',
  'usr-lhu@lockdown-lockups.com',
  'usr-uby@lockdown-lockups.com',
  'usr-nvl@lockdown-lockups.com',
  'usr-vmk@lockdown-lockups.com',
  'usr-oza@lockdown-lockups.com',
  'usr-xgk@lockdown-lockups.com',
  'usr-uyp@lockdown-lockups.com',
  'usr-jwo@lockdown-lockups.com',
  'usr-eie@lockdown-lockups.com'],
 'Legal': ['usr-gwu@lockdown-lockups.com',
  'usr-nho@lockdown-lockups.com',
  'usr-utl@lockdown-lockups.com',
  'usr-wcb@lockdown-lockups.com',
  'usr-uct@lockdown-lockups.com',
  'usr-miu@lockdown-lockups.com',
  'usr-bde@lockdown-lockups.com',
  'usr-zvn@lockdown-lockups.com',
  'usr-cka@lockdown-lockups.com',
  'usr-exs@lockdown-lockups.com',
  'usr-gdd@lockdown-lockups.com',
  'usr-xbv@lockdown-lockups.com',
  'usr-rdb@lockdown-lockups.com',
  'usr-xzr@lockdown-lockups.com',
  'usr-shs@lockdown-lockups.com',
  'usr-tcn@lockdown-lockups.com',
  'usr-ime@lockdown-lockups.com',
  'usr-cao@lockdown-lockups.com',
  'usr-ibc@lockdown-lockups.com',
  'usr-hha@lockdown-lockups.com',
  'usr-yil@lockdown-lockups.com',
  'usr-ibd@lockdown-lockups.com',
  'usr-cvk@lockdown-lockups.com',
  'usr-vxf@lockdown-lockups.com',
  'usr-sgw@lockdown-lockups.com',
  'usr-hga@lockdown-lockups.com',
  'usr-hep@lockdown-lockups.com',
  'usr-sgo@lockdown-lockups.com',
  'usr-ylf@lockdown-lockups.com',
  'usr-fjd@lockdown-lockups.com',
  'usr-wpu@lockdown-lockups.com',
  'usr-rkl@lockdown-lockups.com',
  'usr-syq@lockdown-lockups.com',
  'usr-fja@lockdown-lockups.com',
  'usr-mpc@lockdown-lockups.com',
  'usr-dyg@lockdown-lockups.com',
  'usr-now@lockdown-lockups.com'],
 'HR': ['usr-wnk@lockdown-lockups.com',
  'usr-rxh@lockdown-lockups.com',
  'usr-zkm@lockdown-lockups.com',
  'usr-uoe@lockdown-lockups.com',
  'usr-iqy@lockdown-lockups.com',
  'usr-qtg@lockdown-lockups.com',
  'usr-qod@lockdown-lockups.com',
  'usr-wub@lockdown-lockups.com',
  'usr-ivp@lockdown-lockups.com',
  'usr-kzd@lockdown-lockups.com',
  'usr-fio@lockdown-lockups.com',
  'usr-gmb@lockdown-lockups.com',
  'usr-pqq@lockdown-lockups.com',
  'usr-chl@lockdown-lockups.com',
  'usr-xre@lockdown-lockups.com',
  'usr-eui@lockdown-lockups.com',
  'usr-lsr@lockdown-lockups.com',
  'usr-zbl@lockdown-lockups.com',
  'usr-xzh@lockdown-lockups.com',
  'usr-bph@lockdown-lockups.com',
  'usr-adi@lockdown-lockups.com',
  'usr-oov@lockdown-lockups.com',
  'usr-pmh@lockdown-lockups.com',
  'usr-dyn@lockdown-lockups.com',
  'usr-xpo@lockdown-lockups.com',
  'usr-qpu@lockdown-lockups.com',
  'usr-cee@lockdown-lockups.com',
  'usr-wew@lockdown-lockups.com',
  'usr-mmw@lockdown-lockups.com',
  'usr-swu@lockdown-lockups.com',
  'usr-uow@lockdown-lockups.com',
  'usr-kfp@lockdown-lockups.com',
  'usr-zik@lockdown-lockups.com',
  'usr-qgc@lockdown-lockups.com',
  'usr-hmw@lockdown-lockups.com',
  'usr-iji@lockdown-lockups.com',
  'usr-ryw@lockdown-lockups.com',
  'usr-gxp@lockdown-lockups.com',
  'usr-dgj@lockdown-lockups.com',
  'usr-vbe@lockdown-lockups.com',
  'usr-tzt@lockdown-lockups.com',
  'usr-ivc@lockdown-lockups.com',
  'usr-frc@lockdown-lockups.com',
  'usr-ndr@lockdown-lockups.com'],
 'Services': ['usr-ord@lockdown-lockups.com',
  'usr-viw@lockdown-lockups.com',
  'usr-upm@lockdown-lockups.com',
  'usr-xsn@lockdown-lockups.com',
  'usr-mlh@lockdown-lockups.com',
  'usr-mmt@lockdown-lockups.com',
  'usr-qir@lockdown-lockups.com',
  'usr-oax@lockdown-lockups.com',
  'usr-yfr@lockdown-lockups.com',
  'usr-adl@lockdown-lockups.com',
  'usr-evp@lockdown-lockups.com',
  'usr-fiq@lockdown-lockups.com',
  'usr-coz@lockdown-lockups.com',
  'usr-rac@lockdown-lockups.com',
  'usr-ami@lockdown-lockups.com',
  'usr-sqk@lockdown-lockups.com',
  'usr-jok@lockdown-lockups.com',
  'usr-qie@lockdown-lockups.com',
  'usr-eid@lockdown-lockups.com',
  'usr-mki@lockdown-lockups.com',
  'usr-naq@lockdown-lockups.com',
  'usr-rvx@lockdown-lockups.com',
  'usr-vdx@lockdown-lockups.com',
  'usr-gnv@lockdown-lockups.com',
  'usr-con@lockdown-lockups.com',
  'usr-way@lockdown-lockups.com',
  'usr-rtk@lockdown-lockups.com',
  'usr-otx@lockdown-lockups.com',
  'usr-nro@lockdown-lockups.com',
  'usr-ikk@lockdown-lockups.com',
  'usr-mys@lockdown-lockups.com',
  'usr-xfo@lockdown-lockups.com',
  'usr-tra@lockdown-lockups.com',
  'usr-une@lockdown-lockups.com',
  'usr-miy@lockdown-lockups.com',
  'usr-xrb@lockdown-lockups.com',
  'usr-jsv@lockdown-lockups.com',
  'usr-pdn@lockdown-lockups.com',
  'usr-fxq@lockdown-lockups.com',
  'usr-itz@lockdown-lockups.com',
  'usr-hci@lockdown-lockups.com',
  'usr-qln@lockdown-lockups.com',
  'usr-eib@lockdown-lockups.com',
  'usr-rjw@lockdown-lockups.com',
  'usr-okf@lockdown-lockups.com',
  'usr-sjc@lockdown-lockups.com',
  'usr-qmo@lockdown-lockups.com',
  'usr-tbt@lockdown-lockups.com',
  'usr-svz@lockdown-lockups.com'],
 'Technical': ['usr-mcr@lockdown-lockups.com',
  'usr-lfl@lockdown-lockups.com',
  'usr-gsw@lockdown-lockups.com',
  'usr-qat@lockdown-lockups.com',
  'usr-wgw@lockdown-lockups.com',
  'usr-udb@lockdown-lockups.com',
  'usr-zoj@lockdown-lockups.com',
  'usr-peg@lockdown-lockups.com',
  'usr-cvh@lockdown-lockups.com',
  'usr-gvw@lockdown-lockups.com',
  'usr-orw@lockdown-lockups.com',
  'usr-tgw@lockdown-lockups.com',
  'usr-iba@lockdown-lockups.com',
  'usr-ebj@lockdown-lockups.com',
  'usr-hvd@lockdown-lockups.com',
  'usr-zwd@lockdown-lockups.com',
  'usr-mvr@lockdown-lockups.com',
  'usr-lxz@lockdown-lockups.com',
  'usr-tqd@lockdown-lockups.com',
  'usr-xvg@lockdown-lockups.com',
  'usr-qjv@lockdown-lockups.com',
  'usr-stu@lockdown-lockups.com',
  'usr-yhv@lockdown-lockups.com',
  'usr-ocw@lockdown-lockups.com',
  'usr-ays@lockdown-lockups.com',
  'usr-fbi@lockdown-lockups.com',
  'usr-gok@lockdown-lockups.com',
  'usr-szb@lockdown-lockups.com',
  'usr-xsi@lockdown-lockups.com',
  'usr-dry@lockdown-lockups.com',
  'usr-jsn@lockdown-lockups.com',
  'usr-jbh@lockdown-lockups.com'],
 'Director': ['usr-dmi@lockdown-lockups.com',
  'usr-hxr@lockdown-lockups.com',
  'usr-kdj@lockdown-lockups.com',
  'usr-vul@lockdown-lockups.com',
  'usr-bkc@lockdown-lockups.com',
  'usr-lnn@lockdown-lockups.com',
  'usr-hfn@lockdown-lockups.com',
  'usr-bsx@lockdown-lockups.com',
  'usr-sdc@lockdown-lockups.com',
  'usr-vui@lockdown-lockups.com',
  'usr-mwj@lockdown-lockups.com',
  'usr-jjg@lockdown-lockups.com',
  'usr-vcj@lockdown-lockups.com',
  'usr-zwq@lockdown-lockups.com',
  'usr-rhd@lockdown-lockups.com',
  'usr-zov@lockdown-lockups.com',
  'usr-nnk@lockdown-lockups.com',
  'usr-zth@lockdown-lockups.com',
  'usr-nic@lockdown-lockups.com',
  'usr-ypo@lockdown-lockups.com',
  'usr-hyo@lockdown-lockups.com',
  'usr-twi@lockdown-lockups.com',
  'usr-vsl@lockdown-lockups.com',
  'usr-asj@lockdown-lockups.com']}

1 - Initial Investigations on Login Data

In [10]:
login_data['hour'] = login_data['datetime'].dt.hour
plt.figure(figsize=(20,10))

for employee in employee_data['user'].unique():
    d = login_data[login_data['user'] == employee]
    d_login = d[d['action'] == 'login']
    d_logoff = d[d['action'] == 'logoff']
    plt.plot(d_login['hour'])
    plt.plot(d_logoff['hour'])