数据对象
pandas主要有两种数据对象：Series、DataFrame
注: 后面代码使用pandas版本0.20.1，通过import pandas as pd引入
1. Series
Series是一种带有索引的序列对象。
简单创建如下：
# 通过传入一个序列给pd.Series初始化一个Series对象, 比如lists1=pd.Series(list("1234"))print(s1)0 11 22 33 4dtype:object
2. DataFrame
类似与数据库table有行列的数据对象。
创建方式如下：
# 通过传入一个numpy的二维数组或者dict对象给pd.DataFrame初始化一个DataFrame对象
# 通过numpy二维数组import numpy as npdf1 = pd.DataFrame(np.random.randn(6,4))print(df1) 0 1 2 30 -0.646340 -1.249943 0.393323 -1.5618731 0.371630 0.069426 1.693097 0.9074192 -0.328575 -0.256765 0.693798 -0.7873433 1.875764 -0.416275 -1.028718 0.1582594 1.644791 -1.321506 -0.337425 0.8206895 0.006391 -1.447894 0.506203 0.977295
# 通过dict字典df2 = pd.DataFrame({ 'A' : 1., 'B' : pd.Timestamp('20130102'), 'C' :pd.Series(1,index=list(range(4)),dtype='float32'), 'D' : np.array([3] * 4,dtype='int32'), 'E' : pd.Categorical(["test","train","test","train"]), 'F' : 'foo' })print(df2)
A B C D E F0 1.0 2013-01-02 1.0 3 test foo1 1.0 2013-01-02 1.0 3 train foo2 1.0 2013-01-02 1.0 3 test foo3 1.0 2013-01-02 1.0 3 train foo
3. 索引
不管是Series对象还是DataFrame对象都有一个对对象相对应的索引，Series的索引类似于每个元素, DataFrame的索引对应着每一行。
查看：在创建对象的时候，每个对象都会初始化一个起始值为0，自增的索引列表, DataFrame同理。
# 打印对象的时候，第一列就是索引print(s1)0 11 22 33 4dtype: object
# 或者只查看索引, DataFrame同理print(s1.index)
增删查改
这里的增删查改主要基于DataFrame对象，为了有足够数据用于展示，这里选择tushare的数据。
1. tushare安装
ipinstall tushare
创建数据对象如下：
import tushare as tsdf = ts.get_k_data("000001")
DataFrame 行列，axis 图解：
2. 查询
查看每列的数据类型
# 查看df数据类型df.dtypesdate objectopen float64close float64high float64low float64volume float64code objectdtype: object
查看指定指定数量的行：head函数默认查看前5行，tail函数默认查看后5行，可以传递指定的数值用于查看指定行数。
查看前5行df.head()date open close high low volume code0 2015-12-23 9.927 9.935 10.174 9.871 1039018.0 0000011 2015-12-24 9.919 9.823 9.998 9.744 640229.0 0000012 2015-12-25 9.855 9.879 9.927 9.815 399845.0 0000013 2015-12-28 9.895 9.537 9.919 9.537 822408.0 0000014 2015-12-29 9.545 9.624 9.632 9.529 619802.0 000001# 查看后5行df.tail()date open close high low volume code636 2018-08-01 9.42 9.15 9.50 9.11 814081.0 000001637 2018-08-02 9.13 8.94 9.15 8.88 931401.0 000001638 2018-08-03 8.93 8.91 9.10 8.91 476546.0 000001639 2018-08-06 8.94 8.94 9.11 8.89 554010.0 000001640 2018-08-07 8.96 9.17 9.17 8.88 690423.0 000001# 查看前10行df.head(10)date open close high low volume code0 2015-12-23 9.927 9.935 10.174 9.871 1039018.0 0000011 2015-12-24 9.919 9.823 9.998 9.744 640229.0 0000012 2015-12-25 9.855 9.879 9.927 9.815 399845.0 0000013 2015-12-28 9.895 9.537 9.919 9.537 822408.0 0000014 2015-12-29 9.545 9.624 9.632 9.529 619802.0 0000015 2015-12-30 9.624 9.632 9.640 9.513 532667.0 0000016 2015-12-31 9.632 9.545 9.656 9.537 491258.0 0000017 2016-01-04 9.553 8.995 9.577 8.940 563497.0 0000018 2016-01-05 8.972 9.075 9.210 8.876 663269.0 0000019 2016-01-06 9.091 9.179 9.202 9.067 515706.0 000001
查看某一行或多行,某一列或多列
# 查看第一行df[0:1] date open close high low volume code0 2015-12-23 9.927 9.935 10.174 9.871 1039018.0 000001
# 查看 10到20行df[10:21] date open close high low volume code10 2016-01-07 9.083 8.709 9.083 8.685 174761.0 00000111 2016-01-08 8.924 8.852 8.987 8.677 747527.0 00000112 2016-01-11 8.757 8.566 8.820 8.502 732013.0 00000113 2016-01-12 8.621 8.605 8.685 8.470 561642.0 00000114 2016-01-13 8.669 8.526 8.709 8.518 391709.0 00000115 2016-01-14 8.430 8.574 8.597 8.343 666314.0 00000116 2016-01-15 8.486 8.327 8.597 8.295 448202.0 00000117 2016-01-18 8.231 8.287 8.406 8.199 421040.0 00000118 2016-01-19 8.319 8.526 8.582 8.287 501109.0 00000119 2016-01-20 8.518 8.390 8.597 8.311 603752.0 00000120 2016-01-21 8.343 8.215 8.558 8.215 606145.0 000001
# 查看看Date列前5个数据df["date"].head() # 或者df.date.head()0 2015-12-231 2015-12-242 2015-12-253 2015-12-284 2015-12-29Name: date, dtype: object
# 查看看Date列,code列, open列前5个数据df[["date","code", "open"]].head() date code open0 2015-12-23 000001 9.9271 2015-12-24 000001 9.9192 2015-12-25 000001 9.8553 2015-12-28 000001 9.8954 2015-12-29 000001 9.545
使用行列组合条件查询
# 查看date, code列的第10行df.loc[10, ["date", "code"]]
date 2016-01-07code 000001Name: 10, dtype: object# 查看date, code列的第10行到20行df.loc[10:20, ["date", "code"]]
date code10 2016-01-07 00000111 2016-01-08 00000112 2016-01-11 00000113 2016-01-12 00000114 2016-01-13 00000115 2016-01-14 00000116 2016-01-15 00000117 2016-01-18 00000118 2016-01-19 00000119 2016-01-20 00000120 2016-01-21 000001
# 查看第一行,open列的数据df.loc[0, "open"]9.9269999999999996
通过位置查询：值得注意的是上面的索引值就是特定的位置。
# 查看第1行()df.iloc[0]date 2015-12-24open 9.919close 9.823high 9.998low 9.744volume 640229code 000001Name: 0, dtype: object# 查看最后一行df.iloc[-1]date 2018-08-08open 9.16close 9.12high 9.16low 9.1volume 29985code 000001Name: 640, dtype: object# 查看第一列，前5个数值df.iloc[:,0].head()0 2015-12-241 2015-12-252 2015-12-283 2015-12-294 2015-12-30Name: date, dtype: object
# 查看前2到4行，第1，3列df.iloc[2:4,[0,2]]
date close2 2015-12-28 9.5373 2015-12-29 9.624
通过条件筛选：
查看open列大于10的前5行df[df.open > 10].head()
date open close high low volume code378 2017-07-14 10.483 10.570 10.609 10.337 1722570.0 000001379 2017-07-17 10.619 10.483 10.987 10.396 3273123.0 000001380 2017-07-18 10.425 10.716 10.803 10.299 2349431.0 000001381 2017-07-19 10.657 10.754 10.851 10.551 1933075.0 000001382 2017-07-20 10.745 10.638 10.880 10.580 1537338.0 000001
# 查看open列大于10且open列小于10.6的前五行df[(df.open > 10) & (df.open < 10.6)].head() date open close high low volume code378 2017-07-14 10.483 10.570 10.609 10.337 1722570.0 000001380 2017-07-18 10.425 10.716 10.803 10.299 2349431.0 000001387 2017-07-27 10.550 10.422 10.599 10.363 1194490.0 000001388 2017-07-28 10.441 10.569 10.638 10.412 819195.0 000001390 2017-08-01 10.471 10.865 10.904 10.432 2035709.0 000001
# 查看open列大于10或open列小于10.6的前五行df[(df.open > 10) | (df.open < 10.6)].head() date open close high low volume code0 2015-12-24 9.919 9.823 9.998 9.744 640229.0 0000011 2015-12-25 9.855 9.879 9.927 9.815 399845.0 0000012 2015-12-28 9.895 9.537 9.919 9.537 822408.0 0000013 2015-12-29 9.545 9.624 9.632 9.529 619802.0 0000014 2015-12-30 9.624 9.632 9.640 9.513 532667.0 000001
3. 增加
在前面已经简单的说明Series, DataFrame的创建，这里说一些常用有用的创建方式。
# 创建2018-08-08到2018-08-15的时间序列，默认时间间隔为Days2 = pd.date_range("20180808", periods=7)print(s2)
DatetimeIndex(['2018-08-08', '2018-08-09', '2018-08-10', '2018-08-11', '2018-08-12', '2018-08-13', '2018-08-14'], dtype='datetime64[ns]', freq='D')# 指定2018-08-08 00：00 到2018-08-09 00:00 时间间隔为小时# freq参数可使用参数, 参考: http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases s3 = pd.date_range("20180808", "20180809", freq="H")print(s2)
DatetimeIndex(['2018-08-08 00:00:00', '2018-08-08 01:00:00', '2018-08-08 02:00:00', '2018-08-08 03:00:00', '2018-08-08 04:00:00', '2018-08-08 05:00:00', '2018-08-08 06:00:00', '2018-08-08 07:00:00', '2018-08-08 08:00:00', '2018-08-08 09:00:00', '2018-08-08 10:00:00', '2018-08-08 11:00:00', '2018-08-08 12:00:00', '2018-08-08 13:00:00', '2018-08-08 14:00:00', '2018-08-08 15:00:00', '2018-08-08 16:00:00', '2018-08-08 17:00:00', '2018-08-08 18:00:00', '2018-08-08 19:00:00', '2018-08-08 20:00:00', '2018-08-08 21:00:00', '2018-08-08 22:00:00', '2018-08-08 23:00:00', '2018-08-09 00:00:00'], dtype='datetime64[ns]', freq='H')# 通过已有序列创建时间序列s4 = pd.to_datetime(df.date.head())print(s4)
0 2015-12-241 2015-12-252 2015-12-283 2015-12-294 2015-12-30Name: date, dtype: datetime64[ns]
4. 修改
# 将df 的索引修改为date列的数据,并且将类型转换为datetime类型df.index = pd.to_datetime(df.date)df.head()
date open close high low volume code date 2015-12-24 2015-12-24 9.919 9.823 9.998 9.744 640229.0 0000012015-12-25 2015-12-25 9.855 9.879 9.927 9.815 399845.0 0000012015-12-28 2015-12-28 9.895 9.537 9.919 9.537 822408.0 0000012015-12-29 2015-12-29 9.545 9.624 9.632 9.529 619802.0 0000012015-12-30 2015-12-30 9.624 9.632 9.640 9.513 532667.0 000001# 修改列的字段df.columns = ["Date", "Open","Close","High","Low","Volume","Code"]print(df.head())
Date Open Close High Low Volume Code date2015-12-24 2015-12-24 9.919 9.823 9.998 9.744 640229.0 0000012015-12-25 2015-12-25 9.855 9.879 9.927 9.815 399845.0 0000012015-12-28 2015-12-28 9.895 9.537 9.919 9.537 822408.0 0000012015-12-29 2015-12-29 9.545 9.624 9.632 9.529 619802.0 0000012015-12-30 2015-12-30 9.624 9.632 9.640 9.513 532667.0 000001# 将Open列每个数值加1, apply方法并不直接修改源数据，所以需要将新值复制给dfdf.Open = df.Open.apply(lambda x: x+1)df.head()
Date Open Close High Low Volume Code date2015-12-24 2015-12-24 10.919 9.823 9.998 9.744 640229.0 0000012015-12-25 2015-12-25 10.855 9.879 9.927 9.815 399845.0 0000012015-12-28 2015-12-28 10.895 9.537 9.919 9.537 822408.0 0000012015-12-29 2015-12-29 10.545 9.624 9.632 9.529 619802.0 0000012015-12-30 2015-12-30 10.624 9.632 9.640 9.513 532667.0 000001# 将Open，Close列都数值上加1,如果多列，apply接收的对象是整个列df[["Open", "Close"]].head().apply(lambda x: x.apply(lambda x: x+1))
Open Closedate 2015-12-24 11.919 10.8232015-12-25 11.855 10.8792015-12-28 11.895 10.5372015-12-29 11.545 10.6242015-12-30 11.624 10.632
5. 删除
通过drop方法drop指定的行或者列。
注意: drop方法并不直接修改源数据，如果需要使源dataframe对象被修改，需要传入inplace=True，通过之前的axis图解，知道行的值(或者说label)在axis=0，列的值(或者说label)在axis=1。
# 删除指定列,删除Open列df.drop("Open", axis=1).head() #或者df.drop(df.columns[1])
Date Close High Low Volume Code date
2015-12-24 2015-12-24 9.823 9.998 9.744 640229.0 0000012015-12-25 2015-12-25 9.879 9.927 9.815 399845.0 0000012015-12-28 2015-12-28 9.537 9.919 9.537 822408.0 0000012015-12-29 2015-12-29 9.624 9.632 9.529 619802.0 0000012015-12-30 2015-12-30 9.632 9.640 9.513 532667.0 000001# 删除第1，3列. 即Open,High列df.drop(df.columns[[1,3]], axis=1).head() # 或df.drop(["Open", "High], axis=1).head() Date Close Low Volume Code date 2015-12-24 2015-12-24 9.823 9.744 640229.0 000001 2015-12-25 2015-12-25 9.879 9.815 399845.0 000001 2015-12-28 2015-12-28 9.537 9.537 822408.0 000001 2015-12-29 2015-12-29 9.624 9.529 619802.0 000001 2015-12-30 2015-12-30 9.632 9.513 532667.0 000001
pandas常用参数
数值显示格式：当数值很大的时候pandas默认会使用科学计数法
# float数据类型以{:.4f}格式显示，即显示完整数据且保留后四位pd.options.display.float_format = '{:.4f}'.format
pandas常用函数
1. 统计
# descibe方法会计算每列数据对象是数值的count, mean, std, min, max, 以及一定比率的值df.describe()
Open Close High Low Volumecount 641.0000 641.0000 641.0000 641.0000 641.0000mean 10.7862 9.7927 9.8942 9.6863 833968.6162std 1.5962 1.6021 1.6620 1.5424 607731.6934min 8.6580 7.6100 7.7770 7.4990 153901.000025% 9.7080 8.7180 8.7760 8.6500 418387.000050% 10.0770 9.0960 9.1450 8.9990 627656.000075% 11.8550 10.8350 10.9920 10.7270 1039297.0000max 15.9090 14.8600 14.9980 14.4470 4262825.0000
# 单独统计Open列的平均值df.Open.mean()10.786248049922001
# 查看居于95%的值, 默认线性拟合df.Open.quantile(0.95)14.187
# 查看Open列每个值出现的次数df.Open.value_counts().head()
9.8050 129.8630 109.8440 109.8730 109.8830 8Name: Open, dtype: int64
2. 缺失值处理
删除或者填充缺失值。
# 删除含有NaN的任意行df.dropna(how='any')
# 删除含有NaN的任意列df.dropna(how='any', axis=1)
# 将NaN的值改为5df.fillna(value=5)
3. 排序
按行或者列排序, 默认也不修改源数据。
# 按列排序df.sort_index(axis=1).head()
Close Code Date High Low Open Volumedate2015-12-24 9.8230 000001 2015-12-24 9.9980 9.7440 10.9190 640229.00002015-12-25 1.0000 000001 2015-12-25 1.0000 9.8150 10.8550 399845.00002015-12-28 1.0000 000001 2015-12-28 1.0000 9.5370 10.8950 822408.00002015-12-29 9.6240 000001 2015-12-29 9.6320 9.5290 10.5450 619802.00002015-12-30 9.6320 000001 2015-12-30 9.6400 9.5130 10.6240 532667.0000
# 按行排序，不递增df.sort_index(ascending=False).head()
Date Open Close High Low Volume Code date2018-08-08 2018-08-08 10.1600 9.1100 9.1600 9.0900 153901.0000 0000012018-08-07 2018-08-07 9.9600 9.1700 9.1700 8.8800 690423.0000 0000012018-08-06 2018-08-06 9.9400 8.9400 9.1100 8.8900 554010.0000 0000012018-08-03 2018-08-03 9.9300 8.9100 9.1000 8.9100 476546.0000 0000012018-08-02 2018-08-02 10.1300 8.9400 9.1500 8.8800 931401.0000 000001
安装某一列的值排序
# 按照Open列的值从小到大排序df.sort_values(by="Open") Date Open Close High Low Volume Codedate 2016-03-01 2016-03-01 8.6580 7.7220 7.7770 7.6260 377910.0000 0000012016-02-15 2016-02-15 8.6900 7.7930 7.8410 7.6820 278499.0000 0000012016-01-29 2016-01-29 8.7540 7.9610 8.0240 7.7140 544435.0000 0000012016-03-02 2016-03-02 8.7620 8.0400 8.0640 7.7380 676613.0000 0000012016-02-26 2016-02-26 8.7770 7.7930 7.8250 7.6900 392154.0000 000001
4. 合并
concat, 按照行方向或者列方向合并。
# 分别取0到2行，2到4行，4到9行组成一个列表，通过concat方法按照axis=0，行方向合并, axis参数不指定，默认为0split_rows = [df.iloc[0:2,:],df.iloc[2:4,:], df.iloc[4:9]]pd.concat(split_rows)
Date Open Close High Low Volume Codedate2015-12-24 2015-12-24 10.9190 9.8230 9.9980 9.7440 640229.0000 0000012015-12-25 2015-12-25 10.8550 1.0000 1.0000 9.8150 399845.0000 0000012015-12-28 2015-12-28 10.8950 1.0000 1.0000 9.5370 822408.0000 0000012015-12-29 2015-12-29 10.5450 9.6240 9.6320 9.5290 619802.0000 0000012015-12-30 2015-12-30 10.6240 9.6320 9.6400 9.5130 532667.0000 0000012015-12-31 2015-12-31 10.6320 9.5450 9.6560 9.5370 491258.0000 0000012016-01-04 2016-01-04 10.5530 8.9950 9.5770 8.9400 563497.0000 0000012016-01-05 2016-01-05 9.9720 9.0750 9.2100 8.8760 663269.0000 0000012016-01-06 2016-01-06 10.0910 9.1790 9.2020 9.0670 515706.0000 000001
# 分别取2到3列，3到5列，5列及以后列数组成一个列表，通过concat方法按照axis=1，列方向合并split_columns = [df.iloc[:,1:2], df.iloc[:,2:4], df.iloc[:,4:]]pd.concat(split_columns, axis=1).head()
Open Close High Low Volume Code date2015-12-24 10.9190 9.8230 9.9980 9.7440 640229.0000 0000012015-12-25 10.8550 1.0000 1.0000 9.8150 399845.0000 0000012015-12-28 10.8950 1.0000 1.0000 9.5370 822408.0000 0000012015-12-29 10.5450 9.6240 9.6320 9.5290 619802.0000 0000012015-12-30 10.6240 9.6320 9.6400 9.5130 532667.0000 000001
追加行, 相应的还有insert, 插入插入到指定位置
# 将第一行追加到最后一行df.append(df.iloc[0,:], ignore_index=True).tail()
Date Open Close High Low Volume Code637 2018-08-03 9.9300 8.9100 9.1000 8.9100 476546.0000 000001638 2018-08-06 9.9400 8.9400 9.1100 8.8900 554010.0000 000001639 2018-08-07 9.9600 9.1700 9.1700 8.8800 690423.0000 000001640 2018-08-08 10.1600 9.1100 9.1600 9.0900 153901.0000 000001641 2015-12-24 10.9190 9.8230 9.9980 9.7440 640229.0000 000001
5. 对象复制
由于dataframe是引用对象，所以需要显示调用copy方法用以复制整个dataframe对象。
绘图
pandas的绘图是使用matplotlib，如果想要画的更细致, 可以使用matplotplib，不过简单的画一些图还是不错的。
因为上图太麻烦，这里就不配图了，可以在资源文件里面查看pandas-blog.ipynb文件或者自己敲一遍代码。
# 这里使用notbook，为了直接在输出中显示，需要以下配置%matplotlib inline# 绘制Open,Low,Close.High的线性图df[["Open", "Low", "High", "Close"]].plot()
# 绘制面积图df[["Open", "Low", "High", "Close"]].plot(kind="area")
数据读写
读写常见文件格式，如csv，excel，json等，甚至是读取“系统的剪切板”这个功能有时候很有用。直接将鼠标选中复制的内容读取创建dataframe对象。
# 将df数据保存到当前工作目录的stock.csv文件df.to_csv("stock.csv")
# 查看stock.csv文件前5行with open("stock.csv") as rf: print(rf.readlines()[:5])
['date,Date,Open,Close,High,Low,Volume,Code\n', '2015-12-24,2015-12-24,9.919,9.823,9.998,9.744,640229.0,000001\n', '2015-12-25,2015-12-25,9.855,9.879,9.927,9.815,399845.0,000001\n', '2015-12-28,2015-12-28,9.895,9.537,9.919,9.537,822408.0,000001\n', '2015-12-29,2015-12-29,9.545,9.624,9.632,9.529,619802.0,000001\n']
# 读取stock.csv文件并将第一行作为indexdf2 = pd.read_csv("stock.csv", index_col=0)df2.head()
Date Open Close High Low Volume Codedate 2015-12-24 2015-12-24 9.9190 9.8230 9.9980 9.7440 640229.0000 12015-12-25 2015-12-25 9.8550 9.8790 9.9270 9.8150 399845.0000 12015-12-28 2015-12-28 9.8950 9.5370 9.9190 9.5370 822408.0000 12015-12-29 2015-12-29 9.5450 9.6240 9.6320 9.5290 619802.0000 12015-12-30 2015-12-30 9.6240 9.6320 9.6400 9.5130 532667.0000 1
# 读取stock.csv文件并将第一行作为index，并且将000001作为str类型读取, 不然会被解析成整数df2 = pd.read_csv("stock.csv", index_col=0, dtype={"Code": str})df2.head()
简单实例
这里以处理web日志为例，也许不太实用，因为ELK处理这些绰绰有余，不过喜欢什么自己来也未尝不可。
1. 分析access.log
日志文件: https://raw.githubusercontent.com/Apache-Labor/labor/master/labor-04/labor-04-example-access.log
2. 日志格式及示例
# 日志格式# 字段说明, 参考:https:///wiki/Access.log ％h％l％u％t \“％r \”％> s％b \“％{Referer} i \”\“％{User-Agent} i \”# 具体示例75.249.65.145 US - [2015-09-02 10:42:51.003372] "GET /cms/tina-access-editor-for-download/ HTTP/1.1" 200 7113 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http:///bot.html)" www.example.com 124.165.3.7 443 redirect-handler - + "-" Vea2i8CoAwcAADevXAgAAAAB TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 701 12118 -% 88871 803 0 0 0 0
3. 读取并解析日志文件
解析日志文件
HOST = r'^(?P<host>.*?)'SPACE = r'\s'IDENTITY = r'\S+'USER = r"\S+"TIME = r'\[(?P<time>.*?)\]'# REQUEST = r'\"(?P<request>.*?)\"'REQUEST = r'\"(?P<method>.+?)\s(?P<path>.+?)\s(?P<http_protocol>.*?)\"'STATUS = r'(?P<status>\d{3})'SIZE = r'(?P<size>\S+)'REFER = r"\S+"USER_AGENT = r'\"(?P<user_agent>.*?)\"'
REGEX = HOST+SPACE+IDENTITY+SPACE+USER+SPACE+TIME+SPACE+REQUEST+SPACE+STATUS+SPACE+SIZE+SPACE+IDENTITY+USER_AGENT+SPACEline = '79.81.243.171 - - [30/Mar/2009:20:58:31 +0200] "GET /exemples.php HTTP/1.1" 200 11481 "http://www.facades.fr/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0; .NET CLR 2.0.50727)" "-"'reg = re.compile(REGEX)reg.match(line).groups()
将数据注入DataFrame对象
COLUMNS = ["Host", "Time", "Method", "Path", "Protocol", "status", "size", "User_Agent"]
field_lis = []with open("access.log") as rf: for line in rf: # 由于一些记录不能匹配,所以需要捕获异常, 不能捕获的数据格式如下 # 80.32.156.105 - - [27/Mar/2009:13:39:51 +0100] "GET HTTP/1.1" 400 - "-" "-" "-" # 由于重点不在写正则表达式这里就略过了 try: fields = reg.match(line).groups() except Exception as e: #print(e) #print(line) pass field_lis.append(fields)
log_df = pd.DataFrame(field_lis)# 修改列名log_df.columns = COLUMNS
def parse_time(value): try: return pd.to_datetime(value) except Exception as e: print(e) print(value)
# 将Time列的值修改成pandas可解析的时间格式log_df.Time = log_df.Time.apply(lambda x: x.replace(":", " ", 1))log_df.Time = log_df.Time.apply(parse_time)
# 修改index, 将Time列作为index，并drop掉在Time列log_df.index = pd.to_datetime(log_df.Time) log_df.drop("Time", inplace=True)log_df.head()
Host Time Method Path Protocol status size User_AgentTime2009-03-22 06:00:32 88.191.254.20 2009-03-22 06:00:32 GET / HTTP/1.0 200 8674 "-2009-03-22 06:06:20 66.249.66.231 2009-03-22 06:06:20 GET /popup.php?choix=-89 HTTP/1.1 200 1870 "Mozilla/5.0 (compatible; Googlebot/2.1; +htt...2009-03-22 06:11:20 66.249.66.231 2009-03-22 06:11:20 GET /specialiste.php HTTP/1.1 200 10743 "Mozilla/5.0 (compatible; Googlebot/2.1; +htt...2009-03-22 06:40:06 83.198.250.175 2009-03-22 06:40:06 GET / HTTP/1.1 200 8714 "Mozilla/4.0 (compatible; MSIE 7.0; Windows N...2009-03-22 06:40:06 83.198.250.175 2009-03-22 06:40:06 GET /style.css HTTP/1.1 200 1692 "Mozilla/4.0 (compatible; MSIE 7.0; Windows N...
查看数据类型
# 查看数据类型log_df.dtypes
Host objectTime datetime64[ns]Method objectPath objectProtocol objectstatus objectsize objectUser_Agent objectdtype: object
由上可知, 除了Time字段是时间类型，其他都是object，但是Size, Status应该为数字
def parse_number(value): try: return pd.to_numeric(value) except Exception as e: pass return 0
# 将Size，Status字段值改为数值类型log_df[["Status","Size"]] = log_df[["Status","Size"]].apply(lambda x: x.apply(parse_number))log_df.dtypesHost objectTime datetime64[ns]Method objectPath objectProtocol objectStatus int64Size int64User_Agent objectdtype: object
统计status数据
# 统计不同status值的次数log_df.Status.value_counts()
200 5737304 1540404 1186 400 251302 37403 3206 2Name: Status, dtype: int64
绘制pie图
log_df.Status.value_counts().plot(kind="pie", figsize=(10,8))
查看日志文件时间跨度
log_df.index.max() - log_df.index.min()Timedelta('15 days 11:12:03')
分别查看起始，终止时间
print(log_df.index.max())print(log_df.index.min())
2009-04-06 17:12:352009-03-22 06:00:32
按照此方法还可以统计Method, User_Agent字段 ，不过User_Agent还需要额外清洗以下数据。
统计top 10 IP地址
91.121.31.184 74588.191.254.20 44141.224.252.122 420194.2.62.185 25586.75.35.144 184208.89.192.106 17079.82.3.8 16190.3.72.207 15762.147.243.132 15081.249.221.143 141Name: Host, dtype: int64
绘制请求走势图
log_df2 = log_df.copy()# 为每行加一个request字段，值为1log_df2["Request"] = 1# 每一小时统计一次request数量，并将NaN值替代为0，最后绘制线性图，尺寸为16x9log_df2.Request.resample("H").sum().fillna(0).plot(kind="line",figsize=(16,10))
分别绘图
分别对202，304，404状态重新取样，并放在一个列表里面req_df_lis = [log_df2[log_df2.Status == 200].Request.resample("H").sum().fillna(0), log_df2[log_df2.Status == 304].Request.resample("H").sum().fillna(0), log_df2[log_df2.Status == 404].Request.resample("H").sum().fillna(0) ]
# 将三个dataframe组合起来req_df = pd.concat(req_df_lis,axis=1)req_df.columns = ["200", "304", "404"]# 绘图req_df.plot(figsize=(16,10))
