Python 加载顺序 python数据_Python 加载顺序


实验环境：Anaconda-> Jupyter

参考数据：利用python进行数据分析

python版本 3.5.2 

path='C:\\Users\\ecaoyng\\Desktop\\work space\\Python\\py_for_analysis\\pydata-book-master\\ch02\\usagov_bitly_data2012-03-16-1331923249.txt'
open(path).readline()

也可以用JSON列表,loads函数，注意地址分隔符要转义

import json
path='C:\\Users\\ecaoyng\\Desktop\\work space\\Python\\py_for_analysis\\pydata-book-master\\ch02\\usagov_bitly_data2012-03-16-1331923249.txt'
records=[json.loads(line) for line in open(path)]
print (records[0])
print (type(records))
{‘gr’: ‘MA’, ‘u’: ‘http://www.ncbi.nlm.nih.gov/pubmed/22415991‘, ‘cy’: ‘Danvers’, ‘h’: ‘wfLQtf’, ‘hh’: ‘1.usa.gov’, ‘r’: ‘http://www.facebook.com/l/7AQEFzjSi/1.usa.gov/wfLQtf‘, ‘a’: ‘Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.78 Safari/535.11’, ‘al’: ‘en-US,en;q=0.8’, ‘ll’: [42.576698, -70.954903], ‘hc’: 1331822918, ‘nk’: 1, ‘tz’: ‘America/New_York’, ‘c’: ‘US’, ‘l’: ‘orofrog’, ‘g’: ‘A6qOVH’, ‘t’: 1331923247}

records[0]['tz']

‘America/New_York’

列表推导式：

timezones=[rec['tz'] for rec in records if 'tz' in rec]
timezones[:10]
[‘America/New_York’, 
 ‘America/Denver’, 
 ‘America/New_York’, 
 ‘America/Sao_Paulo’, 
 ‘America/New_York’, 
 ‘America/New_York’, 
 ‘Europe/Warsaw’, 
 ”, 
 ”, 
 ”]
def get_counts(seq):
    counts={}
    for x in seq:
        if x in counts:
            counts[x] +=1
        else:
            counts[x]=1
    return counts

get_counts(timezones)

用python代码对时区进行计数：
传统python:

def get_counts(seq):
    counts={}
    for x in seq:
        if x in counts:
            counts[x] +=1
        else:
            counts[x]=1
    return counts

get_counts(timezones)

也可以用defaultdict，将dic的初始值设置为0

from collections import defaultdict
def get_counts2(seq):
    counts=defaultdict(int) # Init with 0
    for x in seq:
        counts[x]+=1
    return counts

tztimes=get_counts2(timezones)
len(timezones)

3440

tztimes.items()
ict_items([(”, 521), (‘Europe/Uzhgorod’, 1), (‘Asia/Pontianak’, 1), (‘Europe/Brussels’, 4), (‘America/Mazatlan’, 1), (‘America/Monterrey’, 1), (‘Asia/Dubai’, 4), (‘Europe/Bratislava’, 3), (‘Asia/Jerusalem’, 3), (‘Europe/Rome’, 27), (‘Asia/Kuala_Lumpur’, 3), (‘America/Indianapolis’, 20), (‘Europe/Lisbon’, 8), (‘Europe/Volgograd’, 1), (‘America/Montevideo’, 1), (‘Europe/Skopje’, 1), …

如果想要得到前十位的时区，用字典的处理技巧：

def top_counts(count_dict,n=10):
    value_key_pairs=[(count,tz)for tz,count in count_dict.items()]
    value_key_pairs.sort()
    return value_key_pairs[-n:]

top_counts(tztimes)
[(‘America/New_York’, 1251), 
 (”, 521), 
 (‘America/Chicago’, 400), 
 (‘America/Los_Angeles’, 382), 
 (‘America/Denver’, 191), 
 (‘Europe/London’, 74), 
 (‘Asia/Tokyo’, 37), 
 (‘Pacific/Honolulu’, 36), 
 (‘Europe/Madrid’, 35), 
 (‘America/Sao_Paulo’, 33)]

也可以用标准库中的collections.Counter

from collections import Counter
counts=Counter(timezones)
counts.most_common(10)
[(‘America/New_York’, 1251), 
 (”, 521), 
 (‘America/Chicago’, 400), 
 (‘America/Los_Angeles’, 382), 
 (‘America/Denver’, 191), 
 (‘Europe/London’, 74), 
 (‘Asia/Tokyo’, 37), 
 (‘Pacific/Honolulu’, 36), 
 (‘Europe/Madrid’, 35), 
 (‘America/Sao_Paulo’, 34)]

用pandas对时区进行计数
DataFrame用于将数据表示成一个表格

from pandas import DataFrame, Series
import pandas as pd; import numpy as np
frame=DataFrame(records)
frame

若不按顺序查看

frame['tz'][:10]
0 America/New_York 
 1 America/Denver 
 2 America/New_York 
 3 America/Sao_Paulo

这里frame的输出是摘要视图，主要用于较大的DataFrame对象。frame[‘tz’]所返回的Series对象有一个value_counts方法可以得到需要的值

frame['tz'].value_counts()

若此时想用matplotlib生成一张图片，需要先将未知或者缺失的时区填上一个替代值。fillna可以替代缺失值，空字符串则可以通过布尔型数组索引加以替换。

clean_tz=frame['tz'].fillna('Missing')
clean_tz[clean_tz== ''] = 'Unknow'
tz_counts=clean_tz.value_counts()
tz_counts[:10]
America/New_York 1251 
 Unknow 521 
 America/Chicago 400 
 America/Los_Angeles 382 
 America/Denver 191 
 Missing 120 
 Europe/London 74 
 Asia/Tokyo 37 
 Pacific/Honolulu 36 
 Europe/Madrid 35 
 Name: tz, dtype: int64

利用plot方法可以得到水平条形图
当然你要打开pylab
打开方法：

%pylab
Using matplotlib backend: Qt5Agg 
 Populating the interactive namespace from numpy and matplotlib
tz_counts[:10].plot(kind='barh',rot=0)

Python 加载顺序 python数据_Python 加载顺序_02

比如取frame的一列

frame['a']
type(frame['a'])

得到： 

0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi… 
 1 GoogleMaps/RochesterNY 
 2 Mozilla/4.0 (compatible; MSIE 8.0; Windows NT … 
 3 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)… 
 … 
 pandas.core.series.Series

比如要将agent字符串的第一节分离出来并得到另外一份用户行为摘要 

results=Series([x.split()[0] for x in frame.a.dropna()])
results[:5]
0 Mozilla/5.0 
 1 GoogleMaps/RochesterNY 
 2 Mozilla/4.0 
 3 Mozilla/5.0 
 4 Mozilla/5.0 
 dtype: object
results.value_counts()[:8]
Mozilla/5.0 2594 
 Mozilla/4.0 601 
 GoogleMaps/RochesterNY 121 
 Opera/9.80 34 
 TEST_INTERNET_AGENT 24 
 GoogleProducer 21 
 Mozilla/6.0 5 
 BlackBerry8520/5.0.0.681 4 
 dtype: int64

假如你想按windows和非windows用户来区分时区的统计信息。为了简单，假如agent字符串中含有windows就认为是windows用户，由于有些agent缺失，所以先移除

cframe=frame[frame.a.notnull()]
operating_system=np.where(cframe['a'].str.contains('Windows'), 'Windows', 'Not Windows')
operating_system[:5]

array([‘Windows’, ‘Not Windows’, ‘Windows’, ‘Not Windows’, ‘Windows’],
dtype=’

print (len(cframe))
by_tz_os=cframe.groupby(['tz'])
by_tz_os.size()

输出 

3440 
 tz 
 521 
 Africa/Cairo 3 
 Africa/Casablanca 1 
 Africa/Ceuta 2 
 Africa/Johannesburg 1 
 Africa/Lusaka 1 
 America/Anchorage 5 
 America/Argentina/Buenos_Aires 1 
 … 
 dtype: int64

为了计算是否和cframe的值相等

by_tz_os.size().sum()

3440

unstack 可以使一列数据设置为列标签

agg_counts=by_tz_os.size().unstack()
agg_counts[:10]

agg_counts=agg_counts.fillna(0)
agg_counts

Python 加载顺序 python数据_python_03

agg_counts['Not Windows'].sum(0)
1194.0
agg_counts.loc['Pacific/Honolulu']
Not Windows 0.0 
 Windows 36.0 
 Name: Pacific/Honolulu, dtype: float64
indexer=agg_counts.sum(1).loc['Pacific/Honolulu']
indexer

36.0

count_subset=agg_counts.take(indexer)[-10:]
count_subset

最后来选取最常出现的时区

Python 加载顺序 python数据_json_04

take的作用官方文档是Analogous to ndarray.take（百度）

作图：

count_subset.plot(kind='barh',stacked=True)

Python 加载顺序 python数据_c/c++_05

from pandas import DataFrame #从pandas库中引用DataFrame 
 df_obj = DataFrame() #创建DataFrame对象 
 df_obj.dtypes #查看各行的数据格式 
 df_obj.head() #查看前几行的数据,默认前5行 
 df_obj.tail() #查看后几行的数据,默认后5行 
 df_obj.index #查看索引 
 df_obj.columns #查看列名 
 df_obj.values #查看数据值 
 df_obj.describe #描述性统计 
 df_obj.T #转置 
 df_obj.sort(columns = ‘’)#按列名进行排序 
 df_obj.sort_index(by=[‘’,’’])#多列排序,使用时报该函数已过时,请用sort_values 
 df_obj.sort_values(by=[”,”])同上

一个numpy的例子

import numpy as np
data = {i: np.random.randn() for i in range(7)}
print (data)
print (dtype(data))
{0: -0.3545697629475486, 1: 1.3098551782757126, 2: -0.17715637837979628, 3: 0.7631836116626937, 4: 0.1599041446132432, 5: 0.7041281969761696, 6: 0.5984084616089319}

def printMax(x, y):  
    '''Prints the maximum of two numbers.\
    The two values must be integers.'''  
    x = int(x) # convert to integers, if possible  
    y = int(y)  

    if x > y:  
        print (x, 'is maximum')  
    else:  
        print (y, 'is maximum')  

printMax(3, 5)  
print (printMax.__doc__)
print ('---------------')
print (help(printMax))
5 is maximum 
 Prints the maximum of two numbers. The two values must be integers. 
 /——————- 
 printMax(x, y) 
 Prints the maximum of two numbers. The two values must be integers.None