1.数据挖掘简介(略)
2.使用Python和IPython Notebook
2.1.安装Python
2.2.安装IPython
2.3.安装scikit-learn
scikit-learn是用Python开发的机器学习库，它包含大量机器学习算法、数据集、工具和框架。它以Python科学计算的相关工具集为基础，其中numpy和scipy等都针对数据处理任务进行过优化，因此scikit-learn速度快、扩展性强，为此做数据挖掘很实用。
scikit-learn可以用Python3提供的pip工具进行安装，之前没有安装Numpy和Scipy的话也会顺便安装。安装命令如下：
pip install scikit-learn
3.亲和性分析示例
3.1什么是亲和性分析
亲和性分析根据样本个体(物体)之间的相似度，确定它们关系的亲疏。亲和性分析的应用场景如下。
(1)向网站用户提供多样化的服务或投放定向广告
(2)为了向用户推荐电影或商品，而卖给他们一些与之相关的小玩意。
(3)根据基因寻找有亲缘关系的人。
......
亲和性有哪些测量方法？
(1)统计两件商品一起出售的频率，或者统计顾客购买商品1后再买商品2的比率。
(2)计算两个体之间的相似度
......
3.2商品推荐
商品销售从线下搬到线上后，很多之前靠人工完成的工作只有实现自动化，才有望将生意做大，向上销售出自英文up-selling，指的是向已经购买商品的顾客推销另一种商品。原来线下由人工完成的商品推荐工作，现在依靠数据挖掘技术就能完成，而且利润大，助推电子商务革命的发展！
我们一起看一个简单的推荐服务：人们之前经常购买的两件商品，以后也很可能同时购买。
作为数据挖掘入门性质的例子，我们希望得到下面的规则：
如果一个人买了商品X，那么他很有可能购买商品Y
3.3在Numpy中加载数据集
import numpy as np
dataset_filename = 'affinity_dataset.txt'
X = np.loadtxt(dataset_filename)3.4实现简单的排序规则
规则的优劣有多种衡量方法，常用的是支持度(support)和置信度(confidence)。
支持度指数据集中应验的次数，有时候需要对支持度进行规范化。
支持度衡量给定规则应验的比例，置信度衡量规则准确率如何，即符合给定条件的所有规则里，跟当前规则结论一致的比例有多大，计算方法为首先统计当前规则的出现次数，再用它除以条件相同的规则数量
如果顾客买了苹果，他们也会购买香蕉的支持度和置信度
num_apple_purchases = 0
for sample in X:
if sample[3] ==1: #This person bought apples
num_apple_purchases += 1
# print('{0} people bought apples'.format(num_apple_purchases)) #ou can try the print way to find difference
print('{0} people bought apples'.format(num_apple_purchases))image.png
同理，检测sample[4]的值是否为1，就可以确定顾客是否也买了香蕉，进而可以计算支持度和置信度。
我们需要统计数据集中所有规则的相关数据，首先分别为规则应验和规则无效两种情况构建字典。字典的键是由条件和结论组成的元组，元组元素为特征在特征列表中的索引值，不要用实际特证名，比如“顾客如果购买了苹果，也买了香蕉”就用(3，4)表示。如果某个个体的条件和结论均与给定规则相符，则表示给定规则对该个体适用，反之无效。
为了计算所有规则的置信度和支持度，首先创建几个字典，用来存储计算结果。这里使用defaultdict，好处是如果查找的键不存在，则返回默认值。需要统计的量有规则应验、规则无效、条件相同的规则数量。
from collections import defaultdict
vaild_rules = defaultdict(int)
invaild_rules = defaultdict(int)
num_occurances = defaultdict(int)#依次对样本的每个个体及个体的每个特征值进行处理。第一个特征为规则的前提条件-----顾客购买了某种商品
for sample in X:
for premise in range(5):#检测个体是否满足条件，如果不满足则检测下一个条件
if sample[premise] ==0:continue#如果条件满足(即值为1)，该条件的出现次数加1，在遍历过程中跳过条件和结论相同的情况，比如“如果顾客购买了苹果，他们也买苹果”，这样的规则无用
num_occurances[premise] += 1
n_sample,n_features = X.shape
for conclusion in range(n_features):
if premise ==conclusion:continue#如果规则适用于个体，规则应验这种情况(vaild_rules字典中，键为由条件和结论组成的元组)增加一次，反之，违反规则情况(invaild_rules字典中)就增加一次
https://github.com/datawhalechina/joyful-pandas datawhale pandas教程
https://github.com/fengdu78/lihang-code 李航Python实现
[ch1_affinity_create]
X = np.zeros((100, 5), dtype='bool')
#dtype can change,such as int,float
X.shape[0]
#0 is row,1 is col
#数组的索引方式是和列表一样的
np.savetxt("affinity_dataset.txt", X, fmt='%d')
#parameters
fmt : str or sequence of strs, optional
A single format (%10.5f), a sequence of formats, or a
multi-format string, e.g. 'Iteration %d -- %10.5f', in which
case `delimiter` is ignored. For complex `X`, the legal options
for `fmt` are:
#create a random float from 0 to 1
a = np.random.random()
print(X[:5].astype(np.int))
[ch1_affinity]
n_samples, n_features = X.shape
print("This dataset has {0} samples and {1} features".format(n_samples, n_features))
#这种print格式用print('{0},{1}'.format(a,b))
#count the people who bought apples
num_apple_purchases = 0
for sample in X:
if sample[3] = 1:
num_apple_purchases += 1
print('{0} people bought apples'.format(num_apples_purchases))
####################################################################################################
##bought 3 but not bought 4
rule_valid = 0
rule_invalid = 0
for sample in X:
if sample[3] == 1: # This person bought Apples
if sample[4] == 1:
# This person bought both Apples and Bananas
rule_valid += 1
else:
# This person bought Apples, but not Bananas
rule_invalid += 1
print("{0} cases of the rule being valid were discovered".format(rule_valid))
print("{0} cases of the rule being invalid were discovered".format(rule_invalid))
####################################################################################################
## not bought 3
rule_valid = 0
rule_invalid = 0
for sample in X:
if sample[3] == 1:
if sample[4] == 1:
rule_valid += 1
else:
rule_invalid += 1
print('{0} rule_valid'.format(rule_valid))
print('{0} rule_invalid'.format(rule_invalid))####################################################################################################
规则是 如果买了苹果，可能也买了香蕉。
规则无效是 如果买了苹果，但没买香蕉
print("The support is {0} and the confidence is {1:.3f}.".format(support, confidence))
# Confidence can be thought of as a percentage using the following:
print("As a percentage, that is {0:.1f}%.".format(100 * confidence))
####################################################################################################
from collections import defaultdict
# Now compute for all possible rules
valid_rules = defaultdict(int)
invalid_rules = defaultdict(int)
num_occurences = defaultdict(int)
for sample in X:
for premise in range(n_features):
if sample[premise] == 0: continue
# Record that the premise was bought in another transaction
num_occurences[premise] += 1
for conclusion in range(n_features):
if premise == conclusion: # It makes little sense to measure if X -> X.
continue
if sample[conclusion] == 1:
# This person also bought the conclusion item
valid_rules[(premise, conclusion)] += 1
else:
# This person bought the premise, but not the conclusion
invalid_rules[(premise, conclusion)] += 1
support = valid_rules
confidence = defaultdict(float)
for premise, conclusion in valid_rules.keys():
confidence[(premise, conclusion)] = valid_rules[(premise, conclusion)] / num_occurences[premise]
####################################################################################################
from collections import defaultdict
rule_valid = defaultdict(int)
rule_invalid = defaultdict(int)
num_premise = defaultdict(int)
n_features = X.shape[1]
for sample in X:
for premise in range(n_features):
if sample[premise] == 0:continue
if sample[premise] == 1:
num_premise[premise] += 1
for conclusion in range(n_features):
if premise == conclusion:continue
if sample[conclusion] == 1:
rule_valid[(premise,conclusion)] += 1
else:
rule_invalid[(premise,conclusion)] += 1
support = rule_valid
confidence = defaultdict(float)
for premise,conclusion in rule_valid.keys():
confidence[(premise,conclusion)] = rule_valid[(premise,conclusion)] / num_premise[premise]
####################################################################################################
for premise, conclusion in confidence:
premise_name = features[premise]
conclusion_name = features[conclusion]
print("Rule: If a person buys {0} they will also buy {1}".format(premise_name, conclusion_name))
print(" - Confidence: {0:.3f}".format(confidence[(premise, conclusion)]))
print(" - Support: {0}".format(support[(premise, conclusion)]))
print("")
####################################################################################################
for premise,conclusion in confidence:
features = ["bread", "milk", "cheese", "apples", "bananas"]
premise_name = features[premise]
conclusion_name = features[conclusion]
print('If someone buy {0} then they may buy {1}'.format(premise_name,conclusion_name))
print('confidence is {0:.3f}'.format(confidence[(premise,conclusion)]))
print('support is {0}'.format(support[(premise,conclusion)]))
#用于打印特定的数据结构，整齐好看
from pprint import pprint
pprint(list(support.items()))
#example
import pprint
data = ("test", [1, 2, 3,'test', 4, 5], "This is a string!",
{'age':23, 'gender':'F'})
print(data)
pprint.pprint(data)image.png
注意
# [Python: 字典列表: itemgetter 函数: 根据某个或某几个字典字段来排序列表]()
from operator import itemgetter
sorted_support = sorted(support.items(), key=itemgetter(1), reverse=True)
# [python中sorted和sort 、reversed和reverse的使用]()
[OneR]
[ch1_oner_application]
#iris.describe
print(dataset.DESCR)
在进行OneR算法分类前需要将数据进行离散化
# Compute the mean for each attribute
attribute_means = X.mean(axis=0)
assert attribute_means.shape == (n_features,)#assert:断言
X_d = np.array(X >= attribute_means, dtype='int')
#X.means(axis):axis = 0 is symbol take col
#assert 1==1 # 条件为 true 正常执行
#assert 1==2 # 条件为 false 触发异常
#sklearn中已经废弃cross_validation,将其中的内容整合到#model_selection中
#将sklearn.cross_validation 替换为 sklearn.model_selection
##origin
# Now, we split into a training and test set
from sklearn.cross_validation import train_test_split
# Set the random state to the same number to get the same results as in the book
random_state = 14
X_train, X_test, y_train, y_test = train_test_split(X_d, y, random_state=random_state)
print("There are {} training samples".format(y_train.shape))
print("There are {} testing samples".format(y_test.shape))
##new
# Now, we split into a training and test set
from sklearn.model_selection import train_test_split
# Set the random state to the same number to get the same results as in the book
random_state = 14
X_train, X_test, y_train, y_test = train_test_split(X_d, y, random_state=random_state)
print("There are {} training samples".format(y_train.shape))
print("There are {} testing samples".format(y_test.shape))
#train_X,test_X,train_y,test_y = train_test_split(train_data,train_target,test_size=0.3,random_state=5)#train_test_split()函数是用来随机划分样本数据为训练集和测试集的，当然也可以人为的切片划分
#优点：随机客观的划分数据，减少人为因素
#test_size：测试数据占样本数据的比例，若整数则样本数量
#zip() 函数用于将可迭代的对象作为参数，将对象中对应的元素打包成一个个元组，然后返回由这些元组组成的列表。
#如果各个迭代器的元素个数不一致，则返回列表长度与最短的对象相同，利用 * 号操作符，可以将元组解压为列表
>>>a = [1,2,3]
>>>b = [4,5,6]
>>>c = [4,5,6,7,8]
>>>zipped = zip(a,b) # 打包为元组的列表
[(1, 4), (2, 5), (3, 6)]
>>>zip(a,c) # 元素个数与最短的列表一致
[(1, 4), (2, 5), (3, 6)]
>>>zip(*zipped) # 与 zip 相反，*zipped 可理解为解压，返回二维矩阵式
[(1, 2, 3), (4, 5, 6)]
class_counts = defaultdict(int)
#Iterate through each sample and count the frequency of each class/value pair
for sample, y in zip(X, y_true):
if sample[feature] == value:
class_counts[y] += 1
a = zip(X, y)
for b in a:
print(b)
image.png
a = zip(X, y)
for b,c in a:
print(b)
image.png
a = zip(X, y)
for b,c in a:
print(c)
image.png
error = sum([class_count for class_value, class_count in class_counts.items()
if class_value != most_frequent_class])
