python数据分析期末复习归纳(更新中)


文章目录

  • python数据分析期末复习归纳(更新中)
  • 前言
  • 一、python语言基础
  • 二、内建数据结构、函数、文件(重点)
  • 元组
  • 列表
  • 内建序列函数
  • 字典
  • 函数
  • 三、Numpy基础(重点)
  • 四、pandas入门(重点)
  • Series
  • DataFrame
  • 五、数据载入
  • 六、数据清洗与准备
  • 七、数据规整:连接、联合与重塑
  • 八、绘图与可视化
  • GOOD LUCK !



前言

可以通过《利用Python进行数据分析》的GitHub仓库获得本书的数据文件和相关材料。

链接:GitHub仓库地址

提示:以下是本篇文章正文内容

一、python语言基础

分值:1~2分

  1. python使用缩进来组织代码,而不是其他语言比如R、C++、java和Perl那样用大括号。
  2. 你见到的python语句都不是以分号结尾,而分号也是可以用于在一行内将多条语句进行分隔:
a = 5;b = 6;c = 7
  1. python语言的一个重要特征就是对象模型的一致性。每一个数值、字符串、数据结构、函数、类、模块以及所有存在于python解释器中的事物都是python的对象。每一个对象都会关联到一种数据类型和内部数据。
  2. 检查两个引用是否指向同一个对象可以用 is 关键字。is not 在你检查两个关键字是不是相同对象时也是有效的。
In []: a = [1, 2, 3]
In []: b = a
In []: c = list(a)

In []: a is b
Out[]: True

In []: a is not c
Out[]: True
'''因为list函数总是创建一个新的Python列表(即一份拷贝),我们可以确定c与a是不同的。
is和==是不同的,因为在这种情况下我们可以得到:'''
In []: a == c
Out[]: True
#is 和 is not的常用之处是检查一个变量是否为None,因为None只有一个实例:
In []: a = None

In []: a is None
Out[]: True
  1. 可变对象与不可变对象。Python中的大部分对象,例如列表、字典、Numpy数组都是可变对象,大多数用户定义的类型(类)也是可变的。可变对象中包含的对象和值是可以被修改的。还有其他的对象是不可变的,比如字符串、元组。
  2. 数值类型。基础的python数字类型就是int 和 float。int 可以存储任意大小的数字。float表示浮点数,每一个浮点数都是双精度64位数值。
  3. 字符串。字符串是Unicode字符的序列,因此可以被看作是除了列表和元组外的一种序列。
In []: s = 'python'

In []: list(s)
Out[]: ['p','y','t','h','o','n']

In []: s[:3]
Out[]: 'pyt'

二、内建数据结构、函数、文件(重点)

元组

定义元组

In []: tup = 4,5,6

In []: tup
Out[]: (4, 5, 6)

使用tuple函数将任意序列或迭代器转换为元组:

In []: tuple([4, 0, 2])
Out[]: (4, 0, 2)

In []: tup = tuple('string')

In []: tup
Out[]: ('s', 't', 'r', 'i', 'n', 'g')

In []: (4,None,'foo')+(6,0)+('bar',)
Out[]: (4, None, 'foo', 6, 0, 'bar')

In []: ('foo','bar')*4
Out[]: ('foo', 'bar', 'foo', 'bar', 'foo', 'bar', 'foo', 'bar')

列表

创建列表

In []: alist = []

In []: list(range(10))
Out[]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

In []: tup
Out[]: ('foo', 'bar', 'baz')

In []: b_list(tup)
In []: b_list
Out[]: ['foo', 'bar', 'baz']

In []: b_list.append('dasf')

In []: b_list
Out[]: ['foo', 'bar', 'baz', 'dasf']

insert()、pop()、remove()方法

In []: b_list.insert(2,'hug')

In []: b_list
Out[]: ['foo', 'bar', 'hug', 'baz', 'dasf']

In []: b_list.pop(3)
Out[]: 'baz'

In []: b_list
Out[]: ['foo', 'bar', 'hug', 'dasf']

In []: b_list.remove('dasf')

In []: b_list
Out[]: ['foo', 'bar', 'hug']

连接两个列表

In []: ['hj',6,(2,3)]+['re',7]
Out[]: ['hj', 6, (2, 3), 're', 7]

#extend()方法消耗更小
In []: x=['hj',6,(2,3)]
In []: x.extend(['re',7])

In []: x
Out[]: ['hj', 6, (2, 3), 're', 7]

排序

In []: a=[4,6,1,9,2,8]

In []: a.sort()

In []: a
Out[]: [1, 2, 4, 6, 8, 9]

In []: a = [4, 6, 1, 9, 2, 8]
In []: sorted(a)
Out[]: [1, 2, 4, 6, 8, 9]

In []: a
Out[]: [4, 6, 1, 9, 2, 8]

In []: b=['df','rewg','fsdvsfdvsdv','d','gfg']
In []: b.sort(key=len)

In []: b
Out[]: ['d', 'df', 'gfg', 'rewg', 'fsdvsfdvsdv']

切片

In []: seq
Out[]: [7, 2, 3, 6, 3, 5, 6, 0, 1]

In []: seq[4:3]
Out[]: []

In []: seq[3:4]
Out[]: [6]

In [35]: seq[3:4]=[6,3]

In []: seq
Out[]: [7, 2, 3, 6, 3, 3, 5, 6, 0, 1]

In []: seq[-6:-2]
Out[]: [3, 3, 5, 6]

内建序列函数

zip可以将列表、元组或者其他序列的元素配对,新建一个元组构成的列表

In []: seq1 = ['foo', 'bar', 'baz']

In []: seq2 = ['one', 'two', 'three']

In []: zipped = zip(seq1, seq2)

In []: list(zipped)
Out[]: [('foo', 'one'), ('bar', 'two'), ('baz', 'three')]

reversed()将序列的元素倒序排列

In []: list(reversed(range(10)))
Out[]: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

字典

创建字典

In []: d1={'a':'some value','b':[1,2,3,4]}

In []: d1
Out[]: {'a': 'some value', 'b': [1, 2, 3, 4]}
#向字典中添加元素
In []: d1[7]=0

In []: d1
Out[]: {'a': 'some value', 'b': [1, 2, 3, 4], 7: 0}

In []: d1[4] = 'banace'

In []: d1
Out[]: {'a': 'some value', 'b': [1, 2, 3, 4], 7: 0, 4: 'banace'}
In []: 'b' in d1
Out[]: True
In []: list(d1.keys())
Out[]: ['a', 'b', 7, 4]

In []: list(d1.values())
Out[]: ['some value', [1, 2, 3, 4], 0, 'banace']

使用update()方法将两个字典合并

In [55]: d1.update({'r':'goo','h':'integer'})

In [56]: d1
Out[56]:
{'a': 'some value',
 'b': [1, 2, 3, 4],
 7: 0,
 4: 'banace',
 'r': 'goo',
 'h': 'integer'}

从序列生成字典

In []: mapping={}

In []: mapping = dict(zip(range(5),reversed(range(5))))

In []: mapping
Out[]: {0: 4, 1: 3, 2: 2, 3: 1, 4: 0}

有效的字典类型
尽管字典的值可以是任何Python对象,但键必须是不可变的对象,比如标量类型(整数、浮点数、字符串)或元组(且元组内对象也必须是不可变对象)

In []: x={[3,4]:54}	#key值为一个可变对象列表,报错 unhashable
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-61-abe1ff6f266b> in <module>
----> 1 x={[3,4]:54}

TypeError: unhashable type: 'list'

函数

三、Numpy基础(重点)

生成ndarray

In []: data2=[[4,5,2],[3,4,5]]
In []: arr1=np.array(data2)

In []: arr1
Out[]: 
array([[4, 5, 2],
       [3, 4, 5]])

In []: arr1.ndim
Out[]: 2

In []: arr1.shape
Out[]: (2, 3)

In []: arr1.dtype
Out[]: dtype('int32')

In []: np.zeros(10)
Out[]: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

In []: np.zeros((3,7))
Out[]: 
array([[0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.]])

In []: arr2=np.array([1,2,3],dtype=np.int64)

In []: arr2
Out[]: array([1, 2, 3], dtype=int64)

In []: arr2.dtype
Out[]: dtype('int64')

Numpy数组算数

四、pandas入门(重点)

pandas采用了很多NumPy的代码风格,但最大的不同在于pandas是用来处理表格型或异质型数据的。而NumPy则相反,它更适合处理同质型的数值类数组数据。

Series

In []: obj = pd.Series([4, 7, -5, 3])
In []: obj
Out[]:
0    4
1    7
2   -5
3    3
dtype: int64

In []: obj.values
Out[]: array([ 4,  7, -5,  3], dtype=int64)

In []: obj.index #与range(4)类似
Out[]: RangeIndex(start=0, stop=4, step=1)

In []: obj2 = pd.Series([4, 7, -5, 3],index=['d','b','a','c'])

In []: obj2
Out[]:
d    4
b    7
a   -5
c    3
dtype: int64

In []: obj2.index
Out[]: Index(['d', 'b', 'a', 'c'], dtype='object')

如果已经有数据包含在Python字典中,你可以使用字典生成一个Series:

In []: sdata = {'Ohio': 35000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}
In []: obj3=Series(sdata)

In []: obj3
Out[]:
Ohio      35000
Texas     71000
Oregon    16000
Utah       5000
dtype: int64

DataFrame

#嵌套字典
In []: pop
Out[]: {'Nevada': {2001: 2.4, 2002: 2.9}, 'Ohio': {2000: 1.5, 2001: 1.7, 2002: 3.6}}

In []: frame3=DataFrame(pop)

In []: frame3
Out[]:
      Nevada  Ohio
2001     2.4   1.7
2002     2.9   3.6
2000     NaN   1.5

转置操作

In []: frame3.T
Out[]:
        2001  2002  2000
Nevada   2.4   2.9   NaN
Ohio     1.7   3.6   1.5

生成时指定索引

In []: pd.DataFrame(pop,index=[2000,2001,2002])
Out[]:
      Nevada  Ohio
2000     NaN   1.5
2001     2.4   1.7
2002     2.9   3.6

DataFrame的values属性

In []: frame3.values
Out[]:
array([[2.4, 1.7],
       [2.9, 3.6],
       [nan, 1.5]])

fill方法会把值前向填充

In []: obj3 = Series(['blue', 'purple', 'yellow'], index=[0, 2, 4])

In []: obj3.reindex(range(6), method='ffill')
Out[]:
0      blue
1      blue
2    purple
3    purple
4    yellow
5    yellow
dtype: object

reindex()

In []: frame = DataFrame(np.arange(9).reshape((3, 3)), index=['a', 'c', 'd'],
     ...:                   columns=['Ohio', 'Texas', 'California'])

In []: frame
Out[]:
   Ohio  Texas  California
a     0      1           2
c     3      4           5
d     6      7           8

In []: frame2 = frame.reindex(['a', 'b', 'c', 'd'])

In []: frame2
Out[]:
   Ohio  Texas  California
a   0.0    1.0         2.0
b   NaN    NaN         NaN
c   3.0    4.0         5.0
d   6.0    7.0         8.0

In []: states = ['Texas', 'Utah', 'California']

In []: frame.reindex(columns=states)#使用column关键字重建索引
Out[]:
   Texas  Utah  California
a      1   NaN           2
c      4   NaN           5
d      7   NaN           8

drop()

#Series中使用
In []: obj=pd.Series(np.arange(5.),index=['a','b','c','d','e'])

In []: obj
Out[]:
a    0.0
b    1.0
c    2.0
d    3.0
e    4.0
dtype: float64

In []: new_obj=obj.drop(['a','c'])

In []: new_obj
Out[]:
b    1.0
d    3.0
e    4.0
dtype: float64

#DataFrame中使用
In []: data = DataFrame(np.arange(16).reshape((4, 4)),^M
     ...:                  index=['Ohio', 'Colorado', 'Utah', 'New York'],^M
     ...:                  columns=['one', 'two', 'three', 'four'])

In []: data
Out[]:
          one  two  three  four
Ohio        0    1      2     3
Colorado    4    5      6     7
Utah        8    9     10    11
New York   12   13     14    15

In []: data.drop(['Colorado', 'Ohio'])
Out[]:
          one  two  three  four
Utah        8    9     10    11
New York   12   13     14    15

In []: data.drop(['two','four'],axis=1)
Out[]:
          one  three
Ohio        0      2
Colorado    4      6
Utah        8     10
New York   12     14

普通的python切片是不包含尾部的,Series的切片与之不同

In []: obj[2:4]	#普通切片
Out[]:
c    2.0
d    3.0
dtype: float64

In []: obj['b':'d']	#按索引切片
Out[]:
b    1.0
c    2.0
d    3.0
dtype: float64

In []: data
Out[]:
          one  two  three  four
Ohio        0    1      2     3
Colorado    4    5      6     7
Utah        8    9     10    11
New York   12   13     14    15

In []: data['two']
Out[]:
Ohio         1
Colorado     5
Utah         9
New York    13
Name: two, dtype: int32

In []: data[['three','one']]
Out[]:
          three  one
Ohio          2    0
Colorado      6    4
Utah         10    8
New York     14   12

In []: data[:2]
Out[]:
          one  two  three  four
Ohio        0    1      2     3
Colorado    4    5      6     7

使用轴标签(loc)和整数标签(iloc)选择数据

In []: data.loc['Colorado', ['two', 'three']]
Out[]:
two      5
three    6
Name: Colorado, dtype: int32

In []: data.iloc[1,[1,2]]
Out[]:
two      5
three    6
Name: Colorado, dtype: int32

In []: data.iloc[[1,2], [3, 0, 1]]
Out[]:
          four  one  two
Colorado     7    4    5
Utah        11    8    9

In []: ser[:2]
Out[]:
0    0.0
1    1.0
dtype: float64
In []: ser.iloc[:2]
Out[]:
0    0.0
1    1.0
dtype: float64

In []: ser.loc[:2]
Out[]:
0    0.0
1    1.0
2    2.0
dtype: float64

了解广播机制

#NumPy
In []: arr=np.arange(12.).reshape((3,4))

In []: arr
Out[]:
array([[ 0.,  1.,  2.,  3.],
       [ 4.,  5.,  6.,  7.],
       [ 8.,  9., 10., 11.]])

In []: arr[0]
Out[]: array([0., 1., 2., 3.])

In []: arr-arr[0]
Out[]:
array([[0., 0., 0., 0.],
       [4., 4., 4., 4.],
       [8., 8., 8., 8.]])
#DataFrame与Series之间也是类似的
In []: frame = DataFrame(np.arange(12.).reshape((4, 3)), columns=list('bde'),
     ...:                   index=['Utah', 'Ohio', 'Texas', 'Oregon'])

In []: series=frame.iloc[0]

In []: frame
Out[]:
          b     d     e
Utah    0.0   1.0   2.0
Ohio    3.0   4.0   5.0
Texas   6.0   7.0   8.0
Oregon  9.0  10.0  11.0

In []: series
Out[]:
b    0.0
d    1.0
e    2.0
Name: Utah, dtype: float64

In []: frame - series
Out[]:
          b    d    e
Utah    0.0  0.0  0.0
Ohio    3.0  3.0  3.0
Texas   6.0  6.0  6.0
Oregon  9.0  9.0  9.0

五、数据载入

考察点:数据读入与写入

#数据读入
In []: !cat examples/ex1.csv
In []: df = pd.read_csv('examples/ex1.csv')
In []: pd.read_table('examples/ex1.csv', sep=',')	#指定分隔符

In []: !cat examples/csv_mindex.csv
In []: parsed = pd.read_csv('examples/csv_mindex.csv',
                     index_col=['key1', 'key2'])
#数据写入
In []: data = pd.read_csv('examples/ex5.csv')
In []: data
In []: data.to_csv('examples/out.csv')
In []: !cat examples/out.csv

六、数据清洗与准备

分值:6分

处理缺失值

In [56]: string_data = pd.Series(['aardvark', 'artichoke', np.nan, 'avocado'])

In []: string_data
Out[]:
0     aardvark
1    artichoke
2          NaN
3      avocado
dtype: object

In []: string_data.isnull()
Out[]:
0    False
1    False
2     True
3    False
dtype: bool

In []: from numpy import nan as NA

In []: data = pd.Series([1, NA, 3.5, NA, 7])

In []: data.dropna()
Out[]:
0    1.0
2    3.5
4    7.0
dtype: float64

In []: data = pd.DataFrame([[1., 6.5, 3.], [1., NA, NA],
    ...:                      [NA, NA, NA], [NA, 6.5, 3.]])

In []: cleaned = data.dropna()

In []: data
Out[]:
     0    1    2
0  1.0  6.5  3.0
1  1.0  NaN  NaN
2  NaN  NaN  NaN
3  NaN  6.5  3.0

In []: cleaned
Out[]:
     0    1    2
0  1.0  6.5  3.0

#传入how='all'时,将删除所有值均为NA的行
In []: data.dropna(how='all')
Out[]:
     0    1    2
0  1.0  6.5  3.0
1  1.0  NaN  NaN
3  NaN  6.5  3.0

In []: data[4]=NA

In []: data
Out[]:
     0    1    2   4
0  1.0  6.5  3.0 NaN
1  1.0  NaN  NaN NaN
2  NaN  NaN  NaN NaN
3  NaN  6.5  3.0 NaN

In []: data.dropna(axis=1,how='all')
Out[]:
     0    1    2
0  1.0  6.5  3.0
1  1.0  NaN  NaN
2  NaN  NaN  NaN
3  NaN  6.5  3.0

使用fillna()方法补全缺失值

In []: df
Out[]:
          0         1         2
0  0.291028       NaN       NaN
1 -0.247507       NaN       NaN
2 -0.846066       NaN  1.815868
3 -2.644441       NaN -1.593109
4  0.687419 -0.576368 -1.207267
5  1.028088 -0.199093  1.090297
6 -1.356587  2.840271  0.588919

In []: df.fillna(0)
Out[]:
          0         1         2
0  0.291028  0.000000  0.000000
1 -0.247507  0.000000  0.000000
2 -0.846066  0.000000  1.815868
3 -2.644441  0.000000 -1.593109
4  0.687419 -0.576368 -1.207267
5  1.028088 -0.199093  1.090297
6 -1.356587  2.840271  0.588919

#可以使用字典,为不同列设定不同的填充值
In []: df.fillna({1: 0.5, 2: 0})
Out[]:
          0         1         2
0  0.291028  0.500000  0.000000
1 -0.247507  0.500000  0.000000
2 -0.846066  0.500000  1.815868
3 -2.644441  0.500000 -1.593109
4  0.687419 -0.576368 -1.207267
5  1.028088 -0.199093  1.090297
6 -1.356587  2.840271  0.588919

In []: df
Out[]:
          0         1         2
0 -0.749045  0.120431 -0.524772
1 -1.170878  0.449045 -0.009419
2 -1.522980       NaN -0.932252
3  0.245718       NaN -0.584712
4 -0.611673       NaN       NaN
5  0.205112       NaN       NaN

In []: df.fillna(method='ffill')
Out[]:
          0         1         2
0 -0.749045  0.120431 -0.524772
1 -1.170878  0.449045 -0.009419
2 -1.522980  0.449045 -0.932252
3  0.245718  0.449045 -0.584712
4 -0.611673  0.449045 -0.584712
5  0.205112  0.449045 -0.584712

In []: df.fillna(method='ffill', limit=1) #limit参数表示处理范围
Out[]:
          0         1         2
0 -0.749045  0.120431 -0.524772
1 -1.170878  0.449045 -0.009419
2 -1.522980  0.449045 -0.932252
3  0.245718       NaN -0.584712
4 -0.611673       NaN -0.584712
5  0.205112       NaN       NaN

#替代值
In []: data
Out[]:
0       1.0
1    -999.0
2       2.0
3    -999.0
4   -1000.0
5       3.0
dtype: float64

In []: data.replace(-999,NA)
Out[]:
0       1.0
1       NaN
2       2.0
3       NaN
4   -1000.0
5       3.0
dtype: float64

In []: data.replace([-999,-1000],[NA,0])
Out[]:
0    1.0
1    NaN
2    2.0
3    NaN
4    0.0
5    3.0
dtype: float64
#参数也可通过字典传递
In []: data.replace({-999:NA,-1000:0})
Out[]:
0    1.0
1    NaN
2    2.0
3    NaN
4    0.0
5    3.0
dtype: float64

七、数据规整:连接、联合与重塑

分值:6分

分层索引

In []: data = pd.Series(np.random.randn(9),^M
    ...:                  index=[['a', 'a', 'a', 'b', 'b', 'c', 'c', 'd', 'd'],^M
    ...:                         [1, 2, 3, 1, 3, 1, 2, 2, 3]])

In []: data
Out[]:
a  1   -1.218316
   2   -0.331598
   3    1.511461
b  1    0.443087
   3    0.080628
c  1    0.088635
   2   -2.549623
d  2   -0.793741
   3   -0.266901
dtype: float64

#使用DataFrame的列进行索引
In [109]: frame = pd.DataFrame(np.arange(12).reshape((4, 3)),^M
     ...:                      index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
     ...:                      columns=[['Ohio', 'Ohio', 'Colorado'],
     ...:                               ['Green', 'Red', 'Green']])

In []: frame
Out[]:
     Ohio     Colorado
    Green Red    Green
a 1     0   1        2
  2     3   4        5
b 1     6   7        8
  2     9  10       11

In []: frame.columns.names=['state','color']

In []: frame
Out[]:
state      Ohio     Colorado
color     Green Red    Green
key1 key2
a    1        0   1        2
     2        3   4        5
b    1        6   7        8
     2        9  10       11

In []: frame['Ohio']
Out[]:
color      Green  Red
key1 key2
a    1         0    1
     2         3    4
b    1         6    7
     2         9   10

重排序与层级排序

In []: frame.swaplevel('key1', 'key2')
Out[]:
state      Ohio     Colorado
color     Green Red    Green
key2 key1
1    a        0   1        2
2    a        3   4        5
1    b        6   7        8
2    b        9  10       11

In []: frame.sort_index(level=1)
Out[]:
state      Ohio     Colorado
color     Green Red    Green
key1 key2
a    1        0   1        2
b    1        6   7        8
a    2        3   4        5
b    2        9  10       11

In []: frame.swaplevel(0, 1).sort_index(level=0)
Out[]:
state      Ohio     Colorado
color     Green Red    Green
key2 key1
1    a        0   1        2
     b        6   7        8
2    a        3   4        5
     b        9  10       11

使用DataFrame的列进行索引

In []: frame = pd.DataFrame({'a': range(7), 'b': range(7, 0, -1),
     ...:                       'c': ['one', 'one', 'one', 'two', 'two',
     ...:                             'two', 'two'],
     ...:                       'd': [0, 1, 2, 0, 1, 2, 3]})
In []: frame
Out[]:
   a  b    c  d
0  0  7  one  0
1  1  6  one  1
2  2  5  one  2
3  3  4  two  0
4  4  3  two  1
5  5  2  two  2
6  6  1  two  3

In []: frame2=frame.set_index(['c','d'])

In []: frame2
Out[]:
       a  b
c   d
one 0  0  7
    1  1  6
    2  2  5
two 0  3  4
    1  4  3
    2  5  2
    3  6  1
#默认情况下,设置为索引的列会从DateFrame中移除,但是也可以留在DataFrame中
In []: frame2=frame.set_index(['c','d'],drop=False)

In []: frame2
Out[]:
       a  b    c  d
c   d
one 0  0  7  one  0
    1  1  6  one  1
    2  2  5  one  2
two 0  3  4  two  0
    1  4  3  two  1
    2  5  2  two  2
    3  6  1  two  3

八、绘图与可视化

分值:6~10分

In []: import matplotlib.pyplot as plt
In []: import numpy as np

In []: data=np.arange(10)

In []: data
Out[]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
In []: plt.plot(data)
Out[]: [<matplotlib.lines.Line2D at 0x283375ca160>]

Python期末作业设计想法 python期末作业写数据分析_元组

GOOD LUCK !



author : Haoyu
school : CSUFT