科学计算系列学习 02：Pandas

科学计算系列学习 01：Numpy
科学计算系列学习 02：Pandas
科学计算系列学习 03：Matplotlib

---

• 生成日期数据

In [50]: pd.date_range('20190701',periods=6)
Out[50]:
DatetimeIndex(['2019-07-01', '2019-07-02', '2019-07-03', '2019-07-04',
               '2019-07-05', '2019-07-06'],
              dtype='datetime64[ns]', freq='D')

• Series

In [55]: np.Series([1,3,4,8,-2],index=['a','b','c','d','e'])
Out[55]:
a    1
b    3
c    4
d    8
e   -2
dtype: int64

• 用np或者直接导入数据生成DataFrame

In [49]: pd.DataFrame(np.random.randn(6,4),index=dates,columns=['a','b','c','d'])
Out[49]:
                   a         b         c         d
2019-07-01 -0.943911  0.930244 -1.002432 -1.495716
2019-07-02 -0.529640  0.559569 -0.552342 -1.403447
2019-07-03  1.226341  0.277729  0.014151  0.154364
2019-07-04 -1.767719 -0.798156 -0.555459 -0.746608
2019-07-05 -0.922795  0.592672  0.295197 -0.187842
2019-07-06  1.384318  0.924977  1.320110 -0.784771

In [48]: pd.DataFrame(np.arange(15).reshape(3,5))
Out[48]:
    0   1   2   3   4
0   0   1   2   3   4
1   5   6   7   8   9
2  10  11  12  13  14

• 传入字典的方式构建DataFrame

key代表列名，行名自动生成从0开始

In [59]: df = pd.DataFrame({'A':1.,'B':pd.Timestamp("20190701"),'C':pd.Series(1,index=list(range(4)),dtype='float32'),'D':np.array([3]*4,dtype='int32'),'E':pd.Categorical(["test","train","test","train"]),'F':'foo'})
Out[59]:
     A          B    C  D      E    F
0  1.0 2019-07-01  1.0  3   test  foo
1  1.0 2019-07-01  1.0  3  train  foo
2  1.0 2019-07-01  1.0  3   test  foo
3  1.0 2019-07-01  1.0  3  train  foo

• 查看每列数据类型

In [61]: df.dtypes
Out[61]:
A           float64
B    datetime64[ns]
C           float32
D             int32
E          category
F            object
dtype: object

• 查看行索引

In [62]: df.index
Out[62]: Int64Index([0, 1, 2, 3], dtype='int64')

• 查看列名称

In [63]: df.columns
Out[63]: Index(['A', 'B', 'C', 'D', 'E', 'F'], dtype='object')

• 查看所有的值

In [64]: df.values
Out[64]:
array([[1.0, Timestamp('2019-07-01 00:00:00'), 1.0, 3, 'test', 'foo'],
       [1.0, Timestamp('2019-07-01 00:00:00'), 1.0, 3, 'train', 'foo'],
       [1.0, Timestamp('2019-07-01 00:00:00'), 1.0, 3, 'test', 'foo'],
       [1.0, Timestamp('2019-07-01 00:00:00'), 1.0, 3, 'train', 'foo']],
      dtype=object)

• 查看描述

只支持数字类型

In [66]: df.describe()
Out[66]:
         A    C    D
count  4.0  4.0  4.0
mean   1.0  1.0  3.0
std    0.0  0.0  0.0
min    1.0  1.0  3.0
25%    1.0  1.0  3.0
50%    1.0  1.0  3.0
75%    1.0  1.0  3.0
max    1.0  1.0  3.0

• 矩阵颠倒，行变列，列变行

In [67]: df.T
Out[67]:
                     0                    1                    2                    3
A                    1                    1                    1                    1
B  2019-07-01 00:00:00  2019-07-01 00:00:00  2019-07-01 00:00:00  2019-07-01 00:00:00
C                    1                    1                    1                    1
D                    3                    3                    3                    3
E                 test                train                 test                train
F                  foo                  foo                  foo                  foo

• 排序

axis：0表示对行索引进行排序，1表示对列索引进行排序；ascending=False表示倒序

In [71]: df.sort_index(axis=0,ascending=False)
Out[71]:
     A          B    C  D      E    F
3  1.0 2019-07-01  1.0  3  train  foo
2  1.0 2019-07-01  1.0  3   test  foo
1  1.0 2019-07-01  1.0  3  train  foo
0  1.0 2019-07-01  1.0  3   test  foo

对指定列E进行排序

In [74]: df.sort_values(by='E')
Out[74]:
     A          B    C  D      E    F
0  1.0 2019-07-01  1.0  3   test  foo
2  1.0 2019-07-01  1.0  3   test  foo
1  1.0 2019-07-01  1.0  3  train  foo
3  1.0 2019-07-01  1.0  3  train  foo

二、选择数据

• 列切片

In [76]: dates = pd.date_range('20190701',periods=6)
In [79]: df = pd.DataFrame(np.arange(24).reshape((6,4)),index=dates,columns=['A','B','C','D'])
Out[79]:
             A   B   C   D
2019-07-01   0   1   2   3
2019-07-02   4   5   6   7
2019-07-03   8   9  10  11
2019-07-04  12  13  14  15
2019-07-05  16  17  18  19
2019-07-06  20  21  22  23

In [81]: df.A
Out[81]:
2019-07-01     0
2019-07-02     4
2019-07-03     8
2019-07-04    12
2019-07-05    16
2019-07-06    20
Freq: D, Name: A, dtype: int32

In [82]: df['A']
Out[82]:
2019-07-01     0
2019-07-02     4
2019-07-03     8
2019-07-04    12
2019-07-05    16
2019-07-06    20
Freq: D, Name: A, dtype: int32

• 行切片

In [84]: df[0:3]
Out[84]:
            A  B   C   D
2019-07-01  0  1   2   3
2019-07-02  4  5   6   7
2019-07-03  8  9  10  11

In [85]: df['2019-07-05':'2019-07-06']
Out[85]:
             A   B   C   D
2019-07-05  16  17  18  19
2019-07-06  20  21  22  23

• 标签选择
  单行

In [94]: df.loc['20190701']
Out[94]:
A    0
B    1
C    2
D    3
Name: 2019-07-01 00:00:00, dtype: int32

多列

In [98]: df.loc[:,['A','B']]
Out[98]:
             A   B
2019-07-01   0   1
2019-07-02   4   5
2019-07-03   8   9
2019-07-04  12  13
2019-07-05  16  17
2019-07-06  20  21

• 位置选择

第三行到第五行，第一列到第三列

In [100]: df.iloc[3:5,1:3]
Out[100]:
             B   C
2019-07-04  13  14
2019-07-05  17  18

• 大于小于等于

打印出A列大于8的所有行

In [103]: df[df.A>8]
Out[103]:
             A   B   C   D
2019-07-04  12  13  14  15
2019-07-05  16  17  18  19
2019-07-06  20  21  22  23

三、更改数据

• 指定标签

In [20]: df.loc['20190701','A']=6

In [21]: df
Out[21]:
             A   B   C   D
2019-07-01   6   1   2   3
2019-07-02   4   5   6   7
2019-07-03   8   9  10  11
2019-07-04  12  13  14  15
2019-07-05  16  17  18  19
2019-07-06  20  21  22  23

• 指定位置

In [24]: df.iloc[2,2]=22

In [25]: df
Out[25]:
             A   B   C   D
2019-07-01   6   1   2   3
2019-07-02   4   5   6   7
2019-07-03   8   9  22  11
2019-07-04  12  13  14  15
2019-07-05  16  17  18  19
2019-07-06  20  21  22  23

• 根据判断条件进行更改

In [26]: df.A[df.A>6]=0

In [27]: df
Out[27]:
            A   B   C   D
2019-07-01  6   1   2   3
2019-07-02  4   5   6   7
2019-07-03  0   9  22  11
2019-07-04  0  13  14  15
2019-07-05  0  17  18  19
2019-07-06  0  21  22  23

四、添加列

• 添加一列 E，设其默认值为NaN

In [29]: df['E']=np.nan

In [30]: df
Out[30]:
            A   B   C   D   E
2019-07-01  6   1   2   3 NaN
2019-07-02  4   5   6   7 NaN
2019-07-03  0   9  22  11 NaN
2019-07-04  0  13  14  15 NaN
2019-07-05  0  17  18  19 NaN
2019-07-06  0  21  22  23 NaN

• 添加一列 F ，指定值，需要指定index

In [31]: df['F'] = pd.Series([1,2,3,4,5,6],index=pd.date_range('20190701',periods=6))

In [32]: df
Out[32]:
            A   B   C   D   E  F
2019-07-01  6   1   2   3 NaN  1
2019-07-02  4   5   6   7 NaN  2
2019-07-03  0   9  22  11 NaN  3
2019-07-04  0  13  14  15 NaN  4
2019-07-05  0  17  18  19 NaN  5
2019-07-06  0  21  22  23 NaN  6

五、处理丢失数据：

• 删除包含NaN的行或列

axis：0代表行，1代表列；any代表包含任意个，all表示全部是。默认为any

In [51]: df
Out[51]:
             A     B     C   D
2019-07-01   0   NaN   2.0   3
2019-07-02   4   5.0   NaN   7
2019-07-03   8   9.0  10.0  11
2019-07-04  12  13.0  14.0  15
2019-07-05  16  17.0  18.0  19
2019-07-06  20  21.0  22.0  23

In [52]: df.dropna(axis=0,how='any')
Out[52]:
             A     B     C   D
2019-07-03   8   9.0  10.0  11
2019-07-04  12  13.0  14.0  15
2019-07-05  16  17.0  18.0  19
2019-07-06  20  21.0  22.0  23

• 将NaN填充指定值

In [60]: df.fillna(value=999)
Out[60]:
                A      B      C   D
2019-07-01  999.0  999.0    2.0   3
2019-07-02    4.0  999.0  999.0   7
2019-07-03    8.0    9.0   10.0  11
2019-07-04   12.0   13.0   14.0  15
2019-07-05   16.0   17.0   18.0  19
2019-07-06   20.0   21.0   22.0  23

• 判断是否有数据丢失

np.any(df.isnull()) ==True

六、导入导出到文件

• 导出

 df.to_csv ('a.csv')

• 导入

默认会自动添加行索引

pd.read_csv('a.csv')

七、数据合并

1、concat

• 纵向或横向合并

默认axis=0，axis=0：纵向合并，axis=1：横向合并。
默认：join = outer ，纵向合并行，横向合并列：outer：合并成并集，inner：合并成交集。
ignore_index=True 横向合并重新排列 列索引号，纵向合并重新排列行索引号
join_axes=[df1.index]，横向合并时，以df1的横向索引为标准

In [100]: df1 = pd.DataFrame(np.ones((3,4))*0,columns=['a','b','c','d'],index=[1,2,3])
In [101]: df2 = pd.DataFrame(np.ones((3,4))*1,columns=['b','c','d','e'],index=[2,3,4])


In [123]: pd.concat([df1,df2])

Out[123]:
     a    b    c    d    e
0  0.0  0.0  0.0  0.0  NaN
1  0.0  0.0  0.0  0.0  NaN
2  0.0  0.0  0.0  0.0  NaN
3  NaN  1.0  1.0  1.0  1.0
4  NaN  1.0  1.0  1.0  1.0
5  NaN  1.0  1.0  1.0  1.0

In [130]: pd.concat([df1,df2],axis=1)
Out[130]:
     a    b    c    d    b    c    d    e
1  0.0  0.0  0.0  0.0  NaN  NaN  NaN  NaN
2  0.0  0.0  0.0  0.0  1.0  1.0  1.0  1.0
3  0.0  0.0  0.0  0.0  1.0  1.0  1.0  1.0
4  NaN  NaN  NaN  NaN  1.0  1.0  1.0  1.0

In [131]: pd.concat([df1,df2],axis=1,ignore_index=True)
Out[131]:
     0    1    2    3    4    5    6    7
1  0.0  0.0  0.0  0.0  NaN  NaN  NaN  NaN
2  0.0  0.0  0.0  0.0  1.0  1.0  1.0  1.0
3  0.0  0.0  0.0  0.0  1.0  1.0  1.0  1.0
4  NaN  NaN  NaN  NaN  1.0  1.0  1.0  1.0

In [137]: pd.concat([df1,df2],axis=1,join_axes=[df1.index])
Out[137]:
     a    b    c    d    b    c    d    e
1  0.0  0.0  0.0  0.0  NaN  NaN  NaN  NaN
2  0.0  0.0  0.0  0.0  1.0  1.0  1.0  1.0
3  0.0  0.0  0.0  0.0  1.0  1.0  1.0  1.0

2、merge

3、追加元素 append

In [143]: df1                                            
Out[143]:                                                
     a    b    c    d                                    
1  0.0  0.0  0.0  0.0                                    
2  0.0  0.0  0.0  0.0                                    
3  0.0  0.0  0.0  0.0                                    
                                                         
In [144]: pd.Series([1,2,3,4],index=['a','b','c','d'])   
Out[144]:                                                
a    1                                                   
b    2                                                   
c    3                                                   
d    4                                                   
dtype: int64                                             
                                                         
In [145]: df1.append(s1,ignore_index=True)               
Out[145]:                                                
     a    b    c    d                                    
0  0.0  0.0  0.0  0.0                                    
1  0.0  0.0  0.0  0.0                                    
2  0.0  0.0  0.0  0.0                                    
3  1.0  2.0  3.0  4.0