正文
博客地址：https://www.cnblogs.com/zylyehuo/
开发环境
- anaconda
- 集成环境：集成好了数据分析和机器学习中所需要的全部环境
- 安装目录不可以有中文和特殊符号
- jupyter
- anaconda提供的一个基于浏览器的可视化开发工具
import numpy as np
import pandas as pd
为了方便操作，将月份和参选人以及所在政党进行定义
months = {'JAN' : 1, 'FEB' : 2, 'MAR' : 3, 'APR' : 4, 'MAY' : 5, 'JUN' : 6,
'JUL' : 7, 'AUG' : 8, 'SEP' : 9, 'OCT': 10, 'NOV': 11, 'DEC' : 12}
of_interest = ['Obama, Barack', 'Romney, Mitt', 'Santorum, Rick',
'Paul, Ron', 'Gingrich, Newt']
parties = {
'Bachmann, Michelle': 'Republican',
'Romney, Mitt': 'Republican',
'Obama, Barack': 'Democrat',
"Roemer, Charles E. 'Buddy' III": 'Reform',
'Pawlenty, Timothy': 'Republican',
'Johnson, Gary Earl': 'Libertarian',
'Paul, Ron': 'Republican',
'Santorum, Rick': 'Republican',
'Cain, Herman': 'Republican',
'Gingrich, Newt': 'Republican',
'McCotter, Thaddeus G': 'Republican',
'Huntsman, Jon': 'Republican',
'Perry, Rick': 'Republican'
}
需求
- 加载数据
- 查看数据的基本信息
- 指定数据截取，将如下字段的数据进行提取，其他数据舍弃
- cand_nm ：候选人姓名
- contbr_nm ： 捐赠人姓名
- contbr_st ：捐赠人所在州
- contbr_employer ： 捐赠人所在公司
- contbr_occupation ： 捐赠人职业
- contb_receipt_amt ：捐赠数额（美元）
- contb_receipt_dt ： 捐款的日期
- 对新数据进行总览,查看是否存在缺失数据
- 用统计学指标快速描述数值型属性的概要。
- 空值处理。可能因为忘记填写或者保密等等原因，相关字段出现了空值，将其填充为NOT PROVIDE
- 异常值处理。将捐款金额<=0的数据删除
- 新建一列为各个候选人所在党派party
- 查看party这一列中有哪些不同的元素
- 统计party列中各个元素出现次数
- 查看各个党派收到的政治献金总数contb_receipt_amt
- 查看具体每天各个党派收到的政治献金总数contb_receipt_amt
- 将表中日期格式转换为'yyyy-mm-dd'
- 查看老兵(捐献者职业)DISABLED VETERAN主要支持谁
加载数据并查看数据的基本信息
df = pd.read_csv('./data/usa_election.txt')
df
对数据进行总览,查看是否存在缺失数据
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 536041 entries, 0 to 536040
Data columns (total 16 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 cmte_id 536041 non-null object
1 cand_id 536041 non-null object
2 cand_nm 536041 non-null object
3 contbr_nm 536041 non-null object
4 contbr_city 536026 non-null object
5 contbr_st 536040 non-null object
6 contbr_zip 535973 non-null object
7 contbr_employer 525088 non-null object
8 contbr_occupation 530520 non-null object
9 contb_receipt_amt 536041 non-null float64
10 contb_receipt_dt 536041 non-null object
11 receipt_desc 8479 non-null object
12 memo_cd 49718 non-null object
13 memo_text 52740 non-null object
14 form_tp 536041 non-null object
15 file_num 536041 non-null int64
dtypes: float64(1), int64(1), object(14)
memory usage: 65.4+ MB
用统计学指标快速描述数值型属性的概要
df.describe()
空值处理。可能因为忘记填写或者保密等等原因，相关字段出现了空值，将其填充为NOT PROVIDE
df.fillna(value='NOT PROVIDE',inplace=True)
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 536041 entries, 0 to 536040
Data columns (total 16 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 cmte_id 536041 non-null object
1 cand_id 536041 non-null object
2 cand_nm 536041 non-null object
3 contbr_nm 536041 non-null object
4 contbr_city 536041 non-null object
5 contbr_st 536041 non-null object
6 contbr_zip 536041 non-null object
7 contbr_employer 536041 non-null object
8 contbr_occupation 536041 non-null object
9 contb_receipt_amt 536041 non-null float64
10 contb_receipt_dt 536041 non-null object
11 receipt_desc 536041 non-null object
12 memo_cd 536041 non-null object
13 memo_text 536041 non-null object
14 form_tp 536041 non-null object
15 file_num 536041 non-null int64
dtypes: float64(1), int64(1), object(14)
memory usage: 65.4+ MB
异常值处理。将捐款金额<=0的数据删除
df['contb_receipt_amt'] <= 0 # 判断哪些值为小于等于0
df.loc[df['contb_receipt_amt'] <= 0] # 捐赠金额小于等于0的行数据
drop_indexs = df.loc[df['contb_receipt_amt'] <= 0].index
df.drop(labels=drop_indexs,axis=0,inplace=True)
df
新建一列为各个候选人所在党派party
df['party'] = df['cand_nm'].map(parties)
df.head()
查看party这一列中有哪些不同的元素
df['party'].unique()
array(['Republican', 'Democrat', 'Reform', 'Libertarian'], dtype=object)
统计party列中各个元素出现次数
df['party'].value_counts()
Democrat 289999
Republican 234300
Reform 5313
Libertarian 702
Name: party, dtype: int64
查看各个党派收到的政治献金总数contb_receipt_amt
df.groupby(by='party')['contb_receipt_amt'].sum()
party
Democrat 8.259441e+07
Libertarian 4.132769e+05
Reform 3.429658e+05
Republican 1.251181e+08
Name: contb_receipt_amt, dtype: float64
查看具体每天各个党派收到的政治献金总数contb_receipt_amt
df.groupby(by=['contb_receipt_dt','party'])['contb_receipt_amt'].sum()
contb_receipt_dt party
01-APR-11 Reform 50.00
Republican 12635.00
01-AUG-11 Democrat 182198.00
Libertarian 1000.00
Reform 1847.00
...
31-MAY-11 Republican 313839.80
31-OCT-11 Democrat 216971.87
Libertarian 4250.00
Reform 3205.00
Republican 751542.36
Name: contb_receipt_amt, Length: 1183, dtype: float64
将表中日期格式转换为'yyyy-mm-dd'
def trandformDate(d):
day,month,year = d.split('-')
month = months[month]#将英文形式的月份转换成了数字形式的月份
return '20'+year+'-'+str(month)+'-'+day
df['contb_receipt_dt'] = df['contb_receipt_dt'].map(trandformDate)
df.head()
查看老兵(捐献者职业)DISABLED VETERAN主要支持谁
# 可以先将源数据中的老兵这个职业对应的行数据取出
df['contbr_occupation'] == 'DISABLED VETERAN'
0 False
1 False
2 False
3 False
4 False
...
536036 False
536037 False
536038 False
536039 False
536040 False
Name: contbr_occupation, Length: 530314, dtype: bool
df_old = df.loc[df['contbr_occupation'] == 'DISABLED VETERAN']
df_old
# 分组：根据候选人分组，对捐赠金额求和
df_old.groupby(by='cand_nm')['contb_receipt_amt'].sum()
cand_nm
Cain, Herman 300.00
Obama, Barack 4205.00
Paul, Ron 2425.49
Santorum, Rick 250.00
Name: contb_receipt_amt, dtype: float64
