python库 mhtml库 python http库

Requests是Python基于Apache2 Licensed许可证的人性化HTTP库。

Python标准库中urllib2提供了不少HTTP 功能，但API不系统。它有点过时，完成最简单的任务也需要大量工作。

下面我们用实例演示访问github。

>>> import requests
>>> r = requests.get('https://api.github.com/user', auth=('[e-mail]', 'password'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf-8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"login":"oychw",...}'
>>> r.json()
{u'disk_usage': 176, u'private_gists': 0, ...}

Requests为Python处理了所有HTTP/1.1操作， 与Web服务的无缝集成。不需要为URL手动添加查询字符串或POST数据进行表单处理。基于urllib3, 能自动处理Keep-alive和HTTP连接池。

特点：

• 国际化域名和 URLs
• Keep-Alive & 连接池
• 持久的 Cookie 会话
• 类浏览器的SSL认证
• 基本/摘要式的身份认证
• 优雅的键/值 Cookie
• 自动解压
• Unicode响应体
• 多段文件上传
• 连接超时
• 支持 .netrc
• 适用于 Python 2.6—3.4
• 线程安全

用户手册

简介

Requests关注PEP 20的部分：

• Beautiful is better than ugly.(美丽优于丑陋)
• Explicit is better than implicit.(明确优于含糊)
• Simple is better than complex.(简单优于复杂)
• Complex is better than complicated.(复杂优于繁琐)
• Readability counts.(可读性)

安装

• 安装： 推荐：pip install requests，其次：easy_install requests
• 最新代码：
• git clone git://github.com/kennethreitz/requests.git
• 下载tar.gz包：curl -OL https://github.com/kennethreitz/requests/tarball/master
• 下载zip包：curl -OL https://github.com/kennethreitz/requests/zipball/master
• 源码安装：python setup.py install

快速入门

• 发送请求：

下面获取Github的公共时间线，并在httpbin演示其他HTTP操作：

>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r = requests.post("http://httpbin.org/post")
>>> r = requests.put("http://httpbin.org/put")
>>> r = requests.delete("http://httpbin.org/delete")
>>> r = requests.head("http://httpbin.org/get")
>>> r = requests.options("http://httpbin.org/get")

• 在URL中传递参数

URL的查询字符串(query string)例如， httpbin.org/get?key=val，在Requests可以用字典的形式构建。比如传递key1=value1和key2=value2到 httpbin.org/get:

>>> import requests
>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.get("http://httpbin.org/get", params=payload)
>>> print(r.url)
http://httpbin.org/get?key2=value2&key1=value1
>>> payload = {'key1': 'value1', 'key2[]': ['value2', 'value3']}
>>> r = requests.get("http://httpbin.org/get", params=payload)
>>> print(r.url)
http://httpbin.org/get?key1=value1&key2%5B%5D=value2&key2%5B%5D=value3

注意字典里值为None的键会忽略。上面第２个例子访问的是http://httpbin.org/get?key1=value1&key2[]=value2&key2[]=value3 。注意key后面需要添加中括号对。

• 响应

>>> import requests
>>> r = requests.get('https://api.github.com/events')
>>> r.text
>>> u'[{"id":"2636319727","type":"PullRequestReviewCommentEvent","actor":{"id":1148601,"login":"i ...}]
>>> r.encoding
'utf-8'
>>> r.encoding = 'ISO-8859-1'

Requests会自动解码服务器的返回。大多数unicode字符集都能无缝解码。请求发出时Requests会基于响应的HTTP头部推测响应的编码。同时还可以设置和查询编码。改变编码后，访问 r.text 将会使用 r.encoding 。

• 二进制响应

r.content可以以字节的方式显示响应。

>>> r.content
b'[{"repository":{"open_issues":0,"url":"https://github.com/...

传输格式gzip和deflate会自动转码。处理图片实例：

>>> from PIL import Image
>>> from StringIO import StringIO
>>> i = Image.open(StringIO(r.content))

• Json响应

Requests内置了JSON解码器:

>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.json()
{u'documentation_url': u'https://developer.github.com/v3/activity/events/#list-public-events', u'message': u"Hello there, wayfaring stranger. If you're reading this then you probably didn't see our blog post a couple of years back announcing that this API would go away: http://git.io/17AROg Fear not, you should be able to get what you need from the shiny new Events API instead."}

JSON解码失败时r.json 就会抛出异常。例如， 401 (Unauthorized) ， ValueError: No JSON object could be decoded等。

• 原始响应

极端的情况下需要查看服务器的原始套接字响应，请求时设置 stream=True：

>>> import requests
>>> r = requests.get('https://api.github.com/events', stream=True)
>>> r.raw
<urllib3.response.HTTPResponse object at 0x7f807dd6f4d0>
>>> r.raw.read(10)
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'

通常需要存为文件:

with open(filename, 'wb') as fd:
    for chunk in r.iter_content(chunk_size):
        fd.write(chunk)

Response.iter_content 能减少直接使用Response.raw的大量处理，下载流时尤其推荐。

• 自定义头

>>> import requests
>>> import json
>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
>>> headers = {'content-type': 'application/json'}
>>> r = requests.post(url, data=json.dumps(payload), headers=headers)

• 更加复杂的POST请求

表单直接以字典形式发送：

>>> import requests
>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.post("http://httpbin.org/post", data=payload)
>>> print(r.text)
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "key1": "value1", 
    "key2": "value2"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, compress", 
    "Content-Length": "23", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-53-generic"
  }, 
  "json": null, 
  "origin": "119.122.150.177", 
  "url": "http://httpbin.org/post"
}

string则会被直接发布出去。Github API v3中接受编码为JSON的POST/PATCH数据

>>> import requests
>>> import json
>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
>>> r = requests.post(url, data=json.dumps(payload))

• POST复杂编码的文件

>>> import requests
>>> url = 'http://httpbin.org/post'
>>> files = {'file': open('/home/andrew/test.xls', 'rb')}
>>>  r = requests.post(url, files=files)
>>> url = 'http://httpbin.org/post'
>>> r.text
u'{\n  "args": {}, \n  "data": "", \n  "files": {\n ...  "url": "http://httpbin.org/post"\n}\n'

可以显式地设置文件名，文件类型和请求头:

>>> import requests
>>> files = {'file': ('report.xls', open('/home/andrew/test.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}
>>> url = 'http://httpbin.org/post'
>>> r = requests.post(url, files=files)
>>> r.text
u'{\n  "args": {}, \n  "data": ""..."url": "http://httpbin.org/post"\n}\n'

还可以直接用文字代替文件：

>>> import requests
>>> url = 'http://httpbin.org/post'
>>> files = {'file': ('report.csv', 'some,data,to,send\nanother,row,to,send\n')}
>>> r = requests.post(url, files=files)
>>> r.text
u'{\n  "args": {}, \n  "data": "", \n  "files": {\n  ...  "json": null, \n  "origin": "14.153.22.104", \n  "url": "http://httpbin.org/post"\n}\n'

multipart/form-data不支持特别大的文件，建议使用requests-toolbelt，参考：toolbelt

• 响应状态码

>>> import requests
>>> r = requests.get('http://httpbin.org/get')
>>> r.status_code
200
>>> r.status_code == requests.codes.ok
True
>>> bad_r = requests.get('http://httpbin.org/status/404')
>>> bad_r.status_code
404
>>> bad_r.raise_for_status()
>>> r.raise_for_status()

上面的requests.codes.ok是内置的状态码查询对象。可以使用 Response.raise_for_status()跑出失败请求(4XX客户端错误或5XX服务器异常)，我们可以通过 Response.raise_for_status() 来抛出异常。r的返回为200，所以返回None，不产生异常。

• 响应头

>>> r.headers
{'content-length': '275', 'server': 'nginx', 'connection': 'keep-alive', 'access-control-allow-credentials': 'true', 'date': 'Tue, 10 Mar 2015 08:21:36 GMT', 'access-control-allow-origin': '*', 'content-type': 'application/json'}

>>> r.headers['Content-Type']
'application/json'

>>> r.headers.get('content-type')
 'application/json'

根据 RFC 2616，HTTP头部不区分大小写。根据RFC 7230，接收方会对服务端对同一key的不同value进行组合。

• Cookies

可以访问响应中包含的Cookie:

>>> import requests
>>> url = 'http://automationtesting.sinaapp.com/login'
>>> r = requests.get(url)
>>> r.cookies.keys()
['saeut', 'trac_form_token', 'trac_session']
>>> r.cookies['saeut']
 'CkMPGlT+tfQiXS9uGYviAg=='

使用cookies参数可以发送你的cookies到服务器:

>>> import requests
>>> url = 'http://httpbin.org/cookies'
>>> cookies = dict(cookies_are='working')
>>> r = requests.get(url, cookies=cookies)
>>> r.text
u'{\n  "cookies": {\n    "cookies_are": "working"\n  }\n}\n'

• 重定向与请求历史

默认对HEAD以外其他所有动作进行位置重定向。Response.history可以看到重定向的记录。

>>> import requests
>>> r = requests.get('http://github.com')
>>> r.url
u'https://github.com/'
>>> r.status_code
200
>>> r.history
[<Response [301]>]

GET, OPTIONS, POST, PUT, PATCH 或者 DELETE可以通过allow_redirects参数禁用重定向，这个设置对HEAD也生效:

>>> import requests
>>> r = requests.get('http://github.com', allow_redirects=False)
>>> r.status_code
301
>>> r.history
[]
>>> r = requests.head('http://github.com', allow_redirects=True)
>>> r.url
u'https://github.com/'
>>> r.history
[<Response [301]>]

• 超时

超时告诉requests在经过timeout参数的秒之后停止等待响应:

>>> import requests
>>> requests.get('http://github.com', timeout=0.1)    
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 387, in send
    raise Timeout(e)
requests.exceptions.Timeout: (<urllib3.connectionpool.HTTPConnectionPool object at 0x7f807dd6f050>, 'Connection to github.com timed out. (connect timeout=0.1)')
>>>

注: 超时不是对整个响应下载的时间限制, 而且指定时间没有收到服务器返回就抛出异常。

• 错误与异常

ConnectionError：网络问题(如DNS失败、拒绝连接等)。

HTTPError: 比较罕见的无效HTTP响应时。

Timeout：请求超时。

TooManyRedirects：超过了设定的最大重定向次数。

requests.exceptions.RequestException是所有具体异常的基类。

高级用法

• Session对象

Session对象能够跨请求保持参数，Session实例发出的所有请求共享cookies。

Session对象具有主Requests API的所有方法。

>>> s = requests.Session()
>>> s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
<Response [200]>
>>> r = s.get("http://httpbin.org/cookies")
>>> print(r.text)
{
  "cookies": {
    "sessioncookie": "123456789"
  }
}

Session也可为request方法提供缺省数据，添加属性即可:

>>> import requests
>>> s = requests.Session()
>>> s.auth = ('user', 'pass')
>>> s.headers.update({'x-test': 'true'})
>>> s.get('http://httpbin.org/headers', headers={'x-test2': 'true'})
<Response [200]>

传递给request方法的字典都会与已有session层的值合并。方法层的参数会覆盖会话的参数。在方法层参数中将键值设置为None，会被自动忽略key。参考： session api

• 请求(Request)和响应(Response)对象

requests.get()等请求主要做两件的事情。一为构建Request 对象。二为收到服务器响应时产生Response 对象。Response对象包含服务器返回和原来的 Request 对象。

>>> import requests
>>> r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')
>>> r.headers
{'content-length': '67559', ...}
>>> r.request.headers
{'Connection': 'keep-alive', 'Accept-Encoding': ...}

• 预请求

当从API或会话调用接收Response对象时，request属性实际上是PreparedRequest。如果你需要修改body或header，可以如下方式进行处理：

from requests import Request, Session

s = Session()
req = Request('GET', url,
    data=data,
    headers=header
)
prepped = req.prepare()

# do something with prepped.body
# do something with prepped.headers

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)

这里没有对Request对象进行特殊处理，而是修改PreparedRequest对象。然后用requests.* 或Session.*.发送。

上述代码没有Request Session。Session层状态，如cookie不会使用。用Session.prepare_request()替换Request.prepare()即可增加状态支持：

from requests import Request, Session

s = Session()
req = Request('GET',  url,
    data=data
    headers=headers
)

prepped = s.prepare_request(req)

# do something with prepped.body
# do something with prepped.headers

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)

• SSL证书验证

使用 verify 参数可以像web浏览器一样为HTTPS请求验证SSL证书: