python 多进程 非阻塞 python多进程数量限制

限制并发连接数的最简单方法是使用线程池：

#!/usr/bin/env python
from itertools import izip, repeat
from multiprocessing.dummy import Pool # use threads for I/O bound tasks
from urllib2 import urlopen
def fetch(url_data):
try:
return url_data[0], urlopen(*url_data).read(), None
except EnvironmentError as e:
return url_data[0], None, str(e)
if __name__=="__main__":
pool = Pool(20) # use 20 concurrent connections
params = izip(urls, repeat(data)) # use the same data for all urls
for url, content, error in pool.imap_unorderred(fetch, params):
if error is None:
print("done: %s: %d" % (url, len(content)))
else:
print("error: %s: %s" % (url, error))

503 Service Unavailable是服务器错误。它可能无法处理负载。

Name or service not known是一个dns错误。如果需要多次请求，请安装/启用本地缓存dns服务器。