1.select模块
1.1概述
This module provides access to the select() and poll() functions available in most operating systems, devpoll() available on Solaris and derivatives, epoll() available on Linux 2.5+ and kqueue() available on most BSD. Note that on Windows, it only works for sockets; on other operating systems, it also works for other file types (in particular, on Unix, it works on pipes). It cannot be used on regular files to determine whether a file has grown since it was last read.
这个模块提供了对多个函数的访问, 包括:大多数系统都支持的select()和poll()函数, Solaris及其衍生系统支持的devpoll(), linxt2.5+支持的epoll()函数, 以及大多数BSD支持的kququ()函数;需要注意:在Windows系统中, 此模块只对socket有效, 在其他系统中可能支持其他的文件类型,例如Unix上对管道也有效; 此模块不能用在常规文件上来确定文件自上次读取后是否已更改;
Note:
The selectors module allows high-level and efficient I/O multiplexing, built upon the select module primitives. Users are encouraged to use the selectors module instead, unless they want precise control over the OS-level primitives used.
selectors模块允许高级和高效的IO多路复用,但它是建立在select模块上的; 鼓励用户去使用selectors模块, 但如果你想更精确的控制系统级别的原语;
1.2模块中的函数
(1)exception select.error
A deprecated alias of OSError.
OSError的别名,不鼓励使用
(2)select.devpoll()
1)(Only supported on Solaris and derivatives.) Returns a /dev/poll polling object; see section /dev/poll Polling Objects below for the methods supported by devpoll objects.
2)devpoll() objects are linked to the number of file descriptors allowed at the time of instantiation. If your program reduces this value, devpoll() will fail. If your program increases this value, devpoll() may return an incomplete list of active file descriptors.
The new file descriptor is non-inheritable.
只在Solaris和其衍生系统上支持, 暂不研究;
(3)select.epoll(sizehint=-1, flags=0)
1)(Only supported on Linux 2.5.44 and newer.) Return an edge polling object, which can be used as Edge or Level Triggered interface for I/O events. sizehint and flags are deprecated and completely ignored.
2)See the Edge and Level Trigger Polling (epoll) Objects section below for the methods supported by epolling objects.
3)epoll objects support the context management protocol: when used in a with statement, the new file descriptor is automatically closed at the end of the block.
The new file descriptor is non-inheritable.
只在Linux2.5.44或更新版本上支持; 返回一个edge polling对象, 可以用作边缘触发或水平触发的io事件的接口; 给出的参数无视就好, 不推荐使用
其余待补充.
(4)select.poll()
(Not supported by all operating systems.) Returns a polling object, which supports registering and unregistering file descriptors, and then polling them for I/O events; see section Polling Objects below for the methods supported by polling objects
不是所有系统都支持; 返回一个polling对象, 支持文件描述符的注册和取消注册,并且轮询他们以查找IO事件; 暂不研究
(5)select.kqueue()
(Only supported on BSD.) Returns a kernel queue object; see section Kqueue Objects below for the methods supported by kqueue objects.
The new file descriptor is non-inheritable.
只在BSD系统上支持; 暂不研究
(6)select.kevent(ident, filter=KQ_FILTER_READ, flags=KQ_EV_ADD, fflags=0, data=0, udata=0)
(Only supported on BSD.) Returns a kernel event object; see section Kevent Objects below for the methods supported by kevent objects.
只在BSD系统上支持; 暂不研究
(7)select.select(rlist, wlist, xlist[, timeout])
This is a straightforward interface to the Unix select() system call. The first three arguments are sequences of ‘waitable objects’: either integers representing file descriptors or objects with a parameterless method named fileno() returning such an integer:
•rlist: wait until ready for reading
•wlist: wait until ready for writing
•xlist: wait for an “exceptional condition” (see the manual page for what your system considers such a condition)Empty sequences are allowed, but acceptance of three empty sequences is platform-dependent. (It is known to work on Unix but not on Windows.) The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks.
The return value is a triple of lists of objects that are ready: subsets of the first three arguments. When the time-out is reached without a file descriptor becoming ready, three empty lists are returned.
Among the acceptable object types in the sequences are Python file objects (e.g. sys.stdin, or objects returned by open() or os.popen()), socket objects returned by socket.socket(). You may also define a wrapper class yourself, as long as it has an appropriate fileno() method (that really returns a file descriptor, not just a random integer).
1)是Unix系统上select()系统调用的简单接口; 开头3个参数是 ''等待的对象" 的序列: 它们要么是代表文件描述符的整数, 要么是具有无参数的file()方法的对象, 对象的file()方法会返回代表文件描述符的整数;
其中: rlist: 等待直到准备好读取; wlist:等待直到准备好写入; xlist: 等待异常情况;
2)参数允许是空序列, 但是否允许3个都是空序列就取决于平台了(已知在Unix上可以但在Windows上不允许); 另外一个参数timeout是浮点数, 指明超时的秒数, 如果不给出timeout参数则函数会一直阻塞到至少有一个文件描述符准备好, 如果参数给为0, 那永远不会阻塞;
3)返回值是3个已经准备好的对象的列表; 它们是头3个输入参数的子集; 如果等待时间超过timeout参数并且没有文件描述符准备好那就返回3个空列表;
4)在序列中可接受的对象类型包括python文件类型(例如sys.stdin, 或open()和os.popen()返回的对象, socket对象等), 你也可以自定义一个封装类, 只要它有合适的fileno()方法(可以返回真的文件描述符, 而不是一个随机整数);
Note:
File objects on Windows are not acceptable, but sockets are. On Windows, the underlying select() function is provided by the WinSock library, and does not handle file descriptors that don’t originate from WinSock.
注意:文件对象在windows系统上是不被接收的, 但socket可以; 在Windows系统中, 基础的select()函数是被WinSock库提供的 , 并不会处理不是WinSock生成的文件描述符;
使用示例:
import socket
import select
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('127.0.0.1', 8000))
s.listen(5)
s.setblocking(False) # 设不设置都行
r_list = [s]
while True:
r, w, e = select.select(r_list, [], [],)
for x in e: continue
for i in r:
if i == s:
con, addr = i.accept()
print(addr, '连接成功')
r_list.append(con)
else:
try:
data = i.recv(1024)
if not data:
r_list.remove(i)
continue
except Exception as e:
print(e)
r_list.remove(i)
continue
print(data.decode('utf8'))
msg = input('发送:')
i.send(msg.encode('utf8'))
#client
import socket
c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.connect(('127.0.0.1', 8000))
while True:
msg = input('发送:')
c.send(msg.encode('utf8'))
data = c.recv(1024)
print(data.decode('utf8'))(8)select.PIPE_BUF
The minimum number of bytes which can be written without blocking to a pipe when the pipe has been reported as ready for writing by select(), poll() or another interface in this module. This doesn’t apply to other kind of file-like objects such as sockets.
This value is guaranteed by POSIX to be at least 512. Availability: Unix.
当管道被select()或其他接口报告已经准备好写入时, 可在无阻塞状态下写入的最小字节数. 不适应于socket对象.
对于Unix系统, POSIX已保证该值最小为512字节;
2. selectors模块
2.1概述
It defines a BaseSelector abstract base class, along with several concrete implementations (KqueueSelector, EpollSelector…), that can be used to wait for I/O readiness notification on multiple file objects. In the following, “file object” refers to any object with a fileno() method, or a raw file descriptor. See file object.
DefaultSelector is an alias to the most efficient implementation available on the current platform: this should be the default choice for most users.
此模块定义了一个抽象类BaseSelector, 以及几个具体实现类(如KueueSelector, EpollSelector等), 可用来等待多个文件对象的就绪通知; 在本文中, 文件对象指的是任何具有fileno()方法的对象或者文件描述符;
DefaultSelector类是当前平台可用的最高效的类的代称: 这应该是大多数用户的默认选择!!
类的等级:
BaseSelector
– SelectSelector
– PollSelector
– EpollSelector
– DevpollSelector
– KqueueSelector
In the following, events is a bitwise mask indicating which I/O events should be waited for on a given file object. It can be a combination of the modules constants below
在下文中, events参数是一个按尾掩码, 用于指示给定文件对象的哪些IO事件应该被监听; 可以是下面几个常量的组合:
EVENT_READ 表示监听可读事件
EVENT_WRITE 表示监听可写事件
2.2 class selectors.SelectorKey
A SelectorKey is a namedtuple used to associate a file object to its underlying file descriptor, selected event mask and attached data. It is returned by several BaseSelector methods.
fileobj
File object registered.
fd
Underlying file descriptor.
events
Events that must be waited for on this file object.
data
Optional opaque data associated to this file object: for example, this could be used to store a per-client session ID.
SelectorKey是一个命名元组(什么东西?), 可以将文件对象与它的基础文件描述符, 选择的事件掩码, 以及附加数据相关联. 几种BaseSelector的方法会返回它的实例.
fileobj: 注册的文件对象
fd: 底层的文件描述符
events: 文件对象必须等待的事件
data: 与文件对象相关联的数据, 例如可用来存储每个客户端的会话ID.
2.3class selectors.BaseSelector
A BaseSelector is used to wait for I/O event readiness on multiple file objects. It supports file stream registration, unregistration, and a method to wait for I/O events on those streams, with an optional timeout. It’s an abstract base class, so cannot be instantiated. Use DefaultSelector instead, or one of SelectSelector, KqueueSelector etc. if you want to specifically use an implementation, and your platform supports it. BaseSelector and its concrete implementations support the context manager protocol.
用于等待多个文件对象的IO事件的就绪; 它支持文件流的注册, 取消注册, 以及一个等待这些流的IO事件的方法(该方法还附带一个超时参数). 它是一个抽象基类, 因此无法实例化; 使用DefaultSelector, 或SelectSelector, KqueueSelector等中的一个来替代(如果你想使用一种专门的实现,并且你的平台支持它); 另外BaseSelector及它的具体实现支持上下文管理协议.
2.3.1类的方法
(1)abstractmethod register(fileobj, events, data=None)
Register a file object for selection, monitoring it for I/O events.
fileobj is the file object to monitor. It may either be an integer file descriptor or an object with a fileno() method. events is a bitwise mask of events to monitor. data is an opaque object.
This returns a new SelectorKey instance, or raises a ValueError in case of invalid event mask or file descriptor, or KeyError if the file object is already registered.
注册一个文件对象以监听其IO事件;
fileobj参数是要监听的文件对象, 可以是整数(文件描述符)或具有fileno()方法的对象; events参数是要监听事件的按位掩码; data参数是对象附带的数据, 该方法并不会使用它;
该方法返回一个SelectorKey的实例,或者是ValueError异常(如果输入的是无效的事件掩码或文件描述符), 或者是KeyError异常(如果文件对象已经注册)
(2)abstractmethod unregister(fileobj)
Unregister a file object from selection, removing it from monitoring. A file object shall be unregistered prior to being closed.
fileobj must be a file object previously registered.
This returns the associated SelectorKey instance, or raises a KeyError if fileobj is not registered. It will raise ValueError if fileobj is invalid (e.g. it has no fileno() method or its fileno() method has an invalid return value).
从selection中注销文件对象, 即从监听列表中移除它; 文件对象应该在关闭前注销.
fileobj参数必须是一个之前注册的文件对象;
该方法返回相关联的SelectorKey实例, 如果文件对象未注册则返回KeyError, 如果文件对象无效(例如没有fileno()方法或该方法的返回值无效)则返回ValueError异常.
(3)modify(fileobj, events, data=None)
Change a registered file object’s monitored events or attached data.
This is equivalent to BaseSelector.unregister(fileobj)() followed by BaseSelector.register(fileobj, events, data)(), except that it can be implemented more efficiently.
This returns a new SelectorKey instance, or raises a ValueError in case of invalid event mask or file descriptor, or KeyError if the file object is not registered.
修改已注册文件的监听事件或其附加数据;
相当于先调用unregister()方法然后调用register()方法, 只不过使用这个方法实现更高效;
返回一个新的SelectorKey实例, 如果事件掩码无效或文件描述符无效则返回ValueError异常, 文件对象未注册则返回KeyError异常;
(4)abstractmethod select(timeout=None)
1)Wait until some registered file objects become ready, or the timeout expires.
2)If timeout > 0, this specifies the maximum wait time, in seconds. If timeout <= 0, the call won’t block, and will report the currently ready file objects. If timeout is None, the call will block until a monitored file object becomes ready.
3)This returns a list of (key, events) tuples, one for each ready file object.
key is the SelectorKey instance corresponding to a ready file object. events is a bitmask of events ready on this file object.Note:
This method can return before any file object becomes ready or the timeout has elapsed if the current process receives a signal: in this case, an empty list will be returned.Changed in version 3.5: The selector is now retried with a recomputed timeout when interrupted by a signal if the signal handler did not raise an exception (see PEP 475 for the rationale), instead of returning an empty list of events before the timeout.
1.等待直到注册的文件对象就绪, 或者超时到期;
2.如果timeout参数>0, 那它表示最大的等待时间秒数; 如果<=0, 那该调用不会阻塞,并返回当前就绪的文件对象; 如果timeout是None, 那一直阻塞直到监听到就绪的文件对象;
3.该方法返回一个列表, 列表中元素是每个就绪文件对象的(key, events)组成的元组;其中key是与文件对象相关联的SelectorKey的实例, events是文件对象已就绪的事件的按位掩码;
4.注意: 此方法可以在没有任何文件对象就绪的情况下返回,例如超时过去,在这种情况下, 将返回一个空列表;
版本3.5更新: selector在被信号打断时,如果这个信号没有引起异常, 那它会重新计算超时而不是返回空列表;
(5)close()
Close the selector.
This must be called to make sure that any underlying resource is freed. The selector shall not be used once it has been closed.
关闭selector, 调用此方法以确保所有底层资源已被释放,selector被关闭后无法再使用;
(6)get_key(fileobj)
Return the key associated with a registered file object.
This returns the SelectorKey instance associated to this file object, or raises KeyError if the file object is not registered.
返回注册文件对象的相关Key;
即返回注册文件对象相关的SelectorKey实例,如果文件对象未注册则抛出异常
(7)abstractmethod get_map()
Return a mapping of file objects to selector keys.
This returns a Mapping instance mapping registered file objects to their associated SelectorKey instance.
返回文件对象和key的映射;
即返回一个映射对象, 映射注册的文件对象及其相关联的SelectorKey实例
2.4 class selectors.DefaultSelector
The default selector class, using the most efficient implementation available on the current platform. This should be the default choice for most users.
默认的selector类, 会自动选择当前平台上最高效的类, 是大多数用户的首选!
2.5其他类,不常用
1)class selectors.SelectSelector
select.select()-based selector.2)class selectors.PollSelector
select.poll()-based selector.3)class selectors.EpollSelector
select.epoll()-based selector.
fileno()
This returns the file descriptor used by the underlying select.epoll() object.4)class selectors.DevpollSelector
select.devpoll()-based selector.
fileno()
This returns the file descriptor used by the underlying select.devpoll() object.5)class selectors.KqueueSelector
select.kqueue()-based selector.
fileno()
This returns the file descriptor used by the underlying select.kqueue() object.
3. 使用示例
示例1: 简单使用,实现多个socket收发消息并发
import socket
import selectors
def accepting(socker):
con, addr = socker.accept()
print(addr, '已连接')
ser.register(con, selectors.EVENT_READ, read)
def read(con):
data = con.recv(1024)
print(data.decode('utf8'))
msg = input('发送:').encode('utf8')
con.send(msg)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('127.0.0.1', 8000))
s.listen(5)
s.setblocking(False) # 阻塞模式设不设都行
ser = selectors.DefaultSelector()
ser.register(s, selectors.EVENT_READ, accepting)
print(ser.get_key(s))
print(ser.get_map())
print('start...')
while True:
e = ser.select()
for key, musk in e: # key是个selectorkey对象, 有属性fileobj, data, 等
cb = key.data
cb(key.fileobj)
print(e)
#client
import socket
c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.connect(('127.0.0.1', 8000))
while True:
msg = input('发送:')
c.send(msg.encode('utf8'))
data = c.recv(1024)
print(data.decode('utf8'))
#result
SelectorKey(fileobj=<socket.socket fd=468, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8000)>, fd=468, events=1, data=<function accepting at 0x00000223AF7AC1E0>)
<selectors._SelectorMapping object at 0x00000223AFAE9630>
start...
('127.0.0.1', 57612) 已连接
[(SelectorKey(fileobj=<socket.socket fd=468, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8000)>, fd=468, events=1, data=<function accepting at 0x00000223AF7AC1E0>), 1)]示例2:使用selectors模块实现多客户端文件上传与下载
import socket
import selectors
import pickle,os
def accepting(socker):
con, addr = socker.accept()
print(addr, '已连接')
ser.register(con, selectors.EVENT_READ, upload)
def upload(con):
file_name = obj_dic[con]['filename']
if os.path.exists(file_name):
# file_size = os.path.getsize(file_name)
mode = 'ab'
else:
mode = 'wb'
with open(file_name, mode) as f1:
data = con.recv(1024)
f1.write(data)
if os.path.getsize(file_name) >= obj_dic[con]['size']:
obj_dic.pop(con)
def download(con):
filename = obj_dic[con]['filename']
if os.path.isfile(filename):
filesize = os.path.getsize(filename)
if obj_dic[con]['sendsize'] == 0:
con.send(pickle.dumps(filesize))
with open(filename, 'rb') as f:
f.seek(obj_dic[con]['sendsize'])
i = f.readline()
con.sendall(i)
obj_dic[con]['sendsize'] += len(i)
if obj_dic[con]['sendsize'] >= filesize:
obj_dic.pop(con)
ser.modify(con, selectors.EVENT_READ, upload)
else:
con.send(pickle.dumps(0))
obj_dic.pop(con)
ser.modify(con, selectors.EVENT_READ, upload)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('127.0.0.1', 8000))
s.listen(5)
ser = selectors.DefaultSelector()
ser.register(s, selectors.EVENT_READ, accepting)
obj_dic = {}
print('start...')
while True:
e = ser.select()
for key, musk in e:
if key.fileobj == s:
cb = key.data
cb(key.fileobj)
else:
if key.fileobj not in obj_dic:
try:
msg = key.fileobj.recv(1024)
except Exception as e:
print(e)
ser.unregister(key.fileobj)
key.fileobj.close()
continue
if not msg: continue
file_dic = pickle.loads(msg)
obj_dic[key.fileobj] = file_dic
obj_dic[key.fileobj]['sendsize'] = 0
if obj_dic[key.fileobj]['action'] == 'download':
ser.modify(key.fileobj, selectors.EVENT_WRITE, download)
download(key.fileobj)
continue
handle = key.data
handle(key.fileobj)
#client
import socket
import os
import pickle
def cmd_handle(cmd=''):
if cmd == '': return False
if cmd.split()[0] not in cmd_dic or len(cmd.split()) < 2:
print('命令错误!')
return False
else:
filename = cmd.split()[1]
if os.path.exists(filename) and cmd.split()[0] == 'upload':
filesize = os.path.getsize(filename)
elif cmd.split()[0] == 'download':
filesize = 0
else:
print('命令错误!')
return False
file_dic = {'filename': os.path.basename(filename),
'size': filesize,
'action': cmd.split()[0]
}
c.send(pickle.dumps(file_dic))
return cmd.split()[0]
def upload(filename):
with open(filename, 'rb') as f:
total_size = 0
for i in f:
c.sendall(i)
total_size += len(i)
print('上传成功!')
def download(filename):
file_info = pickle.loads(c.recv(1024))
if file_info == 0: print('文件不存在!')
else:
filesize = file_info
total_size = 0
download_path = os.path.join('E:/shit/', filename)
with open(download_path, 'wb') as f:
while total_size < filesize:
data = c.recv(1024)
f.write(data)
total_size += len(data)
print('下载成功!')
cmd_dic = {'upload': upload, 'download': download}
c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.connect(('127.0.0.1', 8000))
print('连接成功!')
while True:
msg = input('请输入命令:').strip()
cmd_name = cmd_handle(msg)
if not cmd_name: continue
filename = msg.split()[1]
cmd_dic[cmd_name](filename)
