Completion-based event queues – Understanding OS-Backed Event Queues, System Calls, and Cross-Platform Abstractions

IOCP stands for input/output completion port. This is a completion-based event queue. This type of queue notifies you when events are completed. An example of this is when data has been read into a buffer.

The following is a basic breakdown of what happens in this type of event queue:

1. We create an event queue by calling the syscall CreateIoCompletionPort.
2. We create a buffer and ask the OS to give us a handle to a socket.
3. We register an interest in Read events on this socket with another syscall, but this time we also pass in the buffer we created in (step 2) , which the data will be read to.
4. Next, we call GetQueuedCompletionStatusEx, which will block until an event has been completed.
5. Our thread is unblocked and our buffer is now filled with the data we’re interested in.

Figure 3.2 – A simplified view of the IOCP flow

epoll, kqueue, and IOCP

epoll is the Linux way of implementing an event queue. In terms of functionality, it has a lot in common with kqueue. The advantage of using epoll over other similar methods on Linux, such as select or poll, is that epoll was designed to work very efficiently with a large number of events.

kqueue is the macOS way of implementing an event queue (which originated from BSD) in operating systems such as FreeBSD and OpenBSD. In terms of high-level functionality, it’s similar to epoll in concept but different in actual use.

IOCP is the way Windows handle this type of event queue. In Windows, a completion port will let you know when an event has been completed. Now, this might sound like a minor difference, but it’s not. This is especially apparent when you want to write a library since abstracting over both means you’ll either have to model IOCP as readiness-based or model epoll/kqueue as completion-based.

Lending out a buffer to the OS also provides some challenges since it’s very important that this buffer stays untouched while waiting for an operation to return.

Windows	Linux	macOS
IOCP	epoll	kqueue
Completion based	Readiness based	Readiness based

Table 3.1 – Different platforms and event queues