Skipping an empty poll set means skipping a wait
time of pollerTime microseconds. If most of the
poll sets are skipped then this loop will be
tighter than expected which could lead to higher
than expected CPU usage. Extending the
nextPollerTime ensures that this loop always
takes about the same time to execute.