Fix kqueue poller asyncore in PyBitmessage
New here? Learn about Bountify and follow @bountify to get notified of new bounties! x

PyBitmessage uses a modified asyncore module with multiple pollers. The kqueue poller doesn't work correctly, I probably broke it when I was optimising it. It seems to connect but once a connection is established it doesn't correctly detect/handle sending/receiving.

Source: https://github.com/Bitmessage/PyBitmessage/blob/v0.6/src/network/asyncore_pollchoose.py#L360

A solution would be a patch that makes the kqueue poller work and I can successfully test it on OSX and FreeBSD (you probably have to test it on just one of these, the same code should work on both).

The patch needs to be a pull request against the v0.6 branch. When creating a pull request, please follow the contribution guidelines, of most importance is that:

  • the pull request is against the v0.6 branch
  • all the commits in the pull request are PGP signed
  • the pull request can be merged using a fast-forward merge (this isn't actually that important anymore)
I finally figured out the source of the problem. When adding the filter, I was bitwise OR-ing KQ_FILTER_READ and KQ_FILTER WRITE, as that's what you can do with all the other pollers. However, it doesn't work as KQ_FILTER_READ is -1 and KQ_FILTER_WRITE is -2 (at least on OSX) and if you OR them, you'll get KQ_FILTER_WRITE, so KQ_FILTER_READ ends up only being triggered on failed connects (on successful connect, the socket turns writable but not readable as normally, the BM client is supposed to start sending data before the server starts sending). So it would connect, send the version packet but never read the reply, and the connection would timeout. With other pollers, you can OR read and write filters both on input and output, with kqueue they are two separate events.
PeterSurda 19 days ago
I'll award the bounty to bbb because at least he bothered to post.
PeterSurda 19 days ago
awarded to bbb

Crowdsource coding tasks.

5 Solutions

Winning solution

I am sure those billions of freeBSD users will jump at a $10 lure. They are lining up already.


`
'''
NOT TESTED

just for documentation

I [Mr Surda] finally figured out the source of the problem [myself] .

When adding/updating the filter, I was bitwise OR-ing KQ_FILTER_READ and KQ_FILTER WRITE,
as that's what you can do with all the other pollers.

However, it doesn't work as

KQ_FILTER_READ is -1

and

KQ_FILTER_WRITE is -2

(at least on OSX)

and if you OR them, you'll get KQ_FILTER_WRITE,
so KQ_FILTER_READ ends up only being triggered on failed connects
(on successful connect, the socket turns writable but not readable as normally,
the BM client is supposed to start sending data before the server starts sending).

So it would connect, send the version packet but never read the reply,
and the connection would timeout.

With other pollers, you can OR read and write filters both on input and output,
with kqueue they are two separate events.

I still need to clean it up but it's working now on my OSX VM, after many hours of debugging. It probably works on FreeBSD as well.

To be fair, the code below posted by an anonymous contributor probably would have worked.
It's sad that it wasn't posted on bountify as it would have been eligible
even though I probably wouldn't have understood why it works.

The good news is that it means 0.6.3 is close, one remaining major issue is to test dandelion but it's more of a design review as debugging.

Peter Surda
Bitmessage core developer


The IOError you are ignoring has important information on the failure (this is a very bad habit of yours through out bitmessage).
Most likely it is informing that the filter is invalid or doesn't apply to socket file descriptors.

Filters are not flags and can't be OR'ed together - you have to create an event for each filter.

-
It helps to refer to correct version of the source code.
select.kqueue() raises IOError in Python 2.7 not OSError as in 3.6.

Code below has been updated

I think your generator is broken but the error handling is better than mine. It still doesn't work but I think I see now the incorrect error handling that I was doing earlier.

Peter Surda
Bitmessage core developer

'''
class KqueueError(OSError):
pass

def kevent_gen(it):
flags = select.KQ_EV_ADD | select.KQ_EV_ENABLE | select.KQ_EV_ONESHOT
for fd, obj in it:
if obj.readable():
yield select.kevent(fd, filter=select.KQ_FILTER_READ, flags=flags)

    if obj.writable() and not obj.accepting:
        yield select.kevent(fd, filter=select.KQ_FILTER_WRITE, flags=flags)

def kqueue_poller(timeout=0.0, map=None):
"""A poller which uses kqueue(), BSD specific."""
if map is None:
map = socket_map

if not map:
    current_thread().stop.wait(timeout)
    return

kqueue = None
try:
    kqueue = select.kqueue()

    events = [kevent_gen(map.items())]

    if not events:
        current_thread().stop.wait(timeout)
        return

    events = kqueue.control(events, len(events), timeout)

    if events > 1:
        events = random.sample(events, len(events))

    for event in events:
        obj = map.get(event.ident)
        if obj is None:
            continue
        try:
            if event.filter == select.KQ_FILTER_READ and can_receive():
                #if obj.accepting:
                #    backlog_size = event.data
                #else:
                #    bytes_available_to_read = event.data
                obj.handle_read_event()
            if event.filter == select.KQ_FILTER_WRITE and can_send():
                #write_space_remaining = event.data
                obj.handle_write_event()
            if event.flags & select.KQ_EV_EOF:
                obj.handle_close()
            if event.flags & select.KQ_EV_ERROR:
                raise KqueueError(event.data)
        except socket.error as e:
            if e.args[0] not in _DISCONNECTED:
                obj.handle_error()
            else:
                obj.handle_close()
        except _reraised_exceptions:
            raise
        except:
            obj.handle_error()
except IOError:
    logger.exception('kqueue error')
    raise
except OSError:
    logger.exception('kqueue control error')
    raise
finally:
    if kqueue:
        kqueue.close()`

print("markdown in here sucks")
'''how bloody terrible
imposs to post py code here, damnit


warning to future posters: if your paypal mail address you are being forced to enter is not registered with paypal, you get nothing at all.

Instead, billionaire Bill Gates will get it for his Malaria "charity".

From the FAQ: Payouts occur via Paypal within 48 hours after your solution is selected as the winner. Your default Paypal email address is the login email address you provided at signup. You can specify an alternate email address at which to receive Paypal payouts by visiting your account settings page after signup.
PeterSurda 19 days ago

so far I have not seen a bleedy penny.

days later we got paypal to work manually via "donation mail link". wow. what an ordeal. Of $10 I got $9,31 minus the exchange fees and what not. Less than 7,25 €uro
bbb 14 days ago
Bitcoin is not supported, just so you know.
bbb 14 days ago
View Timeline