zmq work manager fails against recent versions of zeromq

[synapticarbors]

I recently cloned a fresh version of westpa and was testing it against Anaconda 2.1.0 (release before the most recent one), and noted that the zmq work manager wasn't an available option for w_run. It looks like there were some core changes to the pyzmq API which are breaking the work manager and the unit tests are failing, with:

  File "/Users/lev/Documents/Projects/github/westpa/lib/wwmgr/work_managers/zeromq/__init__.py", line 44, in <module>
    from zmq.core.message import Message as Frame
ImportError: No module named core.message

[synapticarbors]

Documenting pyzmq API incompatibilities:

• zmq.core no longer exists; from zmq.core.message import Frame should be changed to from zmq.backend import Frame.
• zmq.HWM no longer exists; The high water mark appears to be a property of the socket zmq.Socket.hwm, so instead of result_socket.setsockopt(zmq.HWM, 1), I think it would simply be result_socket.hwm = 1.

Making these two changes allows a bunch of the tests to successfully run, but I'm still unable to pass all of them. Currently hanging on the test external shutdown signal causes shutdown. There appears to be a decent amount of work to do to fix everything up and there's an open question about how and if we should support multiple pyzmq versions. The one we support now is fairly old. Since pyzmq is core technology in the scipy stack (as it forms the basis of IPython), I think we shouldn't necessarily attempt to continue to support really old versions since there is a good chance that people will be staying fairly current.

[mczwier]

Agreed. Especially given that I've waited on advances in numpy and h5py to do things I've wanted to do, and how very easy it is to use Anaconda to keep numpy, scipy, and h5py up-to-date, I think keeping current with major anaconda releases (those announced on their web site) is probably a decent benchmark.

[mczwier]

I've assigned this to myself, though PRs/patches are welcome. Until I get a chance to look seriously at this (within a month), any further details about which tests are failing (or hanging) would be helpful.

[synapticarbors]

@mczwier Any update on this? I saw you had a branch where you were working on this a while back, but it looks like you haven't had a chance to work on this recently.

[astatide]

I've been using this on basically every cluster I have access to, and I haven't encountered a single bug as regards data I/O.

About the only thing I've run into is it seems to need TCP, and it needs a longer timeout period than the old work manager.

Adam

[synapticarbors]

@ajoshpratt when you say that you've been using "this", do you mean @mczwier's updated branch, or the current zeromq implementation that's in the master branch?

[astatide]

Ah, yes, sorry. The updated branch on @mczwier's fork. Not the current implementation in master.

[synapticarbors]

Great. Are there plans to merge the refactored zmq work manager soon? Also, has there been any recent discussions of moving forward with some of the goals of packaging westpa using conda, setting up continuous integration, etc?

[astatide]

Nothing in that vein, no. I've been prepping some significant changes on other parts (mostly the toolset) on the DEVELOPMENT branch. On that note, it would probably be useful for any interested parties to be involved in a discussion on the changes I've made there, if only so everyone is on board. They're all necessary; it's just a matter of whether the implementation is good, construction-wise.

It might also be a good time to discuss the wwmgr fork with @mczwier.