[WIP] Distributed processing #60
Conversation
…try_splits is not distributed yet
|
@nippoo @rossant I have now implemented distributed processing using IPython.parallel in this branch. It won't be highly efficient yet because I did some slightly lame things in order to get it working at all, which should be possible to make more efficient. Also, try_splits is not distributed yet, and so this isn't at all faster. However, it should hopefully work for normal iterations now, and I think even with the inefficiencies for a large data set it will be much faster when distributed across multiple machines. Could you try this out? The ipython notebook in dev can be used, you need to start up an appropriate engine before running it. It may also be the case that KK2 needs to be installed on each machine it runs on? Not sure how IPython handles that. |
|
thanks @thesamovar we'll have a look! yeah KK2 probably needs to be installed on every machine first. I'm wondering if we can use docker to simplify deployement on many machines -- there should not be a significant performance hit, but that remains to be checked |
[ci skip[
|
@rossant there's no need to build images or deploy them: assuming the hardware is the same, just share a folder in the PATH over NFS or similar. This is exactly how I share KK1/SD2 over our servers. Legion and every computing cluster will run the same software on all nodes and share a directory over NFS. It's simple! |
|
@nippoo neat! |
|
@nippoo wait, this will only work for our code, not for conda dependencies..? |
|
It works for everything - just make sure the shared miniconda installation / venv is in the PATH... |
Conflicts: dev/testing.py klustakwik2/clustering.py
No description provided.