MoleQueue

It is increasingly important for desktop applications to be able to easily execute codes, feed them input data, and ultimately collect, ingest, and analyze output data. In many scientific domains, it is not feasible to run these codes exclusively on the desktop, and so there is a need to have a simple way of directing the execution of third party codes both locally and on more powerful remote computational resources such as supercomputers, clusters, and cloud instances. The MoleQueue application is an open-source project that addresses this issue and provides integration of heterogeneous computational resources for desktop applications.

MoleQueue is an open-source, cross-platform, system-tray resident desktop application for abstracting, managing, and coordinating the execution of tasks both locally and on remote computational resources. It is built and tested on Linux, Mac OS X, and Windows, with nightly binaries currently available for Mac OS X and Windows. Users can set up local and remote queues that describe where the task will be executed. Each queue can have programs, with templates to facilitate the execution of the program. Input files can be staged, and output files collected using a standard interface.

The MoleQueue application is written in C++ using the Qt framework. It can execute programs directly on the local machine, and uses SSH to communicate with remote batch scheduling systems with support for Open Grid Scheduler (formerly Sun Grid Engine) and PBS. The backend communication is abstracted, and support is currently being added for UIT (a SOAP protocol for communicating with military HPC resources using ezHPC).

For desktop clients wishing to run jobs, there are a number of options available for submitting, querying, and retrieving job results. The MoleQueue application starts a local server listening on a named local socket (respecting standard user file permissions), which uses the JSON-RPC 2.0 specification to communicate. This can be used directly or through a C++ Qt interface that offers a standard signal/slot interface for GUI applications. There is a pure Python client written using ZeroMQ over local sockets, which can also be used from applications already making use of ZeroMQ. As local sockets and JSON are both widely-supported in a large array of languages, adding new code to your application is not difficult and the available methods are detailed on the relevant wiki pages.

Users are presented with a minimal Qt-based user interface where they can configure queues and programs, and inspect the status of the queue(s). Logging is provided so that any problems can be examined and diagnosed, and right-client context menu integration of other desktop clients can be used to analyze results. The MoleQueue application is evolving rapidly, and being used in several projects being developed at Kitware for effectively integrating simulation and calculation codes with pre- and post-processing applications used in the scientific discovery process.