A distributed, fault-tolerant work queue for Firebase
Since you're using Firebase, you'll probably want to create jobs directly from the browser. To do this, setup a new Firebase location reference that will serve as your queue. To start, you may want to make sure that this location is writable by everyone but only readable by your server (i.e. worker) process.
In this example, we'll assume that your queue is located at
https://my-firebase.firebaseio.com/myQueue. In your client code, you'll want to create a new child of the
pendingJobs child of that location reference, like so:
var jobs = '';jobspush my: 'job' ;
That's it. You've now pushed a job onto the queue for a worker to process sometime later.
It should be noted that workers pull from the beginning of the queue (FIFO). If you have jobs of varying importance you can use a priority to run some jobs before others.
jobspushsetWithPriority important: 'job' 0;jobspushsetWithPriority less: 'important' 100;
Firework also provides an API for creating jobs from within your server process. A
Firework.Queue (see below) has a
push method for this purpose.
var queue = require'firework'createQueue'';queuepush my: 'job' ;
Important: The following job property names are reserved:
The easiest way to start processing the jobs on a Firework queue is to use the
$ firework create-worker.js -w 5
-w argument specifies the number of workers to use. When a worker has an unrecoverable error it is removed from the worker pool and replaced with a new one.
create-worker.js module that you pass to
firework should export a function that is used to create new workers. To process jobs from the queue we pushed onto in the Creating Jobs section above, our
create-worker.js file could look something like this:
var Firework = require'firework';var queue = FireworkcreateQueue'';return FireworkcreateWorkerqueue// process the given job.// call the callback when you're done, optionally with// any error that was encountered while doing the work.callbackerror;;;
When Firework is done processing a job it stores the job along with some metadata in the
startedJobs child location of your queue. Thus,
pendingJobs contains a list of all jobs that still need to be done and
startedJobs contains a list of all jobs that you have attempted.
If a job fails, it will have
_error properties. You can use the
_error property to determine the reason of the failure. Once you've fixed the problem, you can retry all jobs that failed using:
$ npm install firework
Please file issues on the issue tracker on GitHub.
To run the tests in node:
$ npm install $ npm test
To run the tests in Chrome:
$ npm install $ npm run test-browser