Background Jobs

As probably most web applications out there, we need to run jobs in the background. Some jobs just run periodically (every hour, or first of each month) while others are spawned by requests that need to perform long computation (generating pdfs, or processing credit cards) allowing the client to see a response immediately and check back later for the result of the long computation. In the Rails world, the long requests are especially painful as each rails process is single threaded (i.e. one request at a time). So, having the ability to offload long requests allows the rails process to handle other requests while the long computation is done in the background. For a while we tried the completely rewritten backgroundrb 1.x. Kudos to Hemant for the nice work. However, I did not feel backgroundrb was the right solution for us for the following reasons,

• We have multiple mongrels spread over multiple machines for redundancy. In order for one mongrel on one machine to start a job and for another mongrel on another machine to get the result of the job, all mongrels need to talk to a single backgroundrb server.
• We use monit to watch mongrels. Since the backgroundrb server starts the workers, it is not possible for monit to watch those workers, nor is it clear how to restart a worker if it were to die.
• The backgroundrb jobs are not persistent. So when the backgroundrb server goes down the whole state is lost.

Searching around, I found background-fu by Jacek Becela (Polish roots? :)). In a nutshell, background-fu uses a simple active record model to persist jobs: the app creates jobs, daemon executes those jobs, the app queries the status of jobs (essentially home grown queue). Out of the box, background-fu solves the problems we had with backgroundrb. Namely, since all communication is done via the database, any mongrel on any machine has access to jobs. Monit can start and monitor the background-fu daemon. The jobs are persisted in the database. Great! Having the right underlying model in place, we went ahead and extended background-fu with the following,

• Added lock_version to jobs which allows me to run multiple daemons and ensure that a job is executed by a one daemon.
• Added ability to execute a job at a particular time in the future.
• Added support for periodic jobs with interval and cron triggers (borrowed from backgroundrb).
• Added integration with exception notifier.
• Added garbage collection of old finished jobs.

The result is the following, job = Job.enqueue!(ExampleWorker, :add, 1, 2) do |job| job.next_run_at = Time.now.tomorrow end job = Job.enqueue!(ExampleWorker, :add, 1, 2) do |job| job.interval_trigger = 60 # seconds end job = Job.enqueue!(ExampleWorker, :add, 1, 2) do |job| job.next_run_at = Time.now.tomorrow job.cron_trigger = "0 30 1,2 * * * *" # you can actually plug-in your own trigger as above is just syntax sugar for, # # job.trigger(CronTrigger, "0 30 1,2 * * * *") end Normally, we use a migration to add / remove periodic background jobs. But as you see above, you can do it programmatically as well.