Cool Way to Control Retry with ActiveJob

ActiveJob brings benefits but it has just simple features yet.

So, I've made a module with the APIs published officially to add some features of controlling retry.

An Issue about Retry

ActiveJob doesn't have enough features about retrying jobs as of today. It only provides theretry_job*1 which enqueues itself again with some options.

An naive sample is here:

class RetryJob < ApplicationJob
  queue_as :default

  rescue_from(StandardError) do
    retry_job(wait: 5.minutes)
  end

  def perform(*args)
    # Do something later
  end
end

Is seems okay. When a job fails for some reason, the job is enqueued and performed in 5 minutes.

However, what if you want to limit the number of retry times? On the above sample, if a job never succeeds, it retries forever. There is, unfortunately, no way to control it with ActiveJob on default settings.

Of course, you can find some gems like ActiveJob::Retry, but there is no dominant gems in this field yet. As far as I can make out, ActiveJob::Retry is the gem gathering stars the most in Github though, it is not enough sophisticated to use in production.

This is an alpha library in active development, so the API may change.

isaacseymour/activejob-retry

Plus, I feel the gem is kinda thick.

What We Really Want to Do

I think it's gonna be okay if we can set the limit number of retires and find out the number of attempts and whether retry count is exceeded or not.

So, what we want to do are:

• Setting the number of retry limit
• Finding out the attempt number
• Checking whether the retry limit is exceeded or not

Like this:

class LimitedRetryJob < ApplicationJob
  queue_as :default
  retry_limit 5

  rescue_from(StandardError) do |exception|
    raise exception if retry_limit_exceeded?
    retry_job(wait: attempt_number**2)
  end

  def perform(*args)
    # Do something later
  end
end

Let's implement these methods.

How to Implement it

To tell you the truth, the official document tells us a great idea of that. Overriding the serialize and the deserialize enables us to carry over instance variables which contain serializable objects.

Now, we can implement the above idea like this:

class ApplicationJob < ActiveJob::Base
  DEFAULT_RETRY_LIMIT = 5

  attr_reader :attempt_number

  class << self
    def retry_limit(retry_limit)
      @retry_limit = retry_limit
    end

    def load_retry_limit
      @retry_limit || DEFAULT_RETRY_LIMIT
    end
  end

  def serialize
    super.merge("attempt_number" => (@attempt_number || 0) + 1)
  end

  def deserialize(job_data)
    super
    @attempt_number = job_data["attempt_number"]
  end

  private

  def retry_limit
    self.class.load_retry_limit
  end

  def retry_limit_exceeded?
    @attempt_number > retry_limit
  end
end

If you put this ApplicationJob, you will be able to set a limit on each jobs through ApplicationJob.retry_limit, get the number of attempts via ApplicationJob#attempt_number, and check if the retry count exceeds the limit or not calling ApplicationJob#retry_limit_exceeded?.

• ApplicationJob.retry_limit
  • To set the number of retry limit
• ApplicationJob#attempt_number
  • To Find out the attempt number
• ApplicationJob#retry_limit_exceeded?
  • To check whether the retry limit is exceeded or not

Use in production

I’ve made a module based on this idea for use in production since it’s not a good idea to add methods, which not all subclasses require, to the superclass.

necojackarc/active_job_retry_controlable.rb - Gist

The module puts ApplicationJob.retry_limit, ApplicationJob#attempt_number and ApplicationJob#retry_limit_exceeded? on your jobs.

Pros

It only calls APIs declared officially, so it's not easy to brake. Just providing simple methods, you can easily make your own retry logic.

Cons

It's dead simple, so you need to implement your own retry features. It never enables jobs to retry themselves automatically.

*1:Apparently, you can use the enqueue instead of the retry_job. Both look identical. ref: ActiveJob::Enqueuing