Ryan Boland

Writing you first background worker within the Rails can be a bit intimidating. It’s a subject that doesn’t seem to be covered very frequently, and requires you learn about a number of different gems. I’m going to be focusing on the tools that I use most frequently - Redis, Sidekiq and Foreman.

Setting up Sidekiq

Sidekiq is a background processing library for Ruby. It adds some handy methods to our classes which make background processing quite simple.

Sidekiq relies on Redis (a key-value store) to maintain a job queue. That’s the first thing you’ll need to install. If you’re using OSX and homebrew, it should be as simple as running brew install redis. After it’s installed, follow the instructions to start it up.

After Redis is installed and running, simply add gem "sidekiq" to your Gemfile and run bundle.

Once the Sidekiq gem is installed, you can start it up by running bundle exec sidekiq.

The Types of Background Workers

Generally, your workers will be running under one of the following conditions:

• Immediately after an action is performed.
• A certain amount of time after an action is performed.
• Regularly, at set intervals.

Let’s go through and see how to implement each case individually.

Running a worker immediately after an action is performed

Sometimes the user will perform an action on our application that takes a considerable amount of time to complete. Some examples of this might be email, image processing, code highlighting. We don’t want the user to sit there waiting for this action to complete, so we send it off to a background worker.

Sending email is probably the most common example. To have our background worker send email for us, we’ll simply make the following change, adding delay and dropping deliver:

# UserMailer.welcome_email.deliver  ## NOT BACKGROUND
UserMailer.delay.welcome_email

Run a time consuming class method call on an ActiveRecord object in a background worker:

# Project.do_long_running_action(@project.id)  ## NOT BACKGROUND
Project.delay.do_long_running_action(@project.id)

For a more complicated task, you can create a worker class that includes the Sidekiq::Worker module:

# app/workers/project_cleanup_worker.rb
class ProjectCleanupWorker
  include Sidekiq::Worker

  def perform(project_id)
    # do lots of project cleanup stuff here
  end
end

# ProjectCleanupWorker.new.perform(@project.id)  ## NOT BACKGROUND
ProjectCleanupWorker.perform_async(@project.id)

Important note! Be sure to only pass primitives or simple objects as arguements to the worker, e.g. .perform_async(@project.id). These arguements must be serialized and placed into the Redis queue, and attempting to serialize an entire ActiveRecord object is inefficient and not likely to work.

Running a worker a set amount of time after an action is performed.

Delaying a worker by a set interval is quite similar to simply backgrounding it. Again, here are the before / after code samples.

Email:

# UserMailer.welcome_email.deliver  ## NOT BACKGROUND

UserMailer.delay_for(5.minutes).welcome_email
# OR
UserMailer.delay_until(1.week.from_now).welcome_email

ActiveRecord class method:

# Project.do_long_running_action(@project.id)  ## NOT BACKGROUND

Project.delay_for(10.minutes).do_long_running_action(@project.id)
# OR
Project.delay_until(2.days.from_now).do_long_running_action(@project.id)

Sidekiq Worker:

# app/workers/project_cleanup_worker.rb
class ProjectCleanupWorker
  include Sidekiq::Worker

  def perform(project_id)
    # do lots of project cleanup stuff here
  end
end

# ProjectCleanupWorker.new.perform(@project.id)  ## NOT BACKGROUND

ProjectCleanupWorker.perform_in(10.minutes, @project.id)
# OR
ProjectCleanupWorker.perform_at(2.days.from_now, @project.id)

Running a worker regularly, at set intervals

Sometimes, you’ll want to set up a scheduled task to perform an action at a regular interval.

Let’s say, for example, you have a user registration process. If someone starts the registration, but doesn’t complete it, you may want to send them a reminder email.

My (and Heroku’s) recommended approach for this is to create a rake task. If you haven’t created your own Rake task before, it’s pretty simple.

desc "Remind users if they haven't completed registration"
task :remind_of_registration => :environment do
  puts "Reminding users of registration"
  # ...
  puts "done."
end

Note! that :environment in the task definition is important! It loads your Rails environment for the Rake task.

For easier testing, you may want to create a separate object to complete the task for you. Despite the fact that this particular type of scheduled task has nothing to do with Sidekiq, I usually still put objects for completing tasks into the app/workers directory and name them [...]_worker.rb.

That setup would look something like this:

desc "Remind users if they haven't completed registration"
task :remind_of_registration => :environment do
  puts "Reminding users of registration"
  RegistrationReminderWorker.new.perform
  puts "done."
end

class RegistrationReminderWorker
  def perform
    # ...
  end
end

At this point, you can call rake remind_of_registration from the command line to run this worker—so, that’s what we need to run at some fixed interval.

If you’re running on Heroku, you can use the Scheduler addon, which is free and very easy to use. The only downside is that there isn’t a ton of flexibility for the intervals that you can choose.

If you’re running on a VPS or something like that, you’ll need to do a bit more work. You will need to set up something similar to cron. There’s a gem called whenever that can handle this.

Firing it up - Foreman

If you want to get started quickly, you can simply run bundle exec sidekiq to start

I like to use the Foreman gem to start my server and all of the background processes. There’s a Foreman Railscasts episode to get you started.

After you’ve got the Foreman gem installed, you’ll want to create two files in the root of your application: Procfile and .env.

The Procfile is essentially a list of instructions for starting up your application and workers. Heroku also uses the Procfile to start everything up, so after you’ve got this set up it will know how to start your workers.

Here’s a basic Procfile example:

web: bundle exec puma -p $PORT -e $RACK_ENV   # Command to start your server
worker: bundle exec sidekiq                   # Command to start sidekiq

Along with that, you’ll want an .env:

RACK_ENV=development
PORT=3000
[any other environment variables you want to define]

When you’ve got these files in place, simply run foreman start, and your server will be started on port 3000 in development, and sidekiq will be started as well!

It’s worth noting that you should not commit .env into version control, so you’ll want to add it to your .gitignore file. Instead, you can check in a file called .env.sample, which can contain the names of the environment variables without any sensitive information.

Our example .env file happens to not contain any sensitive information, but .env is a great place to store things such as API keys/secrets.

Recommended viewing:

• Railscasts Sidekiq Episode

Thanks for reading!

If you have any questions or comments tweet me @bolandrm!

In my next post, a follow up to this one, I’m going to be explaining how to implement undo functionality using Sidekiq background workers. If that sounds interesting to you, be sure to subscribe to my mailing list so you don’t miss it!