Writing you first background worker within the Rails can be a bit intimidating. It’s a subject that doesn’t seem to be covered very frequently, and requires you learn about a number of different gems. I’m going to be focusing on the tools that I use most frequently - Redis, Sidekiq and Foreman.
Setting up Sidekiq
Sidekiq is a background processing library for Ruby. It adds some handy methods to our classes which make background processing quite simple.
Sidekiq relies on Redis (a key-value store) to maintain a job queue. That’s the first thing you’ll need to install. If you’re using OSX and homebrew, it should be as simple as running
brew install redis. After it’s installed, follow the instructions to start it up.
After Redis is installed and running, simply add
gem "sidekiq" to your
Gemfile and run
Once the Sidekiq gem is installed, you can start it up by running
bundle exec sidekiq.
The Types of Background Workers
Generally, your workers will be running under one of the following conditions:
- Immediately after an action is performed.
- A certain amount of time after an action is performed.
- Regularly, at set intervals.
Let’s go through and see how to implement each case individually.
Running a worker immediately after an action is performed
Sometimes the user will perform an action on our application that takes a considerable amount of time to complete. Some examples of this might be email, image processing, code highlighting. We don’t want the user to sit there waiting for this action to complete, so we send it off to a background worker.
Sending email is probably the most common example. To have our background worker send email for us, we’ll simply make the following change, adding
delay and dropping
# UserMailer.welcome_email.deliver ## NOT BACKGROUND UserMailer.delay.welcome_email
Run a time consuming class method call on an ActiveRecord object in a background worker:
# Project.do_long_running_action(@project.id) ## NOT BACKGROUND Project.delay.do_long_running_action(@project.id)
For a more complicated task, you can create a worker class that includes the
# app/workers/project_cleanup_worker.rb class ProjectCleanupWorker include Sidekiq::Worker def perform(project_id) # do lots of project cleanup stuff here end end # ProjectCleanupWorker.new.perform(@project.id) ## NOT BACKGROUND ProjectCleanupWorker.perform_async(@project.id)
Important note! Be sure to only pass primitives or simple objects as arguements to the worker, e.g.
.perform_async(@project.id). These arguements must be serialized and placed into the Redis queue, and attempting to serialize an entire ActiveRecord object is inefficient and not likely to work.
Running a worker a set amount of time after an action is performed.
Delaying a worker by a set interval is quite similar to simply backgrounding it. Again, here are the before / after code samples.
# UserMailer.welcome_email.deliver ## NOT BACKGROUND UserMailer.delay_for(5.minutes).welcome_email # OR UserMailer.delay_until(1.week.from_now).welcome_email
ActiveRecord class method:
# Project.do_long_running_action(@project.id) ## NOT BACKGROUND Project.delay_for(10.minutes).do_long_running_action(@project.id) # OR Project.delay_until(2.days.from_now).do_long_running_action(@project.id)
# app/workers/project_cleanup_worker.rb class ProjectCleanupWorker include Sidekiq::Worker def perform(project_id) # do lots of project cleanup stuff here end end # ProjectCleanupWorker.new.perform(@project.id) ## NOT BACKGROUND ProjectCleanupWorker.perform_in(10.minutes, @project.id) # OR ProjectCleanupWorker.perform_at(2.days.from_now, @project.id)
Running a worker regularly, at set intervals
Sometimes, you’ll want to set up a scheduled task to perform an action at a regular interval.
Let’s say, for example, you have a user registration process. If someone starts the registration, but doesn’t complete it, you may want to send them a reminder email.
My (and Heroku’s) recommended approach for this is to create a rake task. If you haven’t created your own Rake task before, it’s pretty simple.
desc "Remind users if they haven't completed registration" task :remind_of_registration => :environment do puts "Reminding users of registration" # ... puts "done." end
:environment in the task definition is important! It loads your Rails environment for the Rake task.
For easier testing, you may want to create a separate object to complete the task for you. Despite the fact that this particular type of scheduled task has nothing to do with Sidekiq, I usually still put objects for completing tasks into the
app/workers directory and name them
That setup would look something like this:
desc "Remind users if they haven't completed registration" task :remind_of_registration => :environment do puts "Reminding users of registration" RegistrationReminderWorker.new.perform puts "done." end
class RegistrationReminderWorker def perform # ... end end
At this point, you can call
rake remind_of_registration from the command line to run this worker—so, that’s what we need to run at some fixed interval.
If you’re running on Heroku, you can use the Scheduler addon, which is free and very easy to use. The only downside is that there isn’t a ton of flexibility for the intervals that you can choose.
If you’re running on a VPS or something like that, you’ll need to do a bit more work. You will need to set up something similar to cron. There’s a gem called whenever that can handle this.
Firing it up - Foreman
If you want to get started quickly, you can simply run
bundle exec sidekiq to start
I like to use the Foreman gem to start my server and all of the background processes. There’s a Foreman Railscasts episode to get you started.
After you’ve got the Foreman gem installed, you’ll want to create two files in the root of your application:
Procfile is essentially a list of instructions for starting up your application and workers. Heroku also uses the
Procfile to start everything up, so after you’ve got this set up it will know how to start your workers.
Here’s a basic
web: bundle exec puma -p $PORT -e $RACK_ENV # Command to start your server worker: bundle exec sidekiq # Command to start sidekiq
Along with that, you’ll want an
RACK_ENV=development PORT=3000 [any other environment variables you want to define]
When you’ve got these files in place, simply run
foreman start, and your server will be started on port 3000 in development, and sidekiq will be started as well!
It’s worth noting that you should not commit
.env into version control, so you’ll want to add it to your
.gitignore file. Instead, you can check in a file called
.env.sample, which can contain the names of the environment variables without any sensitive information.
.env file happens to not contain any sensitive information, but
.env is a great place to store things such as API keys/secrets.
Thanks for reading!
If you have any questions or comments tweet me @bolandrm!
In my next post, a follow up to this one, I’m going to be explaining how to implement undo functionality using Sidekiq background workers. If that sounds interesting to you, be sure to subscribe to my mailing list so you don’t miss it!