Migrating data in Rails using Rails migrations

Put this one in the “should’ve been obvious” column…

For an embarassingly long while, when I had a feature ready for deploy that required production data be modified for a whole bunch of records – for example, downcasing all name attributes – I would write a one-off rake task to update those records, then delete that task.

Not only is this bad code since it doesn’t keep a record of what happened, it’s also tedious because it’s an extra step that must be taken by whoever deploys the code.

Luckily there’s a better way - just put the updating login in a migration!

Example:

class DowncaseName < ActiveRecord::Migration
  def up
    User.where.not(name: nil).find_each do |user|
       user.update!(name: user.name.downcase)
    end
  end
end

This example isn’t great because 1. there’s almost certainly a way to do this with pure SQL without instantiating every user record, and 2. it’s not reversible. But since I’m lazy and since it illustrates the point of the post, I’m leaving it.

count vs size on ActiveRecord select queries

I stumbled on this bit of weirdness when using the SQL query User.select(:id, :name, :email).count. It threw a PG::UndefinedFunction error. Using User.select(:id, :name, :email).size worked instead.

Out of curiosity, I did some digging. According the the ActiveRecord docs,

not all valid select expressions are valid count expressions. The
specifics differ between databases. In invalid cases, an error from
the database is thrown.

I also stumbled on an issue on rails Github issues page about this, and a very helpful comment from rafaelfranca explained that I can also use count(:all).

Object overload! Careful of instantiating objects in Ruby

Wowzers, I just tackled a major memory issue in an app I’ve been developing, and it was nearly all due to rampant over-instantiation of ActiveRecord objects.

Here’s the TL;DR: Use SQL queries whenever possible to grab info from the database, rather than using Ruby code.

Now for the longer version. First, I knew I had a memory issue because my server was crashing and it was showing up the logs saying that memory could not be allocated. But I had to find out what the issue was first.

Oink to the rescue. It’s a handy little Ruby gem that will log how much memory and how many ActiveRecord objects are instantiated for each request.

I installed Oink, ran through a few requests that seemed kinda slow, and lo and behold, I found the problem: One of my requests was generating 1500 objects!

Here’s what was happening. I was looping through an array of IDs, and for each one I was calling

account.users.order(:id).select{ |user| user.id == id }.first

Don’t laugh. I have a baby at home, I was sleep deprived, I don’t know why I did it this way. But yes, this is instantiating all User objects in an account, and it’s doing it every single time it iterates through the loop. Hence, 1500 objects instantiated per request.

This was easily solved by a simple find_by_id with the id from the array.

That took me down to only one instance of each User object, which is still not fantastic. So, I ended up using one of my favorite ARel methods, pluck, to get just the attributes of the users that I need without actually creating the User objects.

That’s the ticket! From 1500 User objects to 0, this should significantly help with speed and memory usage. I ended up combing through the rest of the app and finding a few other spots where object instantiation was a bit out of control, though not nearly as bad as that one.

It became a kind of game - and in the end, I’d say I won.

Thanks to Engine Yard for the great tips. Lots more in their linked post.

Asynchronous processing pitfalls and ActiveRecord locking

My app is humming along, but there are a few sections that have to do a lot of heavy lifting, whether that be image processing, updating several different related objects, or looping through all of a has_many association to manipulate every record.

These types of things are perfect places to push some logic to a background process. In other words, make it asynchronous. So, if a user clicks a button to update several objects, unless the user needs to see those updates completed immediately, then a tool like Sidekiq can help run some expensive logic as a background process and allow the user to continue surfing the site while that process runs.

There are plenty of tutorials about how to set this up, so that’s not what I want to talk about here.

One of the potential pitfalls of processing things asynchronously, particularly if the processing involves updating database records, is multiple records being updated at the same time. For example, one user updates a Project in a background process, and another user updates the same Project somewhere else in the app that doesn’t use a background process. We want to last update to be the one that sticks, but if both updates are happening at the same time, things could get hairy.

This is the dreaded “race conditions” issue, and luckily there’s a pretty simple fix.

On ActiveRecord - and I believe this will work with any SQL database but definitely at least on Postgres and MySQL - I can lock a specific row while editing, then unlock it afterwards, so that each edit takes place in its own little bubble.

Account.transaction do
  acc = Account.lock.find(id)
  acc.balance -= 25
  acc.save!
end

This finds an account and locks it, then updates the balance, saves, and unlocks.

This is really important when dealing with numbers, money, and balances. Let me share a parable to explain:

There is an Account with $100 balance. One user in that account makes a transaction for$75.

A split second later, another user in that account makes a transaction for $90.

What should happen is the second user’s transaction should be rejected for lack of funds.

What actually happens without locking is that for each transaction, the first step is that the server runs account = Account.find(id) to retrieve the account, and since this is happening before either transaction has finished, the balance on the account in both cases is $100. So, after the first user's transaction, the balance is updated to$25 and the account is saved. But on the second user’s transaction, the account object that was pulled from the database still shows a balance of $100, so the server deducts the$90 charge from the balance and saves it, thereby allowing the transaction to go through and setting the saved account balance as $10.

Is your head spinning yet?

When we lock the account per transaction, we avoid this issue by only allowing the account to be edited on transaction at a time.

It’s a cool concept but also can be tricky to wrap your head around. Here’s a blog post I borrowed heavily from for my post. Or check out the Rails API docs.

Turbocharged custom Rails / ActiveRecord validations

My code just got a lot less janky when dealing with start and end times. Before, when I wanted to validate that a user-submitted end time came after a start time, I was doing something like this:

validate :start_time_is_before_end_time
def start_time_is_before_end_time
    unless start_time.to_i < end_time.to_i
        errors.add(:end_time, 'must come after start time')
    end
end

Converting the values to epoch time and comparing isn’t terrible, but I just found a cool way to extract this out to a custom validator to make it simpler and more readable:

# app/models/model.rb
require "active_model/sequential_validator"
validates :start_time, :end_time, sequential: true

# lib/active_model/sequential_validator.rb
class SequentialValidator < ActiveModel::Validator
    def validate record
        values = options[:attributes].map do |attribute|
           record.send(attribute)
        end.compact
        if values.sort != values
            record.errors.add options[:attributes].last, "cannot be before #{options[:attributes].first}"
        end
    end
end

True, the actual validator code is not as easy to read I don’t think, but it takes some of the burden out of the model, and it also allows reusing the sequential validation in other models.

(Thanks to botandrose and the calagator project for this cool piece of code.)

More readable for ActiveRecord statements using Placeholder Conditions

Here’s another readability hack for Ruby ActiveRecord code. It’s called “Placeholder Conditions.”

This:

where("start_time < :time", time: Time.now)

works the same as this

where("start_time < ?", Time.now)

The second is more terse, but if you are inserting a lot of values into the WHERE statement, then the first is nice for readability.

Source: Ruby on Rails Guides

String conditionals on ActiveRecord callbacks

I’ve got a model for push notifications that uses Parse.com, and it sends the data to Parse on a callback:

before_create :send_to_parse

The problem is, I have a whole bunch of specs that create push notifications, and I don’t want any sent to Parse during testing. Now, I don’t have the Parse credentials accessible within the test environment, but that throws an error when it tries to send to Parse. So, what I really want is to just send to Parse when we’re not testing.

Turns out this is dead simple, albeit a little strange at first:

before_create :send_to_parse, unless: "Rails.env.test?"

I’ve used conditionals with symbolized method names before, but the string bit threw me off. However, it’s simple valid Ruby code, you can put it in a string and it will be evaluated as such.

This is straight from the docs, and I thought it was super cool.

ActiveRecord querying on Postgres array column

On a Rails project I’m working on, we have a model that has a db column of array datatype. This is one of those Postgres special datatypes that isn’t present on many other SQL databases, and there’s even some contention about whether it should be an option at all.

At any rate, this is deeply ingrained in our project currently, so I don’t want to go reinventing the wheel at this point and convert this particular column to a separate model and create another has_many/belongs_to relationship. All I want is a way to find all of the objects in this model that have an empty array in that column.

Turns out there’s no built in ActiveRecord finder for this, but the raw SQL isn’t too terrible:

Model.where("NULL = ALL(array_column_name)")

Hat Tip: https://coderwall.com/p/sud9ja/rails-4-the-postgresql-array-data-type

Unique objects from a three way join

I have a model that is essentially a three-way join of three models. Let’s call this model Widget. It has a user_id, account_id, and place_id. I want to make sure each of those widget objects is unique in some way - in other words, I don’t want any widget object with the same user_id, account_id, and place_id.

I should probably validate for uniqueness, but only within the confines of what is in the other columns. Here’s where uniqueness scope comes in handy:

validates_uniqueness_of :user_id, scope: [:account_id, :place_id]

Now, if we try to create a Widget with the same user_id, account_id, AND place_id as one already in the system, ActiveRecord will yell at us. But if any of those values are different from what’s already in the db, we’re good to go.

If this was going to be user-facing, I’d probably also want to add a custom error message to this, otherwise it would say something like ‘user_id has been taken’, which isn’t ideal. However, these Widget objects will be created in the background, not explicitly by users, so we shouldn’t ever see these error messages.

Now let’s say we want to control for what might already be in the db. Suppose that we didn’t find out this was an issue until a bunch of people already created duplicate Widgets. Sure, we could create a rake task to comb the db and delete the duplicates, and that would probably be a good idea at some point if we truly don’t need them. But what if for some reason we wanted duplicates to save to the db, but on certain pages only wanted to display unique Widget objects?

Running Widget.all.uniq doesn’t do anything for us, because each widget object has a unique id/primary key. So, assuming we don’t need the primary key on the page we’re displaying these widgets (and since this is essentially just a three-way join with no other attributes, I think that’s a fair assumption), we can pull all other attributes besides the primary id for all the widget objects using ActiveRecord’s select:

Widget.select(:user_id, :account_id, :place_id)

This will give us all of the widget objects in the system, but with nil values for any attributes we didn’t specify - namely, the id field. Now, since we don’t have the unique id to deal with, we can easily filter duplicates using uniq:

Widget.select(:user_id, :account_id, :place_id).uniq

Just to make it cleaner, I would probably make that a scope on Widget so I could give it a name and have it make more sense for other coders on the project.

class Widget < ActiveRecord::Base
  scope :unique, -> { select(:user_id, :account_id, :place_id).uniq }
  # other stuff in Widget class
end

Widget.unique

Bingo bango bongo.

Speed up your app's delegation skills using `:includes`

My last post talked about delegating methods to a parent model using the delegate module. I want to talk about performance issues you could experience doing this, and a simple way to fix them and speed them up.

When querying databases, there's something called "eager loading" that essentially is a way for the code to pull associated objects that may be needed at the same time as pulling the main object, thereby saving a separate DB call and hopefully saving some time. I won't pretend to know exactly how this works, I just know that it's supposed to be faster to do it this way.

In my previous example, I had a Course with an associated CourseName, because I wanted to be able to create multiple Courses with the same name but different times and dates. I delegated the :name method to CourseName, so that instead of calling course.course_name.name, I can simply call course.name to achieve the same results. However, when I do that, it has to load the CourseName object on the fly, and that could result in some additional sluggishness. (Apparently, this is hotly debated whether these N+1 queries are faster or slower.)

This is where eager loading is helpful.

When you are finding a course with Course.find(params[:course_id]), throw in one additional thing to make this eager load the associated course name:

  Course.includes(:course_name).find(params[:course_id])

This helps remove the N+1 query and will hopefully help my performance. Now let's say I have a School object which has_many :courses. When I call school.courses, if I'm going to be displaying the course name as well, I'm into N+1 query land again. The easiest way to fix this is by adding it directly into the association in the School model:

class School < ActiveRecord::Base
  has_many :courses, -> { includes(:course_name) }
end

Whambo.

Hat Tip: http://blog.arkency.com/2013/12/rails4-preloading/ for the help in understanding preloading