Migrating data in Rails using Rails migrations

Put this one in the “should’ve been obvious” column…

For an embarassingly long while, when I had a feature ready for deploy that required production data be modified for a whole bunch of records – for example, downcasing all name attributes – I would write a one-off rake task to update those records, then delete that task.

Not only is this bad code since it doesn’t keep a record of what happened, it’s also tedious because it’s an extra step that must be taken by whoever deploys the code.

Luckily there’s a better way - just put the updating login in a migration!

Example:

class DowncaseName < ActiveRecord::Migration
  def up
    User.where.not(name: nil).find_each do |user|
       user.update!(name: user.name.downcase)
    end
  end
end

This example isn’t great because 1. there’s almost certainly a way to do this with pure SQL without instantiating every user record, and 2. it’s not reversible. But since I’m lazy and since it illustrates the point of the post, I’m leaving it.

No return value in rails console

A quick tip when working in rails console. If you're testing commands to run and don't want the console to output the return value, just append a semicolon to your ruby code.

  User.all
  # => [#<user:0x007fd72a408630> {:id => 1, :name => "Mr. T", ...}, tons and tons of other records]
  User.all;
  # => nothing!

OK, so that's a contrived example, but this really comes in handy

• when creating records, or 
• if you need to update several records in one shot and don't want to see the return values, or
• POSTing to an external API and avoiding all the cruft that comes in the response

Classy Ruby enumerating

Given two users:

User.create(first_name: "Joe", last_name: "Louis")
#=> User id: 1, first_name: "Joe", last_name: "Louis"

User.create(first_name: "Muhammed", last_name: "Ali")
#=> User id: 2, first_name: "Muhammed", last_name: "Ali"

Say you want to get an array of arrays, with each individual array containing two elements: the first and last name, and the id. You could do something like this, which is perfectly valid but also gross to read:

User.pluck(:id, :first_name, :last_name).map do |user|
  id = user[0]
  first = user[1]
  last = user[2]
  ["#{first} #{last}", id]
end

However, it’s much more readable to use tuple syntax:

User.pluck(:id, :first_name, :last_name).map do |(id, first, last)|
  ["#{first} #{last}", id]
end
#=> [
      ["Joe Louis", 1],
      ["Muhammed Ali", 2]
    ]

Funky Ruby Class Method Syntax

I’ve run across this syntax a few different times but never grasped how it worked, and I guess never cared enough to look it up. Turns out it’s dead simple.

This:

class Person  
  def self.species
    "Homo Sapien"
  end
end

is equivalent to this:

class Person  
  class << self
    def species
      "Homo Sapien"
    end
  end
end

Thanks to Yehuda Katz for finally clearing that up for me.

Literally the coolest way to build an array

One of Ruby’s literal syntax shorthands, %i essentially loops over each word, converts to a symbol and adds it to an array. For example:

%i(dog cat mouse)
# => [:dog, :cat, :mouse]

Caveat: This syntax was added in Ruby 2.0, so don’t go trying it in any previous version.

Familiarize quickly in new code with this trick

Here’s a quick and nifty little trick that comes in extremely handy in a lot of situations, but particularly when dealing with an unfamiliar codebase, such as at a new job or working with a previously unused external gem.

In the rails console, enter:

ls Model 

and it will show all the class and instance methods, as well as associations and a bunch of other cool stuff for that model.

I’ve begun using this in place of these two:

Model.methods
Model.new.methods

Not only because it’s simpler with less typing but it else ends up being much easier to read.

count vs size on ActiveRecord select queries

I stumbled on this bit of weirdness when using the SQL query User.select(:id, :name, :email).count. It threw a PG::UndefinedFunction error. Using User.select(:id, :name, :email).size worked instead.

Out of curiosity, I did some digging. According the the ActiveRecord docs,

not all valid select expressions are valid count expressions. The
specifics differ between databases. In invalid cases, an error from
the database is thrown.

I also stumbled on an issue on rails Github issues page about this, and a very helpful comment from rafaelfranca explained that I can also use count(:all).

Object overload! Careful of instantiating objects in Ruby

Wowzers, I just tackled a major memory issue in an app I’ve been developing, and it was nearly all due to rampant over-instantiation of ActiveRecord objects.

Here’s the TL;DR: Use SQL queries whenever possible to grab info from the database, rather than using Ruby code.

Now for the longer version. First, I knew I had a memory issue because my server was crashing and it was showing up the logs saying that memory could not be allocated. But I had to find out what the issue was first.

Oink to the rescue. It’s a handy little Ruby gem that will log how much memory and how many ActiveRecord objects are instantiated for each request.

I installed Oink, ran through a few requests that seemed kinda slow, and lo and behold, I found the problem: One of my requests was generating 1500 objects!

Here’s what was happening. I was looping through an array of IDs, and for each one I was calling

account.users.order(:id).select{ |user| user.id == id }.first

Don’t laugh. I have a baby at home, I was sleep deprived, I don’t know why I did it this way. But yes, this is instantiating all User objects in an account, and it’s doing it every single time it iterates through the loop. Hence, 1500 objects instantiated per request.

This was easily solved by a simple find_by_id with the id from the array.

That took me down to only one instance of each User object, which is still not fantastic. So, I ended up using one of my favorite ARel methods, pluck, to get just the attributes of the users that I need without actually creating the User objects.

That’s the ticket! From 1500 User objects to 0, this should significantly help with speed and memory usage. I ended up combing through the rest of the app and finding a few other spots where object instantiation was a bit out of control, though not nearly as bad as that one.

It became a kind of game - and in the end, I’d say I won.

Thanks to Engine Yard for the great tips. Lots more in their linked post.

Debugging Capybara Poltergeist tests in Rails

I made the switch from selenium webdriver to poltergeist on my capybara tests when the suite started taking 20 minutes to run. (For my process of getting the test suite under control, check out my previous post.) While poltergeist has definitely been a boon, it can sometimes be hard to debug because I can’t physically see the page. That’s where this tip comes in handy, stolen from the Quickleft blog:

Find the Port Address and Pause the Test 

Sometimes I just want to see what's going on in the browser. Although save_and_open_page gives you some idea of what's going on, you can't really click around, because the page it gives you is static and lacking assets. To dig into what's going on, I like to use a trick I learned from my mentor, Mike Pack. 

Just above the broken line in your test, add these two lines of code. Note that you have to have pry installed in your current gemset or specified in the Gemfile for this to work. 

puts current_url
require 'pry'; binding.pry 

Run the specs, and when they pause, copy the url and port number from the test output. 

Open your browser and paste the address into the window. Voila! You're now browsing your site in test mode!

Thanks again to the Quickleft folks for this cool tip - it’s one of five tricks for less painful testing.

URL Encoding and Silly Safari

I have a web app that also acts as an API for an iOS, and I ran into a tricky situation. One of my models on the web app takes a URL as input, and it validates it using the validates_url gem. However, some URLs when visited from within the iOS app would give a Webkit error and show as invalid.

After some head scratching, it turned out that those URLs had encoding issues that were handled fine by all modern browsers except Safari, the default browser on iOS. For example, something with curly brackets like "http://www.example.com?ID={12345}" would work in most browser but not in Safari/iOS.

To get around this, in addition to using validates_url, I had to escape each URL during validation using URI.escape(url). The above URL would then become "http://www.example.com?ID={12345}". (Notice that the left curly bracket changed to %7B, the right turned to %7D - check out this HTML URL Encoding Reference for details)

But wait, there’s more!

It turns out that on subsequent saves, the URL would be escaped again, and it wouldn’t recognize the %7B and %7D as encoding, it would just see the % sign as an invalid character needing escape. So, on the second save, the same URL would be transformed again into "http://www.example.com?ID=%7B12345%7D". Notice the “25” inserted in there twice?

The fix ended up being something that reads very strange and seems unnecessary, but that works perfectly:

URI.escape(URI.unescape(url))

So, if a URL is already escaped, it unescapes it first, then re-escapes it. If the URL is not escaped, then unescaping it does nothing, then it gets escaped as it should.

If there’s a better way to do this, I haven’t found it, and am open to suggestions.

Asynchronous processing pitfalls and ActiveRecord locking

My app is humming along, but there are a few sections that have to do a lot of heavy lifting, whether that be image processing, updating several different related objects, or looping through all of a has_many association to manipulate every record.

These types of things are perfect places to push some logic to a background process. In other words, make it asynchronous. So, if a user clicks a button to update several objects, unless the user needs to see those updates completed immediately, then a tool like Sidekiq can help run some expensive logic as a background process and allow the user to continue surfing the site while that process runs.

There are plenty of tutorials about how to set this up, so that’s not what I want to talk about here.

One of the potential pitfalls of processing things asynchronously, particularly if the processing involves updating database records, is multiple records being updated at the same time. For example, one user updates a Project in a background process, and another user updates the same Project somewhere else in the app that doesn’t use a background process. We want to last update to be the one that sticks, but if both updates are happening at the same time, things could get hairy.

This is the dreaded “race conditions” issue, and luckily there’s a pretty simple fix.

On ActiveRecord - and I believe this will work with any SQL database but definitely at least on Postgres and MySQL - I can lock a specific row while editing, then unlock it afterwards, so that each edit takes place in its own little bubble.

Account.transaction do
  acc = Account.lock.find(id)
  acc.balance -= 25
  acc.save!
end

This finds an account and locks it, then updates the balance, saves, and unlocks.

This is really important when dealing with numbers, money, and balances. Let me share a parable to explain:

There is an Account with $100 balance. One user in that account makes a transaction for$75.

A split second later, another user in that account makes a transaction for $90.

What should happen is the second user’s transaction should be rejected for lack of funds.

What actually happens without locking is that for each transaction, the first step is that the server runs account = Account.find(id) to retrieve the account, and since this is happening before either transaction has finished, the balance on the account in both cases is $100. So, after the first user's transaction, the balance is updated to$25 and the account is saved. But on the second user’s transaction, the account object that was pulled from the database still shows a balance of $100, so the server deducts the$90 charge from the balance and saves it, thereby allowing the transaction to go through and setting the saved account balance as $10.

Is your head spinning yet?

When we lock the account per transaction, we avoid this issue by only allowing the account to be edited on transaction at a time.

It’s a cool concept but also can be tricky to wrap your head around. Here’s a blog post I borrowed heavily from for my post. Or check out the Rails API docs.

Turbocharged custom Rails / ActiveRecord validations

My code just got a lot less janky when dealing with start and end times. Before, when I wanted to validate that a user-submitted end time came after a start time, I was doing something like this:

validate :start_time_is_before_end_time
def start_time_is_before_end_time
    unless start_time.to_i < end_time.to_i
        errors.add(:end_time, 'must come after start time')
    end
end

Converting the values to epoch time and comparing isn’t terrible, but I just found a cool way to extract this out to a custom validator to make it simpler and more readable:

# app/models/model.rb
require "active_model/sequential_validator"
validates :start_time, :end_time, sequential: true

# lib/active_model/sequential_validator.rb
class SequentialValidator < ActiveModel::Validator
    def validate record
        values = options[:attributes].map do |attribute|
           record.send(attribute)
        end.compact
        if values.sort != values
            record.errors.add options[:attributes].last, "cannot be before #{options[:attributes].first}"
        end
    end
end

True, the actual validator code is not as easy to read I don’t think, but it takes some of the burden out of the model, and it also allows reusing the sequential validation in other models.

(Thanks to botandrose and the calagator project for this cool piece of code.)

Indifferent Access in Ruby Hashes (with ActiveSupport)

I kept seeing this “Hash with indifferent access” in open source projects, but never knew quite what it was. Turns out it’s something I use in Rails all of the time without knowing, particularly in dealing with params. It allows me to access values in a hash using either string or symbolized keys.

For example:

params['name'] == params[:name]
# => true

However, a normal Hash in Ruby does not allow this:

hash = { 'name' => 'Fred' }
hash['name']
# => 'Fred'
hash[:name]
# => nil
hash['name'] == hash[:name]
# => false

We can fix this by making this into a hash with indifferent access:

new_hash = hash.with_indifferent_access
new_hash['name'] == new_hash[:name]
# => true

The hash can also be defined/initialized with indifferent access, either by calling that method on the initial hash assignment, or by creating it using ActiveSupport (which is what the with_indifferent_access method does under the hood anyway):

hash = ActiveSupport::HashWithIndifferentAccess.new(name: 'Fred')
hash['name'] == hash[:name]
# => true

Note: When working in straight Ruby, this will not work as it’s using ActiveSupport. One would likely need to run gem install active_support and require it properly.

Source: apidock and RoR api

String conditionals on ActiveRecord callbacks

I’ve got a model for push notifications that uses Parse.com, and it sends the data to Parse on a callback:

before_create :send_to_parse

The problem is, I have a whole bunch of specs that create push notifications, and I don’t want any sent to Parse during testing. Now, I don’t have the Parse credentials accessible within the test environment, but that throws an error when it tries to send to Parse. So, what I really want is to just send to Parse when we’re not testing.

Turns out this is dead simple, albeit a little strange at first:

before_create :send_to_parse, unless: "Rails.env.test?"

I’ve used conditionals with symbolized method names before, but the string bit threw me off. However, it’s simple valid Ruby code, you can put it in a string and it will be evaluated as such.

This is straight from the docs, and I thought it was super cool.

ActiveRecord querying on Postgres array column

On a Rails project I’m working on, we have a model that has a db column of array datatype. This is one of those Postgres special datatypes that isn’t present on many other SQL databases, and there’s even some contention about whether it should be an option at all.

At any rate, this is deeply ingrained in our project currently, so I don’t want to go reinventing the wheel at this point and convert this particular column to a separate model and create another has_many/belongs_to relationship. All I want is a way to find all of the objects in this model that have an empty array in that column.

Turns out there’s no built in ActiveRecord finder for this, but the raw SQL isn’t too terrible:

Model.where("NULL = ALL(array_column_name)")

Hat Tip: https://coderwall.com/p/sud9ja/rails-4-the-postgresql-array-data-type

Unique objects from a three way join

I have a model that is essentially a three-way join of three models. Let’s call this model Widget. It has a user_id, account_id, and place_id. I want to make sure each of those widget objects is unique in some way - in other words, I don’t want any widget object with the same user_id, account_id, and place_id.

I should probably validate for uniqueness, but only within the confines of what is in the other columns. Here’s where uniqueness scope comes in handy:

validates_uniqueness_of :user_id, scope: [:account_id, :place_id]

Now, if we try to create a Widget with the same user_id, account_id, AND place_id as one already in the system, ActiveRecord will yell at us. But if any of those values are different from what’s already in the db, we’re good to go.

If this was going to be user-facing, I’d probably also want to add a custom error message to this, otherwise it would say something like ‘user_id has been taken’, which isn’t ideal. However, these Widget objects will be created in the background, not explicitly by users, so we shouldn’t ever see these error messages.

Now let’s say we want to control for what might already be in the db. Suppose that we didn’t find out this was an issue until a bunch of people already created duplicate Widgets. Sure, we could create a rake task to comb the db and delete the duplicates, and that would probably be a good idea at some point if we truly don’t need them. But what if for some reason we wanted duplicates to save to the db, but on certain pages only wanted to display unique Widget objects?

Running Widget.all.uniq doesn’t do anything for us, because each widget object has a unique id/primary key. So, assuming we don’t need the primary key on the page we’re displaying these widgets (and since this is essentially just a three-way join with no other attributes, I think that’s a fair assumption), we can pull all other attributes besides the primary id for all the widget objects using ActiveRecord’s select:

Widget.select(:user_id, :account_id, :place_id)

This will give us all of the widget objects in the system, but with nil values for any attributes we didn’t specify - namely, the id field. Now, since we don’t have the unique id to deal with, we can easily filter duplicates using uniq:

Widget.select(:user_id, :account_id, :place_id).uniq

Just to make it cleaner, I would probably make that a scope on Widget so I could give it a name and have it make more sense for other coders on the project.

class Widget < ActiveRecord::Base
  scope :unique, -> { select(:user_id, :account_id, :place_id).uniq }
  # other stuff in Widget class
end

Widget.unique

Bingo bango bongo.

Hack your Rspec setup using tags

I ran into an issue in rspec the other day where I had a before block to setup about 10 different tests, but I needed to write one more that was similar enough that it belonged in the same context but it required slightly different setup so the before block I already defined.

What to do? I could break out the existing tests into a nested context block and use the setup there, then do another nexted context block for my new test, but I really didn’t want to do that because all these tests should belong in the same context. I just wanted to skip the before block on one specific test.

Rspec tags to the rescue!

In this case, my new test was a JS test and all others were not (which is partly why the setup was different. So, the test was already tagged js: true, and in mybefore` block, I modified it like so:

before(:each) do |test|
  # Run this only on tests NOT tagged with js: true
  if test.example.metadata[:js] == nil
    # Existing setup code
  end
end

Works like a charm, and I still have all my related tests within the same context like I wanted.

Hat Tip: http://stackoverflow.com/questions/27864080/how-can-i-skip-some-setup-for-specific-rspec-tags

Simplifying Rails views with pluralize

"2 User(s) been updated."

This is a silly problem that I kept running into before I found out about pluralize.

Now, instead of having to do a conditional checking for number of users and printing the line based on that number, I can do this using pluralize in Rails:

pluralize(@users, 'User')

Or, if I want to use the Ruby version of pluralize (or, more accurately, the ActiveSupport version):

require 'active_support/inflector'
"User".pluralize(@users)

If @users is more than one, outputs "Users". If @users is 1, outputs "User". This is nice for when you don't want to display the number, just the pluralized word (i.e. "Users updated").

Beware reserved ENV names in Rails

Not all environment variable names are created equal.

For example, if you are using Ruby on Rails, do not declare a VERSION environment variable, there is already one a reserved. The reserved VERSION envar is used by Rails in migrating database changes, so if it is changed at all, you will have problems when making new migrations.

That was a fun error to debug...

Testing CSV exports/downloads in Ruby on Rails with Capybara

As low-tech as CSV is, it doesn't appear to be going away anytime soon. So, I've run into a few different Ruby on Rails projects that want import from/export to CSV capability, and because I like tests, I wanted to make sure I could test not only that the CSV is exported but that the content of the CSV is correct.

Here's how to do this using Capybara and Rspec:

First, switch the driver to something that works (webkit or rack_test, but not selenium). To do this, you can run this within your code block:

Capybara.javascript_driver = :webkit

Just make sure to change it back to your preferred driver after you're done. Another option is to do this in your spec_helper.rb:

config.before(:each, webkit: true) do
    Capybara.javascript_driver = :webkit
end

config.after(:each, webkit: true) do
    Capybara.javascript_driver = :selenium 
    # or whatever your preferred driver is
end

Then just tag any CSV tests with the webkit: true tag.
(Thanks to Kevin Bedell on StackOverflow for this one.)

Once you get the driver configured properly, you can check the headers and the actual content of the CSV trivially:

click_on 'Download as CSV'
# Make sure the page is a CSV
header = page.response_headers['Content-Disposition']
expect(header).to match /^attachment/
expect(header).to match /filename="my_file.csv"$/

# Then check the content
MyModel.all.each do |record|
  expect(page).to have_content record.field1
  expect(page).to have_content record.field2
end

(Props to Andrew on StackOverflow for this bit.)