Using Pry in production (part 2)

Last week, we took a look at the basics of using Pry in production as a replacement for the standard IRB console. Today, let's look at a practical scenario and see how Pry can help us debug a misbehaving request.

Why use Pry?

After all, we already have New Relic which gives us a stack trace of every error in our application.

Well, many errors do not result in an exception. Oftentimes the application responds, but with the wrong data. And oftentimes as well, the error only occurs in rare cases dependent on production data: it's hard to reproduce an error locally if you don't even know what makes the error occur in the first place.

There are 2 ways developers deal with this problem:

Have a full copy of the production environment (with up-to-date production data). While this may be feasible for small datasets, it becomes harder and harder as the data grows. Customers will also often complain about an error that just appeared, making it more likely that the important data does not yet exist in the test environment.
Connect a local development environment to a remote database. I find this method too dangerous: it's a disaster waiting to happen. You should not trust yourself with a direct connection to the production database on your machine. Ever.

In these situations, I find that Pry offers just enough functionality to do some actual debugging in production, while keeping you relatively safe.

PryNav

I like pry-nav better than pry-debugger or pry-byebug because it works with every version of Ruby, and being pure ruby, doesn't require me to include the debugger gem in production. I can thus safely include pry and pry-nav in my Gemfile, keeping them unloaded, and only require them when I start the Rails console manually with my pryrc. You get the best of both worlds: no impact on running production servers, but the flexibility of a basic debugger when you need one. And it works everywhere.

# Gemfile
gem "pry", :require => false
gem "pry-nav", :require => false

Scenario

One of my apps is a customer support app, and so it deals a lot with discussions, comments, support staff, etc. I had a case recently where a customer could not assign a particular discussion to a particular user. It worked (and had been working) for every other discussion and every other user, so it was fairly obvious that there was something peculiar about the data that triggered a bug. A first exploration of the code involved did not reveal any obvious clue, so this was thus a perfect use case for Pry.

Step 1: Authenticate

Rails provides you with a few methods to interact with your app directly from the console (see this blog post for a good introduction). The first step of every debugging session is often to authenticate as the user with something like that:

[1] pry(main)> app.post "/login",  :user => {:login => 'joe', :password => 'password'}

For real applications however, this is unpractical as passwords are not stored in clear text in the database. We still need to login as our user though... While we can't use a login/password, most apps will have a current_user method or similar, that we can patch.

First, let's edit our current_user method:

[1] pry(main)> edit Authenticated#current_user -p

And overwrite it:

def current_user
  return User.find(1234)
end

From now on, every request will be made as user 1234.

Step 2: Set our breakpoint

We know our entry point: the controller action. So let's start there:

[2] pry(main)> edit DiscussionsController#assign -p

and add a call to Pry:

def assign
  binding.pry
  [...]
end

Now, call our action:

[3] pry(main)> app.post "/discussions/problems/1234/assign", :user_id => 768

From: /data/APP/current/app/controllers/discussions_controller.rb @ line 72 DiscussionsController#assign:

    71: def assign
 => 72:   binding.pry
    73:   @discussion.assign(user)
          [...]
    89: end

[1] pry(#<DiscussionsController>)>

So let's try the basics: run the line and verify if it works or not.

[1] pry(#<DiscussionsController>)> @discussion.assign(user)
=> nil

[2] pry(#<DiscussionsController>)> @discussion.assignments
=> []

That doesn't fair well.. Let's step inside the method instead:

[3] pry(#<DiscussionsController>)> step

From: /data/APP/current/app/models/discussion/concerns/assignments.rb @ line 87 Discussion#assign:

    86: def assign(user)
 => 87:   unless assigned?(user)
    88:     assignment = Assignment.create(:discussion => self, :user => user)
    89:   end
    90: end
    
[4] pry(#<DiscussionsController>)>

Something looks fishy:

[4] pry(#<DiscussionsController>)> assigned?(user)
=> true

[5] pry(#<DiscussionsController>)> assignments
[]

Let's look inside assigned?

[6] pry(#<DiscussionsController>)> step

From: /data/APP/current/app/models/discussion/concerns/assignments.rb @ line 123 Discussion#assigned?:

    122: def assigned?(user)
 => 123:   REDIS.sismember(RedisKey.assignments(self.id), user.id)
    124: end
    
[7] pry(#<DiscussionsController>)>

And there is our bug! The assigned? method looks in our Redis cache to see if the discussion is already assigned to this user, but assignments looks in the database. Somehow the data in Redis has been corrupted, and is no more in sync with the database.

Step 3: Fix it

Now that we understand the bug, we can go back to our development environment and start writing tests, a fix, etc.. We can also start looking at logs to figure out how the Redis data got corrupted in the first place: we will have to fix that as well.

Conclusion

While Pry should not be used as the "be-all and end-all" for production problems, it does come in handy to debug issues related to data, that are hard or impossible to reproduce in other environments. By selectively patching code in the session, and stepping into requests, we can quickly figure out the source of problems, and then go back to our normal workflow to write the fix.

I hope I was able to convince you that Pry is totally awesome. But don't take my word for it, try it for yourself: you'll be hooked!

That's it for today, cheers!