Posted: November 4th, 2012 | Author: | Filed under: posts | Tags: , , | No Comments »

One of the biggest improvements at in October was our migration into a new production environment. We had outgrown our previous environment and took the opportunity to improve many aspects of our production setup as we built out the new systems.

The majority of our servers are now virtualized using VMWare ESX. Of the BetDash servers, only our database remains on physical hardware – this is at the recommendation of the Oracle MySQL consultant we worked with on our deployment.

This is what our infrastructure looks like in our new environment and how our stack is distributed across the different tiers:

  • Citrix Netscaler load balancers
  • Web servers running Apache, serving the contents of our Rails app’s public folder, i.e. all static assets (CSS/JS/Images). This content is also replicated into our CDN.
  • Passenger Enterprise on top of Apache for our front-end application servers, serving our primary Rails 3.2 app. These boxes are presently 8 virtual CPUs and 8 GB RAM
  • Resque, Resque Scheduler, and other proprietary backend applications run on our backend application servers – these are 16 virtual CPUs and 32 GB RAM.
  • Dedicated instances for our Admin panel application, Redis, Memcached, and other internal applications utilized by our system.
  • Redis is currently deployed in a master/slave replicated setup, however, we’re keeping a close eye on Redis Sentinel emergence from beta and are also looking to deploy Redis Failover on top of Apache Zookeeper.
  • MySQL is on physical hardware, configured as a RedHat Conga Cluster with the database files themselves sitting on a SAN. Each node in the cluster has 12 physical CPUs and 96 GB RAM.
  • Dedicated MySQL instances for reporting and data warehousing

The instances are built with Puppet and the applications are deployed out of git using a combination of Capistrano and Paddy Power’s proprietary release tool.

One of the most common comments in regard to our stack and environment is the question of why we’re running on our own VMs and hardware as opposed to hosting in the AWS cloud like everyone else. The regulated nature of real-money gaming requires that our systems sit in the Isle of Man and thus this rules out the IaaS options. It was extremely interesting to see the number of leading Rails consultancies, which we were speaking with in regards to our deployment strategy and tuning our application for the new environment, that haven’t worked with non-IaaS systems in years.

With our move to these systems, we’re well positioned as we continue to scale the BetDash platform, thanks to the hard work of a large cross-functional project team. Our next step will be to move to run in an active/active configuration across Paddy Power’s multiple data center sites.

Upgrade Ruby Enterprise Edition

Posted: February 21st, 2011 | Author: | Filed under: posts | Tags: , | No Comments »

A security update for Ruby has been released to respond to multiple vulnerabilities. All versions of the interpreter are affected, including the Ruby Enterprise Edition interpreters.

Fortunately, it’s easy to upgrade with RVM:

Tell RVM to upgrade your specific REE installation to the latest:

rvm update                                                   #upgrade RVM
rvm upgrade ree-1.8.7-2011.02 ree-1.8.7-2010.02              #upgrade REE

Accept Y to all options to move elements of your current install.

Then reset REE to be your default:

rvm ree-1.8.7-2011.02
rvm --default ree-1.8.7-2011.02

Following this, confirm that the new install of Ruby is active for your shell session (which ruby). Then, you will likely need to install bundler again to make sure you have 1.0.10 for the upgraded copy of gem 1.5. Also, run bundle install from the app root to make sure your gemset is up to spec.

Despite telling the upgrade process to move everything, I still needed to reinstall Passenger (3.0.2) and reconfigure it for Apache:

gem install passenger

With this reinstallation, you’ll also need to edit your Apache config file (on Mac OS X: /etc/apache2/httpd.conf ) to point to the new Passenger gem.

Restart Apache from System Preferences > Sharing > Web Sharing and you should be good to go.

Hey, what caused that ActiveRecord::ConnectionAdapters::TableDefinition gooblygook on my table?!

Posted: November 24th, 2010 | Author: | Filed under: posts | Tags: | No Comments »

Maybe you’ve run into this, too? You’re taking a look at the tables generated by your Rails migration and you see this really odd column.


Here’s another look:


Searching the web didn’t turn up much in this case, so I thought I’d post what I found when figuring out the root cause.

See the issue in this migration?

No? I didn’t the first time I looked, either.

It’s the comma – at the tail end of the last column definition before t.timestamps.

This little typo doesn’t look like much, but it’s the cause of that wacky column.

Here’s what the table looks like when you run the migration with the comma:

And here’s what it looks like when you fix your typo:

Ah! Much better.

As for creating a migration to fix this, I just resolved the typo and dropped the column manually. It’s a little difficult to script this because the column name will be different in each database, so, it’s some variation of this (the part in bold on each database that the original typo ran on):

ALTER TABLE `friendships` DROP `#<ActiveRecord::ConnectionAdapters::TableDefinition:0x10245fd78>`;

Hope this helps someone else avoid wasting time on this!

Why is the user experience for adding Facebook apps so poor?

Posted: January 12th, 2010 | Author: | Filed under: posts | Tags: , , , | No Comments »

A friend linked me to a random promotions app on Facebook the other day and it stopped me in my tracks for a moment.  Unlike most apps I’ve seen, the first screen was something very interesting – it was actually inside the application itself and was not the not the lovely “Allow Access” screen that Facebook users are accustomed to.

This made me wonder – pretty much most of the biggest Facebook apps I can think of ask for permission before letting you do anything.  I understand why Facebook is protecting profile information, but as a developer, you can configure your initial app screen so as not to require permission to access content.  The promotions app did this and we”ve set it up for our application.

So, the question is – why are so many huge Facebook app developers happy with the poor user experience of having the “Allow Access” screen being the first thing that users encounter with their application?  Why not show a screen that gives you a better idea of what the app is and then if the users actually wants to use the app, then they would request access.  Is this simply a ploy for the publishers to get more “active users” and thus boost themselves in the AppData standings?  Or are Facebook users quickly becoming numb to releasing their private information and the age of privacy is really over, as Facebook’s Zuckerberg recently claimed?

FYI – for Rails developers using Facebook, it’s pretty easy to not require the Allow Access screen on your welcome controller or whatever serves as your landing screen.  Just skip the Facebooker ensure_app_installed before_filter and don’t make calls that try to access the user’s information:

skip_before_filter :ensure_app_installed

Collaborative Team Facebook Development

Posted: January 10th, 2010 | Author: | Filed under: posts | Tags: , , , , | No Comments »

For as much as has been written regarding developing Facebook applications with Rails (which isn’t that much considering the staggering popularity of Facebook and Rails), there is even less written on developing Facebook applications in a team environment.

There are a couple of potential strategies. One of the few articles out there describes the challenges in developing apps as a team:

This sets up some challenges for a development team. First off, your application probably relies on Facebook services, meaning it can only be fully tested from within Facebook. This implies that the external Facebook server needs to be connected to your development server. Since many Rails developers run off their own development machine behind a firewall, this is potentially awkward. Similarly, the fact that the Facebook application has a single URL mapping makes it difficult for multiple developers to work on the same application without tripping over each others” work.

The author goes on to discuss the solution that their team is using, consisting of a series of Rake tasks and a staging server to facilitate testing in a team environment.

The other option, which is supported natively by Facebooker, is using an SSH tunnel from the local development workstation to an intermediary server and having the Facebook app point at that server. (This necessitates having an understanding IT team that doesn’t lose that much sleep over some open ports..) The SSH tunnel is discussed by many of the Facebooker tutorials available (here, here, and here).

We adapted the SSH tunnel strategy to our team environment. Each developer will have a development Facebook app and then will point that to their assigned port on our staging server. Then they can tunnel into the staging server from their local workstation. In this way, everyone can be testing their work independently without needing to coordinate deploys through to a staging and central test application. The below diagram illustrates this.

Ok, so we have our plan, but how to go about setting this up? In particular, the other tutorials don”t talk about what you do or do not have to do on the server side. In truth, the tunnel is fairly easy to configure and seems to be working well.

On the server side, the key is making sure that the ports you need aren”t blocked and are accessible for purposes of the tunnel. No need to configure virtual hosts or mongrels – the staging server is a simple relay. You will probably need to configure SSH to turn on GatewayPorts. That”s just a couple of quick steps:

  • sudo vi /etc/ssh/sshd_config
  • Find the line GatewayPorts if it”s there.
  • Change or add to: GatewayPorts yes
  • Restart SSH: sudo /etc/init.d/ssh restart

(based on other blog posts I came across, be sure to indicate GatewayPorts yes and not GatewayPorts clientspecified)

Now, via Facebook”s Developers application, each developer should configure an application for testing purposes. Set the Canvas Callback URL to (Facebook requires that you have a trailing slash) and you should be good to go. Depending on your application, you may need to configure other settings (other callback URLs or iFrame/FBML, but the canvas callback URL is the primary setting. Make a note of the API key and the secret key.

Finally, we can setup the tunnel. Go into the Rails app that you pulled down from your Git repp and edit config/facebooker.yml. Add in the API key and secret key from your development application you just configured. For the canvas page name, enter the name of your application, as configured on Facebook. Then change the callback URL to the same URL the app is configured for: (again, trailing slash!). Then enter the SSH username and hostname for your staging server, save the file, and you should be good to go.

At this point, you should be able to start your development server (ruby script/server) and then start the tunnel (rake facebooker:tunnel:start). If you’re on Windows, be sure to run these commands through Cgywin. At this point, you should be able to go to and see your development app. While the app is loading, you should also see the SSH command window and the Ruby Webrick/Mongrel window showing the incoming connection and the Rails system handling the requests.