One of the biggest improvements at BetDash.com in October was our migration into a new production environment. We had outgrown our previous environment and took the opportunity to improve many aspects of our production setup as we built out the new systems.
The majority of our servers are now virtualized using VMWare ESX. Of the BetDash servers, only our database remains on physical hardware – this is at the recommendation of the Oracle MySQL consultant we worked with on our deployment.
This is what our infrastructure looks like in our new environment and how our stack is distributed across the different tiers:
- Citrix Netscaler load balancers
- Web servers running Apache, serving the contents of our Rails app’s public folder, i.e. all static assets (CSS/JS/Images). This content is also replicated into our CDN.
- Passenger Enterprise on top of Apache for our front-end application servers, serving our primary Rails 3.2 app. These boxes are presently 8 virtual CPUs and 8 GB RAM
- Resque, Resque Scheduler, and other proprietary backend applications run on our backend application servers – these are 16 virtual CPUs and 32 GB RAM.
- Dedicated instances for our Admin panel application, Redis, Memcached, and other internal applications utilized by our system.
- Redis is currently deployed in a master/slave replicated setup, however, we’re keeping a close eye on Redis Sentinel emergence from beta and are also looking to deploy Redis Failover on top of Apache Zookeeper.
- MySQL is on physical hardware, configured as a RedHat Conga Cluster with the database files themselves sitting on a SAN. Each node in the cluster has 12 physical CPUs and 96 GB RAM.
- Dedicated MySQL instances for reporting and data warehousing
The instances are built with Puppet and the applications are deployed out of git using a combination of Capistrano and Paddy Power’s proprietary release tool.
One of the most common comments in regard to our stack and environment is the question of why we’re running on our own VMs and hardware as opposed to hosting in the AWS cloud like everyone else. The regulated nature of real-money gaming requires that our systems sit in the Isle of Man and thus this rules out the IaaS options. It was extremely interesting to see the number of leading Rails consultancies, which we were speaking with in regards to our deployment strategy and tuning our application for the new environment, that haven’t worked with non-IaaS systems in years.
With our move to these systems, we’re well positioned as we continue to scale the BetDash platform, thanks to the hard work of a large cross-functional project team. Our next step will be to move to run in an active/active configuration across Paddy Power’s multiple data center sites.