BetDash wins EGR’s Innovation in Gaming Award

Last week, at the EGaming Review 2012 Operator Awards, BetDash took home the Innovation in Gaming award. Presented to the company with the most disruptive innovation, the award is a great testament to the contributions of everyone on the BetDash team and within the Paddy Power organization.

Three years ago, we had an early prototype, some grand visions, but very little background in what is a huge industry outside of the US. Now we are poised to disrupt a part of that industry. Lots of exciting potential for BetDash!

Amazing experiment by One Laptop Per Child

Just to give you a sense of what these villages in Ethiopia are like, the kids (and most of the adults) there have never seen a word. No books, no newspapers, no street signs, no labels on packaged foods or goods.

“We left the boxes in the village. Closed. Taped shut. No instruction, no human being. I thought, the kids will play with the boxes! Within four minutes, one kid not only opened the box, but found the on/off switch. He’d never seen an on/off switch. He powered it up. Within five days, they were using 47 apps per child per day. Within two weeks, they were singing ABC songs [in English] in the village. And within five months, they had hacked Android. Some idiot in our organization or in the Media Lab had disabled the camera! And they figured out it had a camera, and they hacked Android.”

Ethiopian kids hack OLPCs in 5 months with zero instruction

Originally from MIT Technology Review

BetDash.com Infrastructure

One of the biggest improvements at BetDash.com in October was our migration into a new production environment. We had outgrown our previous environment and took the opportunity to improve many aspects of our production setup as we built out the new systems.

The majority of our servers are now virtualized using VMWare ESX. Of the BetDash servers, only our database remains on physical hardware – this is at the recommendation of the Oracle MySQL consultant we worked with on our deployment.

This is what our infrastructure looks like in our new environment and how our stack is distributed across the different tiers:

• Citrix Netscaler load balancers
• Web servers running Apache, serving the contents of our Rails app’s public folder, i.e. all static assets (CSS/JS/Images). This content is also replicated into our CDN.
• Passenger Enterprise on top of Apache for our front-end application servers, serving our primary Rails 3.2 app. These boxes are presently 8 virtual CPUs and 8 GB RAM
• Resque, Resque Scheduler, and other proprietary backend applications run on our backend application servers – these are 16 virtual CPUs and 32 GB RAM.
• Dedicated instances for our Admin panel application, Redis, Memcached, and other internal applications utilized by our system.
• Redis is currently deployed in a master/slave replicated setup, however, we’re keeping a close eye on Redis Sentinel emergence from beta and are also looking to deploy Redis Failover on top of Apache Zookeeper.
• MySQL is on physical hardware, configured as a RedHat Conga Cluster with the database files themselves sitting on a SAN. Each node in the cluster has 12 physical CPUs and 96 GB RAM.
• Dedicated MySQL instances for reporting and data warehousing

The instances are built with Puppet and the applications are deployed out of git using a combination of Capistrano and Paddy Power’s proprietary release tool.

One of the most common comments in regard to our stack and environment is the question of why we’re running on our own VMs and hardware as opposed to hosting in the AWS cloud like everyone else. The regulated nature of real-money gaming requires that our systems sit in the Isle of Man and thus this rules out the IaaS options. It was extremely interesting to see the number of leading Rails consultancies, which we were speaking with in regards to our deployment strategy and tuning our application for the new environment, that haven’t worked with non-IaaS systems in years.

With our move to these systems, we’re well positioned as we continue to scale the BetDash platform, thanks to the hard work of a large cross-functional project team. Our next step will be to move to run in an active/active configuration across Paddy Power’s multiple data center sites.

The key question for managers to ask themselves – How do employees feel after they interact with you?

There was an excellent nugget of management advice in Bob Sutton’s Stanford Entrepreneurial Thought Leaders podcast, What Great Leaders Do. Dr. Sutton, Professor of Management Science and Engineering in the Stanford Engineering School, referenced work by Rob Cross, from the University of Virginia. Apparently, the findings were derrived from a network analysis study done for a large management consulting firm. (If you want to listen to the section, it starts around 43:25 in the podcast.)

In this study, Cross included a question in the study that had interesting results:

After you talk with your boss, do you have more or less energy?

When analyzing the results from the study and subsequent studies using the same question, they found a strong correlation between whether the employee was promoted, management’s performance evaluation of the employee (whether the employee was promoted/fired) and whether the manager is surrounded with a network of energized innovators.

Therefore, Dr. Sutton suggests that the most salient question for managers to ask themselves is:

How do people feel after they interact with you?

I think this is a great way of boiling down what can be difficult to evaluate for yourself (how am I doing as a manger) into something straightforward to focus on. In your interactions with your employees, energize them. Empower them.

I think we all know from personal experience how difficult it is when you have a manager that drains your energy. Where interacting with them sucks your enthusiasm for the job away. It’s not a good experience for either party.

Make sure you’re energizing & empowering your employees each and every time you interact with them.

(More on this from Bob Sutton here)

Engineer Shortage

There’s been an increasing amount of conversation recently about how difficult it is for startups to find qualified engineers. And yet at the same time, recent graduates with strong engineering credentials are saying that they can’t find jobs.

What’s going on here?

(Let’s set aside those who aren’t looking to work for a startup. Nothing wrong with that – but the hiring process for corporations is governed more by the macroeconomic situation and waiting for executives to feel comfortable that the analysts on Wall Street are comfortable with increased hiring expenditures. I can’t provide much insight into that. But I can offer some thoughts for recent graduates who are chomping at the bit to get out there and put the skills that they’ve spent years honing to good use, and they’re hungry for any opportunity, especially at a startup where their contributions will actually move the needle.)

I think the immediate reaction is to try to attribute this apparently misalignment of supply and demand to some generalization that startups won’t hire recent graduates.

In December, Vivek Wadhwa wrote a post on TechCrunch, Shortage of Engineers or a Glut: No Simple Answer. The post made a lot of useful points and quoted some smart people (including a comment that I left on his blog).

After thinking about this again recently, I went back and read some of the comments on the post.

Most of the comments seemed to be along one of the following themes:

1) Schools don’t train me in what’s really being used in industry. All of the job requirements that I read ask for X years of experience with Y skill (that I don’t have). What is wrong with education?

2) Startups don’t just hire anyone. They are looking for engineers who have strong fundamental skills, can pick up new technologies and run with them, and are willing to tackle any problem.

There is truth in both of these themes.

School doesn’t teach me what I need to know for industry

Perhaps the Information Sciences and Technology program at Penn State was unique in this regard, but the Dean and professors would regularly get up in front of students and say something along the lines of, “Technology is an industry which requires lifelong learning. We will teach you how to learn. Then you can use that skill to learn what you need for your career.” Sure, we learned some useful things along with the theories, but it was always made clear to us that we shouldn’t expect the curriculum to teach us everything we would ever need for a career in technology.

Good thing, too. Because the vast majority of the skills that I use on a day-to-day basis in my COO/CTO role were things that I’ve learned myself.

One of the comments on the post spoke to this point: “I was shocked that so many of my classmates didn’t apply what they learned — about how to learn — to themselves. In the time between my CS degree and my first industry job, I’d taught myself version control, design patterns, unit tests, embedded systems programming, user interface design, and so on. It’s easy to get a good job when so many of my classmates thought that college was the last time they’d ever have to learn something!”

This is why so many companies look for candidates who contribute to Open Source. Because contributions aren’t something that you regularly do as a part of school or as a part of most jobs. Instead, it’s something you do on your own accord, and it demonstrates both initiative and the ability to pick up something and run with it.

Startups don’t hire just anyone

That’s true – they don’t. But it doesn’t mean that you aren’t already or can’t become the kind of person that they will hire.

I make this point in my comment on Vivek Wadhwa’s personal blog, that led to the subsequent TechCrunch article:

Just because engineers are graduating, or there are engineers on the bench with decades of experience, this doesn’t immediately solve the issue that a startup CTO faces in trying to staff their team. At the start, an entrepreneur is looking for a driven technologist who can come in and partner with them to translate their vision into reality. If you’re making hire #2 or beyond, you’re looking for specific needs – startups often don’t have the luxury to train someone into their technology stack and are looking for folks who can hit the ground running and get stuff done. As a result, simply having an engineering degree or decades of experience from BigCompany clearly does not make an engineer an immediate fit.

Try to find an engineer with the right experience and the ability to get stuff done — this is what entrepreneurs are struggling with and is what drives the entrepreneurs and investors to complain that there isn’t enough talent in the market.

Startups are looking to fulfill a need. There’s no leadership development program to train a pipeline of technologist for future needs three years away. Startups have a problem. They need to grow their product. Now. Faster. If you’re not the solution to that problem, you won’t get in.

So, become the solution.

Pick the area that you want to work in. Ruby on Rails web development? Great. Objective C iPhone development? Sounds good. Now start writing code. Show that you get stuff done (code on GitHub, contribute to open source, blog!). If you have experience already, find ways to make it relevant to the position that you’re looking for.

And to really get a gold star, don’t limit yourself to the one technology that you start with. If you’re doing Ruby on Rails development, it’s likely that you may need to do some basic systems administration on a Linux server to get your stack up and running. Embrace that opportunity as a chance to prove that you can do whatever it takes to get the job done. Same thing when you need to setup memcached or Redis or whatever additional technologies might come onto your radar. Because a startup will expect you to pickup new technologies, and to become productive rapidly.

Finally, you need to get in touch – I think this point is under-appreciated: startups don’t have big recruitment search budgets (or a lot of time) – if we can’t find you easily, we can’t hire you. Promote yourself. You are a brand. Figure out how to get found.

I sent some very similar advice to my summer development interns at the end of August. I closed with the comment that, if you put some time into really considering the skillets that you’ll need in the job that you want, and then really make an effort to build those skillets (and any supporting skillets on your own), and then do the extra work to publicize what you’re doing, you will be easily heads and shoulders above 90% of the other students you”re graduating with who could do the work, but won”t bother.

Much has been said about the Gen Y folks who are eager to get out there and start making a difference in the world. You don’t need anyone’s permission to go get started. What are you waiting for?

Upgrade Ruby Enterprise Edition

A security update for Ruby has been released to respond to multiple vulnerabilities. All versions of the interpreter are affected, including the Ruby Enterprise Edition interpreters.

Fortunately, it’s easy to upgrade with RVM:

Tell RVM to upgrade your specific REE installation to the latest:

rvm update                                                   #upgrade RVM
rvm upgrade ree-1.8.7-2011.02 ree-1.8.7-2010.02              #upgrade REE

Accept Y to all options to move elements of your current install.

Then reset REE to be your default:

rvm ree-1.8.7-2011.02
rvm --default ree-1.8.7-2011.02

Following this, confirm that the new install of Ruby is active for your shell session (which ruby). Then, you will likely need to install bundler again to make sure you have 1.0.10 for the upgraded copy of gem 1.5. Also, run bundle install from the app root to make sure your gemset is up to spec.

Despite telling the upgrade process to move everything, I still needed to reinstall Passenger (3.0.2) and reconfigure it for Apache:

gem install passenger
passenger-install-apache2-module

With this reinstallation, you’ll also need to edit your Apache config file (on Mac OS X: /etc/apache2/httpd.conf ) to point to the new Passenger gem.

Restart Apache from System Preferences > Sharing > Web Sharing and you should be good to go.