Archive | Thoughts / Experiences RSS for this section

Real Travel Acquired by Uptake and The Back Story

Today is the day that it has become public knowledge that my company Real Travel has been acquired by Uptake Networks. It’s been over 5 years since I wrote the first line of PHP code for Real Travel. I wanted to share a bit about my ride so far and I look forward to even more excitement within Real Travel as part of the Uptake Network.

Real Travel’s Conception

Over 5 years ago I was introduced by to Ken Leeder by a mutual friend of ours. After a few breakfast meetings with a few of us (Michael T., Christina B., and Ken L.) we put together some wireframes and I went to work in the moonlight. Six months later we had a pilot/prototype that was built by me using PHP/MySQL with some help from a contract designer who gave me some photoshop designs. (Christina also helped with some of the design.) The application was quite simple then; we had lists of hotels in destinations and a review form for hotels.

A few more months later I quit my job [yes, the link no longer works.:)] and the company was incorporated, and our initial seed round of fundingwas coming together. This was also the time that we hired a new CTO and Chief Architect (who are no longer with us.) The decision was made to rewrite the pilot using the then beta ASP.NET 2. I pushed back but other were more familiar with Microsoft technologies and won the battle. Knowing now what was coming next, in retrospect I wish I had fought harder for an open source LAMP stack that couldn’t scale.

The Pain of Learning What You Already Knew (Again)

There was a “culture” clash in respect to development and design approaches. Within three months we went from my pilot’s 5-10K lines of PHPcode and uber simple database schema, you know the kind that you can mange from the command-line without a plethora of GUI tools, to over 100K lines of code of C# and a object oriented db schema in PostgreSQL (not the built in kind of object oriented that PostgreSQL provides, but a new home grown schema that was designed for fourth normal form.)

I will never forget the day when our architect showed the database schema relation map, it looked like a ball of yarn with so many relationship lines that the shapes of tables themselves were indistinguishable as rectangles. This was my first run in with doing it the “right” way the first time fallacy, at least to this extreme. As an engineer with then 5 years of experience I expressed my concern with the complexity but was quickly extinguished by the group-think cliches like “this is how the big guys do it.”

We started to release our site weekly and launched our site at the Web 2.0 Conference Launch Pad. The development was slow due to the complexity of the database. Not kidding, it took about 10 SQL inserts in order to add a single photo to the database; tables for strings, dates, root objects, photo, photo renditions, and so on.

During the year or two following…. a lot of “fun” happened”….. and our site’s architecture was extremely brittle and we spent a good part of our time debugging strange bugs, slow queries (with at least 5 if not 8 joins in them,) and strange IIS bugs. (A very educational experience for me building the ivory tower inefficiently.)

Born Again

I was playing with a new toy on the side called Ruby on Rails and became really enchanted in the framework and the Ruby language its self. I finally felt free from the 90 second compile time and memory cache load that ourASP.NET app required between code changes. It reminded me of thePHP/MySQL days and I realizes just how horribly over engineered our system was. I started to propose a rewrite but it went over like a lead balloon, but this did not deter me. I did freelance projects in Rails and PHPjust to keep my sanity. I believed in Real Travel and didn’t want to leave; I was drawn in by the opportunity to be the catalyst for positive change the second I was given a chance. The time would come.

As time went on and our releases became more distant apart, and our site became slower and slower, my challenge to rewrite the site became that much more appealing, but it was becoming a bigger task each day. Each time I had to change a line of code I had to wait (And wait) for hours to compile the app, start it and test it, then more hours and even days to release it, I would say “if this were Ruby on Rails it would have been done a while ago.” We knew as a company that something had to be done and as a team we were unable to develop new features and move our company forward.

As a company, we were forced by the market to change our midset, and accepted that something had to be done but a port to Rails was still out of the question. We attempted to de-normalize our tables and rewrite the code base in ASP.NET and C#, but that only proved to take even more time and we were still on ASP.NET.

A New Chapter With Ruby on Rails

It wasn’t easy, but I continued to champion the move to Ruby on Rails and we started to build all of our new development on it. We ended up having two applications, the main one was APS.NET the new one Ruby on Rails. With a use of a load balancer we were able to make much of this transparent and we found ourselves spending must of out time where it could count—on the Rails application. In fact there was a time where only one Windows development machine existed on a desk in case we had to make a change to the old system we could make the change, test it, and then push the code to production. After many many long discussions and debates we finally made the decision to make the port to Rails. It was also about this time that we decided that unless we started to use SCRUM we would either all quit or jump off of a bridge.

SCRUM, Agile, TDD/BDD, Quality, and Accountability

We had all learned our lesson. As a company we went to SCRUM training and this was one of the most pivotal points in my tener at Real Travel (or in my career.) We began to form good process and better working relationships. I begin to jump into rspec and getting the quality built in to the product it was a start, each day getting better and better.

About 11 two-week sprints later (579 story points) we had a new version ofReal Travel up and running on Ruby on Rails. I had the pleasure of powering down the last windows server—it was nice after all of the pain.

Nothing was stopping us: continuous improvement, open communication, retrospectives, developer productivity, and backlog directed self organized team. Within a year we ported the system and made major improvements to our site and even started to make some decent revenue. The summer of 2009 was great for us, major traffic gains, increased revenue, and then the Google problems started. Although I did the Rails development I could not have done it without Chris Sloan and Francisco Marin as team members, peers, and heros.

Google: The Authoritarian Mime

I won’t bore you with the details in this post, but due to some spammed pages in our site Google had decided to kick us out of their index and not tell us why. Fortunately after some time we found the problem (we found and deleted some pages that were link spammed with V1agra links and had 5K links pointing into them) and once it was fixed our traffic started to come back a bit.

Then months later we had another problem with a load balancer configuration which caused our old site’s links to become 404s. This was not obvious to us right away, but like the other Google problems, it was a problems for us and affected our traffic. We fixed the problem but we were cut deeply but the two complications. Our traffic started coming back along with our revenues, but it was slower and was a long rough ride. Traffic and revenues were going in the right direction however. We knew we could get our traffic back and get things going so we marched on.

Becoming “That” Team and Company

Through the couple of years we started to tune our SCRUM process and went from two-week sprints to one week sprints, one day planning meetings to 1 hour planning meetings, and one a week releases to releasing multiple times we day with continuous release. It started to become really exhilarating when we could run an on the street low-fi paper test with the kind people of Palo Alto, come up with some up hypothesis on a change, push out a split test (aka. AB test) and then release the winner in a day. If there was a bug and it affected our site, we could have the fix out in minutes.

What Now?

With traffic bouncing back we were engaged by Uptake and I will let Tech Crunch, and the many other blogs tell the rest of the story. I consider myself privileged to be able to learn with and from my fellow team members and ride the ebbs and flos from the first line of code that I wrote on that first pilot and I look forward to the future as Uptake and Real Travel aim to provide the best travel experiences on the web.

So, Sloan, pull the next test from the top of the test queue and lets test it!

FAIL: COMPROMISED SSH Public Key on Ubuntu

Last night I was setting up a new application on my server and while I was configuring capistrano I came across this strange problem and didn’t immediately find much help on google, so I thought I would post this to help someone else along.

I had already setup my ssh keys months ago but when I tried to ssh into my subversion repository it would ask me for a password/passphrase and it just about drove me crazy.

I came across this article in google and checked off each potential problem and nothing. Then I saw that my key was conprimized when I ran the “ssh-vulnkey -a” command.


capistrano@allison:~/.ssh$  ssh-vulnkey -a
Unknown (no blacklist information): 2048 5a:b4:d6:94:10:14:e1:a0:35:35:ff:c6:08:e6:9f:10 
Not blacklisted: 2048 5f:43:c2:f0:fb:e6:52:c4:90:59:fb:d2:e0:fe:66:d0 
Unknown (no blacklist information): 2048 ab:5e:39:5c:33:f0:02:e3:cf:cd:99:84:ca:9e:f8:e1 Paul@paul-hepworths-computer.local
COMPROMISED: 2048 81:85:1d:a7:b1:c6:ff:b2:d5:3f:60:3e:2e:c0:25:5c capistrano@mislice
COMPROMISED: 1024 fa:87:13:5f:0c:01:3e:53:b9:a1:ff:4a:8a:29:b2:a1 capistrano@mislice

So I searched google some more to find out how to fix the problem. I regenerated keys multiple times on my client server and no-dice.

Then after searching and searching I found this tutorial and followed it to update openssl and openssh and regenerate my private keys .

What a relief! (and a waste of time, but now I am secure I guess)

 

Destination Derby

A half hour ago I released a game that I designed with the help of a friendand the input from my team members. The game took just under three weeks from inception to completion, and I also moved with my family to a new apartment as well.

It was fun to work on this project and I hope that it is successful.

Oh, on a more technical note, I wrote the game using Rails 2.0 with no plugins and only a handful of tables. It was fun on every level.

Now here comes the real question… can you beat me to where ever it is you want to go? If you are reading this, please invite me to your race (Paul Hepworth/ dderby[at]omniop[dot]com.)

Here is the link: http://realtravel.com/ddracing

You can also get to the game by going to any RealTravel destination page and click the “play” button when prompted.

Thanks again Phil for your creative flare major contribution.

 

Camping in the front of the line for my Son

Below are photos of what I went through last night while camped out and waited to register my son for an extremely popular Dual Immersion program that had only 15 open spots.

I managed to get second in line by showing up at 9pm last night. As I hung out, socialized, and slept, other parents trickled in throughout the night and morning. There were hundreds who wanted a spot and we got one, in fact we were the first to get one. Hooray! It was very refreshing to see so many parents who wanted the very best for their children and were willing to sacrifice dignity, sleep, and a warm bed—all this for their children’s education.

The temperature dropped to freezing, but I hung tight with a handful of dedicated parents. The lines actually ended up wrapping the parking lot. It wasn’t easy, but it was worth it.

I woke up at 2:50am and thought I would capture the moment. The first sleeping bag is mine.
I was happy that I came the night before because the line ended up being extremely long.

Oh and I made the channel 2 news this evening. 🙂

 

Google’s FREE blog backup service!

A month ago this blogs MySQL database with InnoDB tables got totally hosed due to some unfortunate events and because it was just this blog I did not have a recent sql dump backup. Anyways, thanks to Google’s index cache I was able to recover most of my blog articles. I still didn’t get all of my comments back, but hey what do you expect from a FREE blog backup service from Google. It was still better than the one that I was paying for that didn’t work. 😉

Google's FREE blog backup service!

A month ago this blogs MySQL database with InnoDB tables got totally hosed due to some unfortunate events and because it was just this blog I did not have a recent sql dump backup. Anyways, thanks to Google’s index cache I was able to recover most of my blog articles. I still didn’t get all of my comments back, but hey what do you expect from a FREE blog backup service from Google. It was still better than the one that I was paying for that didn’t work. 😉

VPS restoration from backup kills your InnoDB database — don’t let it happen to you!

A month ago my then VPS provider, JaguarPC, has some really freaky hardware issues, that to this day I have no idea what happened and they ended up restoring a two week old backup of the whole server which included my VPS. When I fired up this blog and a couple of other sites they failed due to mysql table corruption. The corrupt databases that used Myisam tables seemed to repair just fine, but all of my InnoDB databases (Rails uses InnoDB by default when you use migrations) were unrecoverable and I ended up having to try other means for getting my data back or at least as much of my data that I could get back.

Here is what I learned:

  1. Never assume that your hosts backups of your VPS will work when they are restored because they perform backups while the server is running and databases don’t like that too much.
  2. Always keep backups of your databases, especially the ones that use the InnoDB table engine, in a SQL dump format.

So here is what I do now to prevent this from happening again:

  1. Perform your own backups of your databases using the methods that are suggested for your db and db table engines.
  2. Get the data into SQL so when your VPS is backed up it will properly backup a dump.

Assuming that you have a file that contains a list of databases with one per line, you can do something like the following and then hook up your script t cron.

#!/bin/bash

cd /var/lib/mysql

if [ ! -d sql_backup ]; then 
  mkdir sql_backup
fi

for db in $(cat databases.txt); do echo $db; mysqldump --single-transaction $db > sql_backup/${db}.sql; done

Good luck, and oh BTW, you might want to get this running on your VPSbefore your host does the restore. 😉

 

VPS restoration from backup kills your InnoDB database — don't let it happen to you!

A month ago my then VPS provider, JaguarPC, has some really freaky hardware issues, that to this day I have no idea what happened and they ended up restoring a two week old backup of the whole server which included my VPS. When I fired up this blog and a couple of other sites they failed due to mysql table corruption. The corrupt databases that used Myisam tables seemed to repair just fine, but all of my InnoDB databases (Rails uses InnoDB by default when you use migrations) were unrecoverable and I ended up having to try other means for getting my data back or at least as much of my data that I could get back.

Here is what I learned:

  1. Never assume that your hosts backups of your VPS will work when they are restored because they perform backups while the server is running and databases don’t like that too much.
  2. Always keep backups of your databases, especially the ones that use the InnoDB table engine, in a SQL dump format.

So here is what I do now to prevent this from happening again:

  1. Perform your own backups of your databases using the methods that are suggested for your db and db table engines.
  2. Get the data into SQL so when your VPS is backed up it will properly backup a dump.

Assuming that you have a file that contains a list of databases with one per line, you can do something like the following and then hook up your script t cron.

#!/bin/bash

cd /var/lib/mysql

if [ ! -d sql_backup ]; then
  mkdir sql_backup
fi

for db in $(cat databases.txt); do echo $db; mysqldump --single-transaction $db > sql_backup/${db}.sql; done

Good luck, and oh BTW, you might want to get this running on your VPSbefore your host does the restore. 😉

 

Subject: please cancel my account

I am looking to change VPS providers from JaguarPC to a company that has a higher level of service and uptime since I have had over 4 or 5 days of downtime and even lost some data due to the restoration form backup that killed my InnoDB tables in MySQL. (enough about that)

Anyways, i found a company called InMotion and after just a few days, here is my last trouble ticket.


To whomever it may concern,

Please cancel my account and refund the charges to my credit card as I would like to take you up on your 30-day money back guarantee.

In case you have questions about why, here is my best effort at giving you both an explanation and some customer feedback that I hope will be helpful to you as you improve your business.

Over the past couple of days I attempted to get going with you guys in hopes that I could move my current sites/customers over to a more responsiveVPS provider with better support, etc. I am sure you are a great company and provide great services to many customers, but to be honest I was not impressed with the process of signing up and how my initial tickets were handled.

1) I was surprised by the $2/ip as there is no mention of this pricing before signup and the sales page implies that you do not charge for ips as long as they are justified. Believe it or not, that was one reason why I considered you guys.

2) The support seemed very reactive overall and although I asked questions that could imply that I was relatively experienced with system administration, it seemed as though i was talked down to and was practically told to be grateful for your help and that I wasn’t charged for you services. (The “service” that I didn’t ask for of changing the port that sshd was going to run on. I only told you that I was going to be changing it in order to have you open up a hole in your firewall to accommodate my customization.)

3) By default I didn’t have root access, and after I was “approved” to have it, I was still unable to ssh into my VPS due to your network firewall. Then once I inquired about it, I had to explain that I needed ssh access and was asked for the last 4-digits of my cc number, and then the ticket was closed by a technician, as if my issue had been resolved. After opening the ticket again and providing my cc info, I was asked for the ip address that I was going to be sshing from so you could add a firewall rule that would allow me access, to which I replied that I accessed the server from many locations.

Long story short: I appreciate your help but I don’t feel like InMotion is where I should be.

Good luck and have a most pleasant day!


I don’t like posting negative stuff (unless its about Microsoft and Visual Studio,) but before I signed up with InMotion I searched google for stuff like “InMotion sucks”, and “InMotion bad”, etc hoping that I could see what other peoples experience were and couldn’t find much. I hope this helps other hackers who will do the same and maybe save them some grief.

And no, this was not a paid advertisement by InMotion. 😉

UPDATE: InMotion sent me an email apologizing for the miscommunications and asking me to reconsider my decision. I will admit that their email was very nice and that they wanted to keep my business and make sure that I am happy. I am sure that their services will be great for some people, personally, I am looking for a provider who is a little bit more interested in the big picture of what it is that I want to accomplish and get setup. When I have issues or problems, I need to be able to ask a specific question and get taken care of. Each support cycle is expensive for me. I think the issues that I am explaining are due to VPS companies whose primary business is shared hosting and their secondary business is VPS—their techs are trained to deal with the shared hosting folks. If I decided to stay it wouldn’t be a horrible thing I suppose, but I have already found an alternative VPSprovider ServInt.

I am sure that InMotion is a good company, but it didn’t work out for me.

People that do because they do

Last night I walked down the street with my family to view a train exhibit that had been put up year after year for over 30 years by a local train enthusiast in an effort to share his Christmas with neighbors, adults, and children alike. Two nights before that I drove by a couple of houses in Sunnyvale, CA that were covered in Christmas lights choreographed to music that could be heard by tuning your car stereo to a certain channel.

On both occasions I wondered to myself, who would have the time to spend on such projects year over year? Then I realized that this was not unlike all of the brilliant and willing software contributors who work tirelessly in open-source communities to build great software that other people can use, enjoy, and even use to make a living.

I am impressed with the passion that so many individuals have that causes them to DO what it is that they do. Its encouraging to see that people with passion can provide a great service to the rest of the world in small but meaningful ways.

While some people stand around wondering what can be accomplished or think about why things should or should not be done by them, others just DO.

So here is a thanks to all of the people that DO because they DO!

For every Ruby on Rails plug-in and gem, every Linux install and presentation given in a local user group—thanks!