written by Paul on December 28th, 2010 @ 12:02 PM
Over ten years ago at the start of my career I was vehemently opposed to “process” and mostly thought that a process was for those who could not be independent and relied on slowing down to mediacracy in order to ensure that the whole organization was synchronized.
Now that I am looking back I realize that I was right and wrong. I have learned that processes are the only way to breakout of mediacracy. I would like to share what has changed my outlook and discuss the pitfalls of applying processes the wrong way.
A process must…
1) be changeable by everyone who operates within it:
Everyone within the process must use their observations and power of perspective to “own” of the portions of the process that they interact with. Nothing is more destructive to a team’s productivity or an employees morale than seeing opportunities to improve, proposing them, and then being told by “the powers that be,” that their ideas are good, but [put traditional management smoothing line here.] This is what caused what Frederick Taylor referred to as “Soldiering” in his paper, The Principles of Scientific Management—people, when “put in their place” will do the minimal in order to get paid but avoid “punishment.”
2) be seen as a way to instill discipline within a team:
Lets face it, we are all different and some of us have more discipline (or other kinds of discipline) than others and even the most well-meaning and disciplined person can quickly become derailed when under the pressure from investors or the excitement of a new idea. Because of this normal tendency that each of us have, a process can ensure that rational thought and perspective is continuously applied.
In XP there is a notion of pair-programming—this tool does much to keep two minds around the same problem, where one might slip off into an interesting but unnecessary deviation within a project, the other can pull the reins and so the pair become more productive because they are guiding each other as a team.
Looking back on my life the times when I was int he best physical shape and frequented the gym was when I had a friend to train or work out with. The dynamics of a process or team culture can take a team very far.
3) allow and even encourage experimentation and testing:
When a potential opportunity for making the process better arises, proposals for improvement must be respected and the ideas should be tested. Like all empirical process, the results should speak for themselves. In fact there is a powerful side effect of testing that is not to be ignored. The Hawthorne Effect illustrates how by simply knowing that you are being measured you perform at a greater level. The idea or process change itself may not directly increase productivity, but just like how a pedometer can get one to move more to lose weight, knowing that you are being tested increases ones focus.
4) not driven top-down or bottom-up, but both:
Right below being told you can’t do something (referenced in #1 above) on the list of productivity killers is being told what to do. In a perfect world your whole company would all have the same epiphany on exactly how to conduct the their agile process, but in the real world, we need to share our ideas, persuade, and be persuaded. Kent Beck has said “The best architectures, requirements, and designs emerge from self-organizing teams”, and when people are empowered people care-less about position warfare and individuals are more likely to take feedback and ideas from the “top” and vice-versa. Processes that evolve on principles of efficiency and are aligned with organizational goals are always going to be more successful than those that are imposed.
5) increase productivity:
If any aspect of a process does not contribute to the productivity and health of an organization or team then they should be changed. One of the beauties of Scrum is the focus from the team and organization on “Inspecting and Adapting.” Process changes should always result in greater productivity!
When I was younger and more allergic to the word “process” I was right to feel that way because many of my observations of what “process” was presented as were indeed dysfunctional applications of agile (or not) “processes.”
Some companies like 37Signals denying the existence of a “process” because of their “process” allergy. They often talk about not having a process, but what they don’t convey properly to their audience (and I think its unfortunate) is that they DO have a process, though not written down, that is deep within their culture. It’s great when the process is natural and doesn’t require a wiki page to describe it, but for most teams and organizations a common vocabulary and understanding of the what, when, and how’s is important because some people need to adapt to a new culture.
Now that I have experienced the benefits of having just-enough process to get hyper-productive my allergies to the word itself have ceased.
Today is the day that it has become public knowledge that my company Real Travel has been acquired by Uptake Networks. It’s been over 5 years since I wrote the first line of PHP code for Real Travel. I wanted to share a bit about my ride so far and I look forward to even more excitement within Real Travel as part of the Uptake Network.
Real Travel’s Conception
Over 5 years ago I was introduced by to Ken Leeder by a mutual friend of ours. After a few breakfast meetings with a few of us (Michael T., Christina B., and Ken L.) we put together some wireframes and I went to work in the moonlight. Six months later we had a pilot/prototype that was built by me using PHP/MySQL with some help from a contract designer who gave me some photoshop designs. (Christina also helped with some of the design.) The application was quite simple then; we had lists of hotels in destinations and a review form for hotels.
A few more months later I quit my job [yes, the link no longer works.:)] and the company was incorporated, and our initial seed round of fundingwas coming together. This was also the time that we hired a new CTO and Chief Architect (who are no longer with us.) The decision was made to rewrite the pilot using the then beta ASP.NET 2. I pushed back but other were more familiar with Microsoft technologies and won the battle. Knowing now what was coming next, in retrospect I wish I had fought harder for an open source LAMP stack that couldn’t scale.
The Pain of Learning What You Already Knew (Again)
There was a “culture” clash in respect to development and design approaches. Within three months we went from my pilot’s 5-10K lines of PHPcode and uber simple database schema, you know the kind that you can mange from the command-line without a plethora of GUI tools, to over 100K lines of code of C# and a object oriented db schema in PostgreSQL (not the built in kind of object oriented that PostgreSQL provides, but a new home grown schema that was designed for fourth normal form.)
I will never forget the day when our architect showed the database schema relation map, it looked like a ball of yarn with so many relationship lines that the shapes of tables themselves were indistinguishable as rectangles. This was my first run in with doing it the “right” way the first time fallacy, at least to this extreme. As an engineer with then 5 years of experience I expressed my concern with the complexity but was quickly extinguished by the group-think cliches like “this is how the big guys do it.”
We started to release our site weekly and launched our site at the Web 2.0 Conference Launch Pad. The development was slow due to the complexity of the database. Not kidding, it took about 10 SQL inserts in order to add a single photo to the database; tables for strings, dates, root objects, photo, photo renditions, and so on.
During the year or two following…. a lot of “fun” happened”….. and our site’s architecture was extremely brittle and we spent a good part of our time debugging strange bugs, slow queries (with at least 5 if not 8 joins in them,) and strange IIS bugs. (A very educational experience for me building the ivory tower inefficiently.)
I was playing with a new toy on the side called Ruby on Rails and became really enchanted in the framework and the Ruby language its self. I finally felt free from the 90 second compile time and memory cache load that ourASP.NET app required between code changes. It reminded me of thePHP/MySQL days and I realizes just how horribly over engineered our system was. I started to propose a rewrite but it went over like a lead balloon, but this did not deter me. I did freelance projects in Rails and PHPjust to keep my sanity. I believed in Real Travel and didn’t want to leave; I was drawn in by the opportunity to be the catalyst for positive change the second I was given a chance. The time would come.
As time went on and our releases became more distant apart, and our site became slower and slower, my challenge to rewrite the site became that much more appealing, but it was becoming a bigger task each day. Each time I had to change a line of code I had to wait (And wait) for hours to compile the app, start it and test it, then more hours and even days to release it, I would say “if this were Ruby on Rails it would have been done a while ago.” We knew as a company that something had to be done and as a team we were unable to develop new features and move our company forward.
As a company, we were forced by the market to change our midset, and accepted that something had to be done but a port to Rails was still out of the question. We attempted to de-normalize our tables and rewrite the code base in ASP.NET and C#, but that only proved to take even more time and we were still on ASP.NET.
A New Chapter With Ruby on Rails
It wasn’t easy, but I continued to champion the move to Ruby on Rails and we started to build all of our new development on it. We ended up having two applications, the main one was APS.NET the new one Ruby on Rails. With a use of a load balancer we were able to make much of this transparent and we found ourselves spending must of out time where it could count—on the Rails application. In fact there was a time where only one Windows development machine existed on a desk in case we had to make a change to the old system we could make the change, test it, and then push the code to production. After many many long discussions and debates we finally made the decision to make the port to Rails. It was also about this time that we decided that unless we started to use SCRUM we would either all quit or jump off of a bridge.
SCRUM, Agile, TDD/BDD, Quality, and Accountability
We had all learned our lesson. As a company we went to SCRUM training and this was one of the most pivotal points in my tener at Real Travel (or in my career.) We began to form good process and better working relationships. I begin to jump into rspec and getting the quality built in to the product it was a start, each day getting better and better.
About 11 two-week sprints later (579 story points) we had a new version ofReal Travel up and running on Ruby on Rails. I had the pleasure of powering down the last windows server—it was nice after all of the pain.
Nothing was stopping us: continuous improvement, open communication, retrospectives, developer productivity, and backlog directed self organized team. Within a year we ported the system and made major improvements to our site and even started to make some decent revenue. The summer of 2009 was great for us, major traffic gains, increased revenue, and then the Google problems started. Although I did the Rails development I could not have done it without Chris Sloan and Francisco Marin as team members, peers, and heros.
Google: The Authoritarian Mime
I won’t bore you with the details in this post, but due to some spammed pages in our site Google had decided to kick us out of their index and not tell us why. Fortunately after some time we found the problem (we found and deleted some pages that were link spammed with V1agra links and had 5K links pointing into them) and once it was fixed our traffic started to come back a bit.
Then months later we had another problem with a load balancer configuration which caused our old site’s links to become 404s. This was not obvious to us right away, but like the other Google problems, it was a problems for us and affected our traffic. We fixed the problem but we were cut deeply but the two complications. Our traffic started coming back along with our revenues, but it was slower and was a long rough ride. Traffic and revenues were going in the right direction however. We knew we could get our traffic back and get things going so we marched on.
Becoming “That” Team and Company
Through the couple of years we started to tune our SCRUM process and went from two-week sprints to one week sprints, one day planning meetings to 1 hour planning meetings, and one a week releases to releasing multiple times we day with continuous release. It started to become really exhilarating when we could run an on the street low-fi paper test with the kind people of Palo Alto, come up with some up hypothesis on a change, push out a split test (aka. AB test) and then release the winner in a day. If there was a bug and it affected our site, we could have the fix out in minutes.
With traffic bouncing back we were engaged by Uptake and I will let Tech Crunch, and the many other blogs tell the rest of the story. I consider myself privileged to be able to learn with and from my fellow team members and ride the ebbs and flos from the first line of code that I wrote on that first pilot and I look forward to the future as Uptake and Real Travel aim to provide the best travel experiences on the web.
So, Sloan, pull the next test from the top of the test queue and lets test it!
A good friend of mine years ago used to use a command-line app called ytalk to show me around the bash shell (thanks Sione!). After a short while I stopped needing his help and so I stopped using ytalk. At work we really wanted to shell-share with remote team members who were unable to use the iChat screenshare because of OS and bandwidth limitations.
I remembered that ytalk was such a good tool for being able to see what someone else was doing in the shell and to show off your bash skills. I thought it was going to be easy to setup on Ubuntu, but as it turns out, although its still an available package, it is dead on install.
So…. here is what I ended up doing and I hope that if you do the same you will be ytalk’in in no time..
On ubuntu install ytalk:
sudo apt-get install ytalk
Change the default/broken inetd.comf configuration:
talk dgram udp wait nobody.tty /usr/sbin/in.talkd in.talkd ntalk dgram udp wait nobody.tty /usr/sbin/in.ntalkd in.ntalkd
talk dgram udp4 wait root /usr/sbin/in.talkd in.talkd ntalk dgram udp4 wait root /usr/sbin/in.ntalkd in.ntalkd
Note the “4” after the udp and the “nobody.tty” change to “root”
In the /etc/services file, make sure the following lines are in there:
paul@box:~$ sudo grep talk /etc/services talk 517/udp ntalk 518/udp
I didn’t have to change anything, but its a good idea to confirm things.
Initiating the chat:
You can do this in a couple of ways, the first and most obvious way is to coordinate with another person/user and ensure that the two of you are only logged in once to the same box ad then type.
paul@box:~$ ytalk fred
Or if your logged on more than once you can specify the tty in the request after finding out which one it is:
paul@box:~$ who fred pts/0 2009-11-06 10:50 (208.X.X.X) fred pts/2 2009-11-06 10:48 (208.X.X.X) paul pts/3 2009-11-06 14:02 (208.X.X.X)
More on that can be found here: http://manpages.ubuntu.com/manpages/intrepid/man1/ytalk.1.html
Thanks to euphemus for the breakthroughs!
Hope you find ytalk as useful and coolific as I do.