How do you....?

Replicating the effectiveness of an onsite team is hard, very hard. So when someone who's been burnt by outsourced research and product development (or has heard horror stories) asks how we surmount this problem, it gets my juices flowing.

1. There is no substitute for sitting across the table and discussing stuff. There just isn't -- no amount of video conferencing and detailed specification is an effective substitute. So we bite the bullet and spend the face time with our customers. And we take on the responsibility to document and communicate with and visit our team.
   Our approach has some unexpected benefits for our clients.The luxury of an onsite team is sometimes used as an excuse to flounder with requirements - if you can walk over to the dev guys and change your mind on a whim, there's no incentive for rigor, is there? When you translate that to the time your expensive local resources spend thrashing, the cost equation just isn't pretty
2. Your best vendor is someone who's been a customer sometime in their lifetime. I've outsourced product dev to India multiple times. Some experiences were mildly satisfactory, others plain horrible. And the difference between a great experience and an awful one are in the little things you do - not rocket science. Those little things -- we pay a lot of attention to them.
3. Most management tools suck for outsourced dev.  We've used Jira/Atlassian, Basecamp, Trello, you name it. They were nice, and passable, and we still use some of them with some of our customers. Our most effective tool? - a homegrown kitchen sink webapp that provides a single consistent interface to email, sprints + deliverables, documents, statuses and messaging. Our teams have three things they deal with - the tools/IDE's that they use for dev and testing, the whiteboards around them -- we've covered all available wallspace in our office with them, and the aforementioned webapp. Simple, effective, not confusing. And for our customers - instant visibility.
4. Each project has a rhythm. We run agile with a very simple goal - put something in the hands of the user with clockwork regularity. The operative word is clockwork. The decision whether or not to deliver once a week, once every two weeks and so on makes a big difference to the outcome. Determining said frequency is a craft, and one that we've honed over time.

There are, of course, many other things that we do - hiring great people, our approach to problem solving and so on, but from a process perspective, this is what we do to ensure success.

Products or Services? A Crisis of Identity

Its a trap a number of services companies fall into. Develop an internal tool or a nice looking application, and voila! maybe this can turn into a product? Trust me, more companies make this mistake than you would think - and occasionally, very occasionally, one does ok in the market. If you've done it and failed, perhaps you'd be relieved to know that some of the big PS outfits have made that mistake as well?

So if your passion is to build product, you have two relatively hard choices (a) Look for external funding (b) Internally fund it from services. Now this is where the crunch comes in.  I had two big issues with external funding in the past when I was part of teams that took it (i) It came with a lot of baggage - a lot (ii) I'm one of the oddballs that has a tougher time spending other people's money than my own. So the question was, how do we internally fund product development? Here's what we do, maybe this helps you craft your own strategy?

1. Be rigorous about selecting what to productize. We go through several ideas a year, and maybe, pick one. This, after a lot of research, and potentially some money spent prototyping some of them.
2. Let go pet ideas unless you can very quickly monetize them. Be incremental in execution (read venturehacks  for a really good exposition on this).
3. Take on services work, but only if it involves a similar style of development to yours. We take on outsourced product development work, preferably in small teams - fits in great.
4. When in doubt, pick the better practice. For client projects, provide the same level of passion and creativity that you would your own efforts. For your own efforts, provide the same level of urgency that you would client projects.

Free to think

The thing to remember when you're outsourcing product development is that you're hiring an extension to your team. If they're a good outfit, trust them to be creative and give them the rope to be. If they're not, don't hire them. Good product development outfits are good problem solvers, its a cultural thing (company culture, nothing ethnic). That said, its amazing how often business arrangements come in the way of a healthy relationship.

An Anecdote
I have to wind back a few years -- we were consulting for a large engineering company that built complex electronics, automating their product configuration processes. Each job was custom configured - and they had painstakingly documented all their business rules in a massive spreadsheet. Enter all relevant data about a job into the spreadsheet, and it would spit out the right set of parts and labor. The problem was, by the time we went in there, the spreadsheet had grown to 4 MB of data and formulas, and they were adding tons of new information by the day.

Now here's where we came in. They had approached the leading light in the configuration software space, and they had come back with (a) 6 months of 2 analysts to extract and document the rules (b) 6 months of a 4-person team to implement a rule editing program. These guys couldn't wait that long, and wanted to know if there was a better way. Well, here's what we did (we got the spreadsheet mid-week)..

1. One of our guys spent the first day writing a simple VBA program that spit out all the data and formulas into an XML structure. The folks at the engineering company had been meticulous -- nearly every formula had been named! We filled in the few remaining gaps.
2. We wanted to understand the "meaning" of each formula, so the next step was to write a parser and substitute the names in the formulas -- so I built a test harness that would validate my parser.  
3. The next step was to find a good parsing engine - we settled on ANTLR -- its a great tool, but we realized we needed more than a few days to really figure it out. I took a chance and emailed Terence Parr, the creator of ANTLR, explained the situation to him, and asked if he could recommend someone. He promptly wrote back "this sounds interesting, and I have a couple of hours this weekend, send it to me".

On Monday morning we had all rules extracted and documented.  From then on it took a 2-person team a couple of months to build an editing interface, and they had a working extensible application.

Lessons for outsourcing product development? 

1. A professional services company simply doesn't have the incentive to be good problem solvers. It cuts into their billing time.
2. The best business arrangement is an extended team structure - a pre-defined monthly consolidated payment for the team, tied to deliverables. Your only decision should be continue or not with the team. The models that don't work are
   • Fixed bids - during the bidding process, the better vendors are at a disadvantage. Their bids come out higher because of the need to be conservative, and the lower bids presuppose some degree of cutting corners. Its a lot better if you, the customer, choose if and where to cut corners rather than the vendor. After a bidder is selected, you, the customer are at a disadvantage - because none of the benefits of creative problem solving accrue to you.
   • $/hr based on years of experience - provides all the wrong incentives to stuff resumes rather than smart, capable engineers.
3. Be prepared to work closely with the team, rather than throw expectations over the wall.

Our Approach to Rigor

Of all there is to teach, teaching people to think is the hardest, but the most rewarding.

The good part of this profession is that I get to do it every day. If I had my druthers, I would have my guys go through Knuth and solve all the problems in there - unfortunately, we are not in the business of paid sabbaticals. So I do the next best - I train them to think on the job.

So here goes a typical conversation - the subject in question is one of developers "D" - a really smart kid, and soaks up stuff like a sponge. Again, paraphrased conversation, and I've added color, but the gist of it stays.
-------------------------------------------------------------------------
D> I'm trying to measure perf on this app - and TFTP (the perf monitoring tool that comes bundled with eclipse) is slowing down the app - so I don't know if performance problems are because of the app, or the tool I'm using to monitor it.
Me> You know the specific things that you want to track, right?
D> Yup
Me> Use AspectJ
D (4 hrs and some back and forth later)> Ok, got it working - this is good stuff! How does it work?
Me> It does byte code injection
D> Where can I learn more about it? Are there books?
Me> Decompile the damn thing
D> What?
Me> Decompile the generated binaries
D> Why?
Me> You'll see
D (1 hour later)> Wow, I get it!
-------------------------------------------------------------------------

For those whose head this went over, my apologies. But here are a few things that any decent developer should be doing..

1. If you use a framework, walk through its source - most stuff that we use today is open source anyway - Tomcat/JBOSS, Spring, Hibernate, Flex. There is no excuse for someone who's spent a year using a framework, and hasn't walked through its source.
2. Know the concept of a control. Most folks trouble-shoot without creating a minimal representation of the problem. Basic science.
3. Think and write down stuff before you code. A lot of developers are notoriously bad at abstract thinking and writing. Practice.

The Mythical $10/hr Programmer

Numbers don't lie. An entry-level programmer in India earns anywhere between Rs.15,000 to Rs 25,000 a month. $500 a month. That, all said, is the equivalent of $3/hr. So why should you, when someone quotes you $10-15/hr for services, run like hell?

Here are some more facts:

1. Indian engineering schools - even the so-called good ones - generally do a poor job. We have an ongoing stream of incoming candidates - we look at 30-40 candidates per month and put them through a simple aptitude test -- 90% of them bomb on it. These are inherently smart kids who've been dumbed down by 4 years of appalling schooling, preceded by 12 years of an unhealthy rat-race.
2. Most decent outfits will invest in the engineers that they hire. The big guys run a finishing school, we throw them in the deep end and have our seniors mentor them - grunt work initially, progressing over months to things that actually add value. 
3. Engineers who've spent more than 3-4 years in a large outsourcing outfit are useless in a product development setting. They'll know how to configure Struts and Spring (and are proud about it!), but have no clue on how software really works. 
4. Salaries go up exponentially with experience - for the good guys. 

Which brings us to the economics of a real product development outsourcing gig. The only feasible way to run an effective product development team in India is to have an experienced lead (someone who's developed product before, ideally in the US) work with a small smart team. The good experienced leads are generally folks that have commanded 6-figure incomes in the US, and trust me, they make close to that in India. I'll let you do the math for the bill rates.

So how do some of outfits give you a great price like what I mentioned?

1. They've designed a much better mechanism than I've seen or can think of. There is that possibility. 
2. They've got a great mix in their team. 1-2 guys who actually do the work. 5-6 that just hang on. Multiple team size by "n" - and you have the typical outsourcing model. 
3. They're in the charity business 

Enough said.

Frequent Deliverables

This builds on an earlier post about 5 things to look for in a Successful Outsourcing Relationship. One of the five most important factors that dictates the success of your outsourced product development is the presence of frequent well defined deliverables.

Almost two decades ago, I had the privilege of teaching contrasting approaches to product development between Japanese and US companies in electronics. The Japanese companies invariably focused on having refined physical prototypes available as the product definition evolved. It allowed the various stakeholders to "touch and feel" the product before deciding on features and progress.

Good outsourced product development has many of the same characteristics. Insist on deliverables that are concrete and representative of final product. Some examples are:

1. Fully functional click-thrus
2. Documentation of business logic, ideally in pseudo-code form
3. Working prototypes of technical risk items
4. Data model with populated sample data
5. Performance numbers
6. Deployed product

These are just some examples, but make sure that they are scheduled frequently, and the ones with the most risk involved (requirements risk, technical risk), happen early in the project. Mix with transparency, and you're protected against partner failure to a large degree.

The Site Visit

This builds on an earlier post about 5 things to look for in a Successful Outsourcing Relationship. One of the most important things to do before you decide to send your work out to an outsourcing provider outside the country is to do a site visit. Let me explain.

My first experience running an outsourcing outfit was in Chennai, India. We had the top floor of a two-story building - there were goats tied up in the lobby of the building, we'd lose power at-least twice a day, and the network was shot - this was when the world was shifting from co-ax. It took me three months to get it all fixed (not the goats though - they belonged to the landlord, so the little goat pellets became an occupational hazard).

Fast forward to India of 2011, much has changed and there are professional outfits in India, but in some companies the culture hasn't changed. There is a lot you can say about how your work will be done based on the working conditions in the company. Some things you may want to find out, silly as they may seem:

1. Is your team working in sweatshop conditions? 
2. What kind of machines do they have?
3. Do they have adequate conference room facilities?
4. Are there adequate whiteboards, markers?
5. How is the work environment?
6. How is the physical security?

If your vendor doesn't care about much of these, lucky if he treats your work any better. So, organize a surprise visit -- call someone you know in the city, or we'd be glad to help if need be.

Why a surprise visit? I was in Technopark, Trivandrum, India visiting one of our partner outfits, when I saw an elephant with a full band outside a building. When I enquired, it appeared that the company in question was welcoming a prospective client. You can imagine how the office looked on the day.

So, before you hand over your hard earned money, give 'em a call and say - "my buddy is in the neighborhood, just 5 minutes away from you. He has a plane to catch in 2 hrs but has kindly agreed to drop in and say hi". If they try to wriggle out, run!

Transparency

This builds on an earlier post about 5 things to look for in a Successful Outsourcing Relationship. One of the five most important factors that dictates the success of your outsourced product development is transparency. Its not an abstract concept, let me explain precisely what it means.

First, a few basics about an "agile" project. Cutting through the fluff, here's what it involves (a great resource to better understand what it takes to have a great dev process is from joelonsoftware) :

1. On a weekly basis, the team (including you) agree on what to shoot for - features and functionality
2. The project lead then breaks this up into tasks -- each task generally not exceeding 8 hours of work - so for a 4 person team, you're looking at about 20 tasks
3. The developers create a spec for each task - not formal, but enough to explain what their approach is
4. They write code
5. There is a daily build setup to run around noon each day, including unit tests
6. There is a smoke test done by the test team for the build (should take no more than 30 mts)
7. There is a deliverable at the end of the week that goes through more rigorous testing

Transparency means:

1. At any time, you should be able to access the code base, see the checkins from each developer (there are plenty of near-free online repositories, we use unfuddle and love it).
2. The project lead should share the task list for the next week with you at the end of every week.
3. You should have access to the spec written by developers for each task - demand that it be done and made available to you.
4. You should have the daily build url. If you're located on the opposite side of the world from your team, thats wonderful - their smoke test issues and broken builds should be fixed at the end of their day, and beginning of yours.
5. You should have access to the bug list - and see that bugs are going through a clean process

This sounds like a lot, but once its setup in a company, its a piece of cake. Demand it.

A Pro-Active Project Lead

This builds on an earlier post about 5 things to look for in a Successful Outsourcing Relationship In my opinion, one of the five most important predictor of success is whether there is a strong player-coach on the ground acting as your Project Lead.

Here are the different configurations that various outfits will propose to you to manage your effort (a) A Project Manager and a Tech Lead (b) A "Senior" resource who will "oversee" your effort (c) A Tech Lead dealing directly with you (d) A hands-on Project Lead who will do part of the work.

(a), (b) and (c) are terrible options if you are trying to develop a product. Here's why:

Approach (a) 
(a) is the option most large outfits will provide you. Unless you have a lot of time on your hands and your schedules are in the order of years, this will drag on. Again, I'm assuming that you're trying to develop a product, not maintain existing enterprise software or add a few features to a mature product. You ask me why? It's common sense when you think about it. The role of a "Project Manager" in most outsourcing outfits is to make sure that the tech guys don't say something stupid to the customer. And they get very good at their jobs in the course of time. Which means that you, the customer, don't get to know about significant risk items till the very last possible minute.

Approach (b)
(b) is an interesting approach. Outfits that make this proposal will usually have one or two very competent folks with a lot of experience - they usually are trying hard to get to be like the big guys, except they have to bid lower. They will blow you away with their knowledge during the pre-sales part of the conversation. The catch is -- they will spend little or no time on a consistent basis on the project.Can't blame 'em -- they're normally responsible for a whole bunch of things in the company -- pre-sales that you just saw, hiring, fire-fighting for larger clients. The "oversight" initially is promised with the best intentions; in reality, its hard for even the best to go through the code and understand issues in a way that serves your purpose.

Approach (c)
(c) is what you get when you hire a team out of a garage. They work great for small well defined "efforts". You want to build a product with this team? I've done it in my past life -- except that the best engineer in my team sat on top of the outsourcing outfit day-after-day, reviewing every task and every line of code. I had to promise to never put him on such a project ever again to keep him on in the company.

Approach (d)
(d) is what you need if you're serious about developing product using a remote team. One of the product companies that I co-founded started with the core team in the US, and outsourcing to a vendor in India. The vendor team was smart, but the lack of leadership within their team coupled with shifting requirements on our part made it chaotic. We wised up, and our team in India now consists of a top-notch engineering lead hired from the US, and a team that he's personally hired and put in place. Our costs have gone down 60% and the quality of our product has improved significantly.

HOW DO YOU TELL IF YOU HAVE A PROJECT LEAD?

These are requirements that you can be upfront about with your vendor, when discussing expectations. The project lead must :

1. Be able to carry on an intelligent conversation with you and add value to any functionality discussions
2. Know the codebase well enough to answer detailed questions from your technical team
3. Have on the tip of their fingertips, detailed tasking data for each team member
4. Be available (their) waking hours with less than 1 hour notice

Quick and Easy Status Reports

Some vendors and freelancers frustrate me. Their work is good, they are easy to get along with -- but their status reporting sucks. A good and regular status report is invaluable to a project -- and averts many headaches, so I figured I would post the format that we use for our weekly status reporting.

But first, here's the template I found from office.microsoft.com

YUCK!

Here's what we use, modify as appropriate, but you get the general gist..
----------------------------------------------------------------------------
Status report for Project ABC, for week 8/1/2011 to 8/8/2011

*What we got done this week

1. Got deliverable1 done
2. Got deliverable2 done
3. deliverable3 is pending

*What we plan to do next week

1. Finish deliverable3
2. Do deliverable4
3. Start on deliverable5

*What we absolutely need from you this week

1. We ran into roadblock1 - the two options to overcome it are option1 and option2. Which would you prefer?
2. Do you want feature1 in pink or blue?

*How we are doing w.r.t. overall project

1. We are behind by 1 week
2. We have spent n hours, and the total billing to date is $m

----------------------------------------------------------------------------

Simple. Easy. Effective.

Be Absolutely Clear about the Expected Outcome

This builds on an earlier post about 5 things to look for in a Successful Outsourcing Relationship In my opinion, the most important predictor of success is whether you (the customer) are absolutely clear about the expected outcome. So you were probably expecting me to expound on how to create a clear functional spec, and about how to scope everything out clearly, and not leave anything to imagination.

Nope. That's not your job as a customer. Your job is to know what you want and be decisive.  Your job is to give clear and prompt feedback when presented with options, and provide clear answers to functional questions. Not just when you kick off an effort, but in the course of executing it. And for god's sake, don't invest too much time in writing detailed specifications -- sounds counter-intuitive -- but I promise I'll explain it in a future blog post.

Sometimes I have friends or friends of friends asking me for help or pointers, and one such request came the other day. The gentleman in question (lets call him "G") is a builder of green housing in Bangalore, and wanted a website that was different and stood out. Our back and forth went like so (I've paraphrased and added color, but haven't taken away from the gist) ..

--------------------------------------------------------------------------------
G> "Need to build a website that stands out from my competitors. Do you know someone who can help me build it?"
Me> "Check out www.templatemonster.com -- any you like there? There are folks that I can point you to that will customize a template anywhere in the range of $200-1000, depending on what you want"
G>"Been through it before, I really need something more unique"
Me>"That could turn expensive -- you're looking at 2-5K"
G>"I know"
Me>"Ok - here's the name and contact info for our partner outfit. They are phenomenal, but just as I told you, expensive"
G>"Thanks! I heard what you told me about expense the first time"
G(a week later)>"These guys are good! I was able to provide them basic directions and they not only got it, but went above and beyond, Thanks!"
--------------------------------------------------------------------------------

I like working with people like G. They know what they want, and why. They ask questions and ask for clarifications, but don't leave you hanging for decisions, or waffle on decisions already made. And many times during a project, especially when developing a product, you run into opportunities or roadblocks -- some require time to think through and reflect, but most can be disposed off quickly if the stakeholder's objectives are clear.

So, when you kick off an outsourcing effort, make sure that you have (a) written down whats important to you in terms of outcome, briefly and clearly, and shared it with your vendor, and (b) make sure that you evaluate, and require your vendor to demonstrate periodically, that you're on course to achieve the outcome you desire.

Setting up a Technology R&D culture

Part of retaining and improving the competence people in any company is to give them the ability to learn new things. Most companies do it in a half-hearted way and as an afterthought, giving a day here or there for employees to tinker with new stuff. And it defeats the purpose to put structure and measures on an essentially creative activity.

So here's how we are going about building up this capability. We zeroed in on the following principles to put together our approach..

1. There should be well defined goals, but no formal processes in place. Think of it as internal skunkworks.
2. The activities should be team driven - its not just about one lone guy tinkering in a corner.
3. There should be some urgency and time boxing, but no hard deadlines

Will it work? Our first effort is focused on google v8 and node.js -- its interesting and different, and we think its potential is a little more than just building a scalable network app. The goal is:

1. To build an interesting app using v8 and node.js
2. that highlights its core features
3. and delivers insight into its future possibilities
4. and can be sent out to potential customers as a proof of our capability

We've given ourselves 3 months of our spare time to do it -- and will post the results as they happen..

Five things to look for in an Outsourcing Relationship

Google "Outsourcing to India", you'll likely run into this old article in the CIO magazine. Wait, wait, don't surf away because VERY LITTLE of this applies to you if aren't a Fortune 500 company or thereabouts, and especially if you're trying to get a PRODUCT built. Checklists with 500 variables to cull from 50 vendors? Multi-year deals? CMMI level 5? You must be kidding.

I've been on both sides of this supply-chain, we extensively used providers when I ran my outfit in Atlanta, including elance and guru.com. A stint back in the '90s and since a year ago, I've been on the supply side based in Bangalore. Well, I looked around, and couldn't find anything halfway decent that explains in a nutshell what to look for to find and manage a good partner. No wonder, you find these horror stories that proliferate in blogs…

---------------------------------------------------------------------------------------------------------------------------------
Here's an excerpt
---------------------------------------------------------------------------------------------------------------------------------
Some of the problems we encountered:

• Communication. You will never hear about problems until it is too late. It's a cultural thing. They lose face if they let someone know they don't know what they are doing or that the result is below par. The problem is that there is no way you can manage this and you cannot take actions. You end up with either a dropped project or a bad product. You need to have someone you trust on the ground.
• Infrastructure. Our team was in Mumbai. Many in high tech use Bangalore. The Internet and phone connections were awful and dropped four to five times a day.
• Loyalty. We noticed a dramatic change in resources constantly moving. People were always leaving for another 20% salary increase. It's all about cash and who will give you more.
• Micro Management. If you are to avoid some impact on lack of information sharing about any problem you must micro manage the project. Preferably, you need someone there full time, which we couldn't.
• Cost. Well, it looks cheaper on the surface but caveot emptor. In the end, this cost us more and we were throwing money away. Furthermore, it is not that cheap any longer.
• Quality. In short, we discovered that the code developed in India was either poor or eventually thrown away. Not very well invested time and money.

---------------------------------------------------------------------------------------------------------------------------------

Some of the problems have been mitigated with time -- infrastructure issues to an extent. Some simple checks and balances, however, would have been well worth the above gentleman's time, so here's is my list (sound like motherhood and apple pie, but aren't as obvious as they look on face value, so could use some elaboration that I'll provide in subsequent blog posts).

• Be absolutely positively clear about the expected outcome.
• A proactive project lead on the ground is worth their weight in gold.
• Demand full transparency
• Show up at your vendors offsite or offshore office, or have a friend show up without advance notice
• Chop the project up into well-defined frequent deliverables.

Notice I didn't mention "cost". Nor "expertise". Notice I didn't go anywhere near "size". They all matter, but none of these is likely to get you in deep doodoo as the five things above.

The Intellectual Property Scare

We start most interactions with prospective customers or vendors with a duly crafted Mutual NDA - for larger clients its mostly a CYA, while for the smaller outfits, I guess it provides a sense of security. That said, I've never had the occasion first or second hand to see any substantive breaches of these by anyone I know.

So when prospective customers do make a big deal out of it when sending out work to India, I'm frankly puzzled. Puzzled not because its not an issue -- there are horror stories, but because its all about common sense management of risk. So here are some things that will keep you safe so that you can focus on things that matter - like building better quality product and selling it to a large number of people.

#1. Honestly evaluate if you're at risk
I had a prospect who spent several days of his time and mine trying to figure how to protect his code base. His application? Available on the web for $19.99 per month unlimited usage. If I were a legitimate competitor, stealing his codebase would be the last thing on my mind.

#2. Know what's at risk
Years ago, we spent a lot of time consulting for large Telcos; at one of them, I was appalled to see the ease with which I could access its customer care information - including social security and credit card numbers. And this at a place teeming with contractors. They got lucky, nobody put that data on a CD and walked out.

When we outsourced our work, we spent a few hours classifying our assets into stuff (a) that could be sent out of the company with minimal risk - about 80% of what we had (b) needed to be obfuscated or dummied up - about 15% (c) under absolutely no circumstances should someone outside the core team have access to it - about 5%. It wasn't hard to do, and buys you a lot of peace of mind.

#3. Know your vendor
Its easy to do a reference check on your vendor. And ask a few questions about their internal practices - do they have separate repositories for each client/project? What kind of access restrictions and tracking do they do? What physical security do they have in place? Have someone local to help you do some due diligence - it's not hard to do.


These are steps you should be taking anyway, whether you're outsourcing or not. Maybe the possibility of sending your work out brought these issues to the forefront - think of it as a way to better your own practices than as an aggravation - your IP is important to you, isn't it?