Over the last 15 years, I’ve heard dozens of stories about the highs and lows of hiring offshore developers. The appeal is obvious, especially for a startup tight on cash. The dangers are also obvious – your team is a world away. After Cashman and I analyzed all the angles, we thought it made sense for Yesware. We did everything we could to make it work. Earlier this week, after two months of trying, we pulled the plug. Team India Fail.
There are many reasons why hiring remote contractors made sense for us:
1. We’re a distributed team already: Cashman lives half-time in Brazil, half-time in Boston. We had already assembled a motley but talented crew from Portugal, Utah, Boston, Bangladesh and points in between. So we were used to developing over the Internets. We use the pretty awesome Assembla for combined code repository and comments/tickets/documentation. We had found contractors through ODesk already, which mostly worked out. So we were already pretty up to speed on the requirements of international software development.
2. We chose a less-cool/more-established/less-expensive technology platform: Rather than go with the sexy Rails, we decided to build our backend on PHP/Zend. This choice was somewhat driven by the wide variety of open source email webapps available to build on. But we chose PHP mostly because we figured there would be a wider pool of international (read “inexpensive”) development talent available to help us build Yesware.
3. We vetted the partner carefully: We never would have considered hiring a whole outsourced team (no, I’m not going to say who it was) if they hadn’t come highly recommended by a close friend of the company. Just the idea of wading into the insanely large pool of international development firms without a guide gives me a stomach ache. But this company had worked with our friend’s past three companies.
Our friend had spent hundreds of thousands of dollars with them. He was currently using them for 3 separate projects. So not only were they giving us the volume discount, they were treating us like their biggest customers. I figured that this behavior would translate directly into how they selected team members for our project. We also reviewed resumes and interviewed (over Skype) each of the proposed team members. We thought we were covering all the bases.
We did our best to get the project off on the right foot. Cashman spent a week in India with the team, getting them up to speed on the project, the platform, the expectations, etc. Beforehand, he exhaustively documented his approach, other contractors’ work, coding standards, etc. When he returned, he was continually available, answering questions, asking questions, basically behaving like a diligent software manager.
Two weeks in, we were getting worried. The team lead was not stepping up to communicate at the level we hoped for. He was apparently having family issues which were distracting from his work. One of the company’s senior managers got heavily involved and promised to keep things on track until the lead could return.
Doubt Creeps In
Four weeks in, we were very worried. Communication was the biggest issue. It was difficult to get our team members to communicate in a granular and routine fashion. We asked repeatedly (and in various media and ways) for the team to add copious notes and comments in Assembla, to write wiki pages, to email, to call, to chat, etc., but it was like pulling teeth. With a 6-member team and an 8.5 hour time difference, we expected to wake up to a big stack of comments, but most days there were only one or two, and many days none. By the end of the day, perhaps there were a few more.
Team India’s tendency seemed to be to say nothing to us or Assembla until they were “done” with a task, at which point we’d have to dig into it to figure out what really happened. Quite a bit of the communication ended up going through the senior manager, with whom Cashman chatted daily. This had good and bad aspects — on the upside, he could act as a translator (both cultural and linguistic) and our agent on site. But on the downside, it was an extra layer. The workers weren’t very comfortable talking directly with Cashman, and we were never sure if things are really okay (but quiet), or if there was some question that we could answer or a problem that needs to be addressed.
Progress was not fast. Only a small handful of tickets had been fully completed since kickoff. A larger number of tickets had significant work done, but we expected to have to ask for significant changes to really finish them up. There were several contributing causes for the lack of measurable progress. Two weeks in, we switched from SVN to Git, which was a fiasco. Team India had no experience with Git, and their on-the-job training caused several days of fire-drills.
Our technical lead continued to have family issues and be basically part-time. He couldn’t step up into the kind of leadership we needed. We heard from another team member that his daughter was sick and in the hospital — which was terrible. We absolutely sympathized, but the work just wasn’t getting done. On the positive side, several of our team members seemed enthusiastic, and they had written a fair amount of code. They seemed to be working hard. At times, people stayed late to meet deadlines — one member worked past midnight to deliver a new demo feature.
Six weeks in, we had a major conference with one of the company founders. I was as clear as I could be that we needed to see some rapid improvements in the communication level, that we needed a new team lead, and that we needed to make much better progress on the tickets.
After that meeting, things started to improve. We got a new team lead and communication was better… until everything caught on fire and corkscrewed into the ground
This guy is a junior developer, less than 2 years out of college, who approached us after he saw our job post on Stack Overflow. #Winning. Two members of Team India team had been working on our browser extension for 1.5 weeks, and hadn’t made nearly this much progress. And on the following Monday, when Cashman updated their tickets to move them off the browser extension onto other tasks, they still spent another day on those tickets, apparently not understanding that he didn’t want them to work on it any more.
The comparison between these two was stark: a developer who makes huge progress, is easy to work with, and has a very short cycle time vs. two devs who still haven’t ramped up and require careful and repeated instruction. This is ce rtainly partially a matter of skills (the new dev had experience making browser extensions), and partially cultural (easier communication between him and Cashman), but it’s not an issue of being remote, as the new dev was remote as well (he’s in Chicago, while we’re in Boston).
Second, there was a commit of new code from Team India, and associated ticket comments, that made Cashman seriously doubt the technical competence of the team members involved. The commit involved code from two members (which they had previously, and privately, merged) and a comment from the new lead. This commit was to get a major new feature working after roughly two weeks of work. In short, the code was terrible. It was needlessly complex, badly formatted, and implemented a crazy control flow that included creating a whole HTTP client inside the web server to fetch a different URL from the same webserver (!), which in turn was configured by hardcoding the developer’s own machine name in the code in four different places. Awful decisions from wall to wall, as far as Cashman could see. But what really hurt was that the two senior people on the team had signed off on this code. The thought that they would endorse this code was a major red flag. Cashman didn’t know why they didn’t notice how bad this code was, but in any case, his trust was irrevocably shaken.
I called one of the founders of the company the next day and told her that we were done. It was a painful conversation after a difficult period of trying to make it work. But sunk costs are sunk costs. The worst mistake you can make is to keep piling good money on a roaring fire. Could we have done things better? Sure. Would they have been able to make the arrangement work? I don’t think so.
In the end, adding the cultural and communications issues on top of the general difficulties of software development make outsourcing a very tricky proposition. For companies that are well established, have definable tasks on standard platforms, perhaps off-shoring a portion of your development effort makes sense. But for startups trying to find product-market fit, unless you have an extremely rare and unnatural advantage or get very lucky, hiring a remote team is just not going to work.