renzhi

In the last two weeks, under the invitation of two friends, I have taken over a development team in disarray. The planned release date of the project was set for April 1st (no, it was not a joke), but by mid April, less than 20% of the work has been done.

To be honest, I have been serving as a technical consultant to the team since last November. That was when the project started. My role was to help with the system architecture design, analysis, algorithm design, etc. I would go in one day per week, to attend meetings, or help with any problem, if necessary. Under the agreement, I was also to design and implement a special algorithm, due to  the fact that it would too much for the team to do. The second reason was that no one in the team was able to design and implement it. The third reason was that this component is more or less independent from the rest of the system, and it would be easy to just plug it in at the end.  But it could be plugged in only after we have an integrated system.

I had a first version implemented before the Chinese New Year (that was at the end of January), and our plan was to have a first integration before the national holiday. Well, the project wasn’t ready for integration. During the holiday, I rewrote the code, improved it quite a bit. I have waited another month, and the project still wasn’t ready for integration. The work I’ve done had never had a chance to integrate into the system.

During all that time, I recommended to hire a real project manager with leadership. It was obvious to me that each team member has his fiefdom, and the in-team communication is broken.

A few weeks ago, I accepted the invitation to take over the leadership of the project, and my major task is to push the release out the door. The first week of project status evaluation was a scary revelation. My initial estimation was 20% completed. I was so wrong.

I spent the first week sizing up the project, re-evaluated the work tasks, and got a better grasp of the project and team situations.

The conclusions are typical of a messed up software project. I can almost see the smile on your face now, for those who are in this industry long enough. Here you go:

1. Lots of developers are self-proclaimed rock star, with their own fiefdom. We only wanted to hire the smartest and the most experienced. As a result, we got a handful of cowboys.
2. Communication is broken, they barely talked to each other between team members in the same group, let alone team members between different groups.
3. The API specifications document is not fully published. Only a few functions are on the internal wiki. Parameters are not clear, return values not fully described, error handling situations not even mentioned. I mentioned about error codes, and I got a blank stare. What error codes? Calling the function will always succeed. That was the answer.
4. Developers who needed the API all had an excuse: the API was not ready, so they sat on their ass, waiting.
5. By mid April, which was already past the project release schedule, we had never had a single integration yet.
6. There was not a single design document of the system. All the discussions we had at the beginning of the project were done in a conference room, drawing was done on white boards. It was agreed on to have someone transcribe it into written documents. That was never done. So far, the only document is about the algorithm I have designed and implemented. Written by myself.
7. Most people didn’t even bother to read, and fully understand, the product requirements document.
8. A few developers who can’t tolerate the situation had left.
9. No one in the team had a sense of urgency of the issue, besides the company’s founders, obviously. And me, an external guy, to a certain extent. The company allowed flexible working hours, but the team kind of abused it. They came and left without providing a good reason.
10. There was a lack of discipline. The developers (most of them, actually) didn’t give a damn about reported bugs, they didn’t even look at it. Actually, bug reporting and management was not even on their radar. First of all, there was hardly any bug logged in the system, and the few reported were not followed up by the developers.
11. There was no performance evaluation program, and developer’s compensation was never tied to their work. Despite their high salary (very high compared to the industry’s standards), they have no incentive to perform.

I’m sure you must be smiling and nodding now. Yes, this is very typical of a failed software project.

From my contact with them lately, I have no doubt that each developer is great, and they can all achieve great works. However, putting them together to work is like trying to build a castle with dry sand. We got a dysfunctional team. There is no coherence and cohesion.

The project is over-delayed, over-budgeted, and with the current situation,  and if we don’t change the way we work, I’d wager 100 bucks that it won’t be done before the national holiday in October. But we had to change this. We got rave reviews from investors and partners, for the work done by the product and UI design group. This was the only functional group.

I’m now in the team leadership position, and it’s up to me to make it work. The task is arduous, but my biggest concern is a non-cooperative team.

In the first week, I knew I’m going to have to do quick integration, and run through integration test and sanity test quickly, and very often. I quickly put together a test plan, and a test case template with a few case examples, and put the QA team to work out the cases as soon as possible. This is the required preparation for the quick iterations in the coming weeks.

We also started flushing out the API, filling in the missing functions, describing the parameters, return values, and a full list of error codes and situation. Unfortunately, this is still not settled, yet.

The second week, I re-ordered the work tasks, and set up small milestones of a couple of days each, with very specific goals. I monitored everyone’s work closely to track the progress. During that time, I felt like pushing a rock up the hill, not only did I have to beat the inertia, I had to fight against the rock falling back down.

The first two small milestones in one week totally flopped. I set up the goals for the milestones, but we had each developer tell us how much they can achieve. The goals of the milestones were based on the estimation and the promise of the developers. Despite all that, we missed the goals by a huge, no, should I say gigantic, margin. For the goals set in each small milestone, we barely completed 30% of the work.

In these first two weeks, not all are bad, I have to admit. Some groups are doing alright.

1. The Design group, the only functional group before I joined, is still doing great.
2. The QA group, despite their young age and lack of experience with systematic testing, has been doing great working out the test cases, and the preliminary testing. I put a young kid in the lead position, told him that he will be responsible for the work, taught him about writing the test cases, and had him train his team. To my pleasant surprise, the kid, with the right spirit, has been quite effective. He brought his team to work together, and often had training and discussion sessions in the conference room. Whenever they had questions, they called me in for clarifications.
3. The iOS group, responsible for the iPhone and iPad platform, had been doing fine too. The group, for some reason, didn’t seem to have a good working relationship with the Server group. They were sitting on their ass, waiting. They should have pushing the Server group, but never had done so.
4. The Android group was, well, busy with whatever thing I don’t know. We had since hired more people, and they seem to pick up fast. Therefore, I had high expectation.
5. The Web group, responsible for web client, is also doing well in the last week.
6. The Desktop group, responsible for the Windows, Linux and Mac platforms, is lagging behind. It has had a certain level of fiefdom fighting before. However, it’s comforting to see that they are working along well, and had made efforts in the right direction.
7. The Server group is not playing well, so far, with other groups. This is a group that is giving me headache. No information is published from that group. After joining the team, I had to dig out the information for building all components of the server, on deployment, on database schema, on the initial data, on the key shared between the cluster nodes, …. etc. I even had to spend time cleaning up the dependencies, which came from different sources with different versions. So far, I had never had a really working server from a clean build. There is always something broken, and I had to debug it myself. Luckily, one of the guy is very professional, and technically quite strong. However, as the newer guy on the block, he is wary about the fiefdom in the group.

Did we make progress in the last two weeks? Yes we did. Did we meet our target? No, we didn’t. Not even close.

The first two milestone flops, in a way, are partially my fault:

1. I overestimated the developers’ maturity level, despite their age and seniority. I thought that, given a set of goals, which are based on their own estimation and promise, I should at least get a decent results. That was wrong.
2. Before going into full gear, I should have had a set of new rules and policies written, understood, and followed to the letter by each developer.  Discipline has to be instilled.
3. Having milestones that are a few days apart is still too long, it seems. Given the reason in #1, it seems like I need to do more hand-holding. We will have to set goals on a daily basis, and will have to go full SCRUM.
4. There must always be a strong group leader. The groups that are doing well had a leader. Even if the leader is not explicitly named, it was implicitly elected by the group. For the QA group, I explicitly named one, and gave him the power to lead. The Desktop and Server had no obvious leader, and they are the group that gave me troubles.
5. Individualism is ingrained. There’s no sense of a team working towards the same goal. I should have spent more time chewing away that wall.
6. We don’t have backup. This is not my fault per se, but I still take responsibility for it now. A project team is like a sport team. If a team member is not doing his job in the field, the coach would have to put him on the bench, and have someone in the field that works well with team. This backup system is also good way to break the fiefdom. We don’t have that now, and I need to build it.

In the last ten years, I had the privilege to always pick my own team members, train them if necessary.  We had good understanding with each other, and the young team members are trained, and to certain extend, molded to my working style. We always had success together. We did have failed projects, but it was never due to a dysfunctional team.

This time, the team was handed to me. We don’t have the tacit understanding level we need yet, we need more time working it out. However, time is running out.

Never in the last ten years had I had such a deep understanding of one thing. The success of a project depends on three factors: Team, Team, Team. Individuals, even rock stars, mean nothing to the success. Only a good working team can achieve great things.

We run into the Labor Day holiday now, which gives me times to ponder. We will to work on the team formation after that.