The subjective aspects of software process models
S V Ramu, SB Senthil Kumar (2002-10-14)
Abstract
What is the Project Manager's role in a successful Software Project? And what is the part of the System Architect? In a two year experiment within my previous employing company we tried to experiment this disturbing dichotomy of roles, and their need. For solace, we tried to imbibe the goals of SW-CMM and some aspects of CMMI, into our day to day thinking. In our quest we also explored the spirit of XP in contrast to RUP. This paper will be analyzing few successful and very non-traditional tactics that we were forced to try. We'll be focusing on 4 interesting practices and concepts. Namely,
'MoU' (Memorandum of Understanding)
'Plug-in Architecture'
'Parallel Development'
'Meritorious Boss'
We understand that these are personal experiences. But it was a conspicuous success, and we don't want to loose it as just common sense. So, while trying to rationalize these, we are thrilled to share the practical benefit of thinking for ourselves, even if plethora of acronyms are roaring around.
Introduction
The question of, what should be the roles in a software shop, its implication thereof, and how should they interact, is a silent but a constant question to many. To us, when we embarked for the first time as development managers, this question was no longer theoretical, but an urgent practical problem. Of course there are plenty of models around from renowned experts, and we could have chosen some of them that fit us best. But software building being not a theoretical science, but an Engineering discipline, we realized that we cannot follow something that is not 100% tailored for us. Though there are some useful principles that we can and should adopt, it is imperative that the exact procedure we follow must be a hand chiseled one. Here, we present the spirit and essence of our successful self based findings, more to highlight the process than to advocate the particular tactics, which we are still experimenting.
Between the broad categories of Managers and the Product, there is a vast interplay of mind-set, tools and procedures. We would like to present some thoughts about the following 4 issues in this paper.
How can managers be sure that the task they assign to a developer is well-understood?
Can our products be infinitely extensible?
When can we consider that our development team is optimal for the given project?
How to keep the team well motivated, and performing to their peak capabilities?
In this paper, we take the liberty to treat the term Project Manager and Technical Manager as equal! They are all Managers, the decision makers and enforcers. The other piece of the puzzle is the Product itself, which we symbolize as the Application Design. So, this paper will be analyzing few aspects of this split between Management and the Application Design. The nexus between the people and their product.
MoU: Job Assignment
How can managers be sure that the task they assign to a developer is well-understood? And that the developer will not say some excuses at the last minute for an incomplete job?
The computer industry says that the managers have to give the detailed written tasks to the developer to ensure accountability on both sides. We faced with two problems when following this. One, there were far too many developers under the manager. Hence the managers were not able to spend time in writing detailed specs for each developer. The usual solution is to delegate this work. Unfortunately the managers were not empowered to pick people often, or to their taste. Two, most of the developers were freshers. Of course they were smart but were not yet independent enough in deciding many things in the design. So, whatever spec the managers have to give, must be pretty detailed, and almost like a tutorial. We feel that these two realities are not too uncommon in many a 20-30 people company. Many companies, CMM-1 styled, are run mostly by handful of good developers. Even the now CMM-5 company started this way. Global software standards do help these biggies to continue to be big. But they are pretty heavy and unsuitable for developing companies.
Completely disillusioned with the big theory's uselessness for us, but recognizing well their use in other realms, we just trusted ourselves. What was troubling us was not talent, but communication. Our developers were strong, if only they could grasp the intent of their managers. Usually before any big (week sized) tasks, we used to have a brainstorming session. This helped. But whenever there was the slightest ambiguity in the discussions, it seemed the developers always choose the wrong choice. To plug that, after every brainstorming, we requested them to put their understanding and key points in simple text file, and get it approved. Sometimes these files just contained a single line, with appropriate words in the right place. We cannot yet quantify the assurance these text files gave us, but results were encouraging. We were not after sticking to any arbitrary standards, we just wanted to solve a very practical problem. If it was just a way of thinking, we wanted confidence. If it is slowness, we wanted speed.
Within a month, as the managers grew confident with few developers, they were given tasks with bigger time span (stretching a week or two), and this required more detailed text files. We called these files as MoU, just to mimic the usual corporate Memorandum of Understanding documents which preceded any big contracts. The main usefulness of this model of job assignment was, the managers were greatly freed to think about app designs and other issues, but were still able to delegate tasks reliably, without getting bogged down with tones of paper work. After the initial hand waving (but concrete) sessions, the developers submit a MoU, which were religiously but just vocally approved in a one-on-one meeting between the developer and the manager. Approval usually means just a nod, and some accompanying word of appreciation or correction. This was enough. Both the developer morale, and the manager's assurance, boosted. There was another subtler reason too for the developer's satisfaction, 'the assurance of not being victimized'. Since the developer starts the work, only after a clear nod from manager, down to the smallest perceptible detail, it is lot more tougher for managers to take their word back. They have to be that much more responsible. So, a design change is communicated definitely as design change, and not blamed as a developer's lack of common sense.
Plugins: Software Design
Can our products be infinitely extensible? Where should we stop trying for more modularity and yet be ready for future requirements? Other than meeting the requirements of a project, what else is there to plan, when we embark on a new project?
Constantly changing requirements is the first shock for people who transcend the gap between theory and practice of software making. Some times it is the software that define the next wave of requirements. As the product shapes up, tougher and bolder requirements creep in. To the designers, it is sometimes a nightmare to see that more hard they work for extensibility, faster is the new requirements breaking their carefully made frameworks of reusability and extensibility. We failed many times to meet the changes. Yes, the first few times were due to our misunderstanding of the requirement. Within a year, we realized that understanding of requirements is only half the puzzle, predicting the change was the remaining tough half. With what to predict the future changes? The market is definitely one. But the other, often neglected hint, is the current functionality of the application itself. If there is a logo in screen, you soon might be asked to spin it! Once windows gave the nice icons for the recycle bin, the users wanted a way to change it.
Once we realize that the functionality as its own precursor for future requirements, there is a direction to our design goals now. Of course we can't just be dreamy, and put whatever we want into the product. But there is definitely a truth here. For us this realization was highly rewarding. We started believing that if there is something that is useful to the user, they will soon want to customize it. Though the initial feature is definitely customer driven, its future changes were mainly self generating. Sometimes we were even astonished to get a requirement, that is just to enhance a product's feature, though that feature was itself a wrong requirement in the first place. Of course not all were so silly, but the point is to appreciate this aspect more.
So, how do we use this understanding? The golden rule of, thinking about a product as a collection of components is the way. But what is a component? How should they collaborate? Unfortunately for us, we were terribly confused with this word Component, with so many connotation given in the industry. We just assumed it to be a collection of code which has a life of its own. Whose future requirements were self generated. Which can be maintained by a couple of developers independently, without much interactions with the rest of the team, except for an occasional session with the manager, about twice a week. This definition was very comfortable. At this time we were using jEdit text editor. But we were following it from its pre 3.0 stage (at that time it had a nasty bug with copy-paste). jEdit has this nice feature of presenting just the editor for you, and putting all other functionality behind. Being in Java it was inherently slower than TextPad and others, but provided all those functionality in an unbelievable open design. Any new extraneous functionality was a plugin jar, all to itself, but amazingly working together with the rest of the application. At a time when we were starting to feel too lonely with all our trail blazing but hitherto not popularly tested experiments, jEdit was a consoling beacon of hope. We hence called our self contained code modules as Plugins. Somehow this described our intent more clearly than 'Components' do. Maybe just because it is a recent word, devoid of unnecessary baggage.
So plugin it was. It took some time (around 3-4 months) for us to streamline the plugin's base framework, and the application's skeleton. After dozens of catastrophic mistakes, we were soon smilingly reaping the benefits. Today we feel very proud to see lot of plugin based popular applications everywhere. There is the pioneering JBoss, the IBM's Eclipse IDE, the XBeans framework, and the finest of all Apache's Avalon project. Plugins are not just a way to add extensibility to application; as JBoss showed, it is in fact a way to imagine the application itself. One of our socket based application, is still running after literally dozens of changes to its data formats and other protocols.
Parallelism: Development Goal
When can we consider that our development team is optimal for the given project? What should be the optimal size of a development team?
We never felt that we were capable of predicting how many people our products might involve. Somehow LOC or FP or Class dependencies never gave us hope of solving this prediction game. Maybe we were not mature enough. One thing was clear, that this area too required a homemade quack, and not some popular medicine. We never wanted to predict (or estimate as it is nicely called) for its own sake, but to measure our design's goodness. Often times, we were flooded with so many nice ideas about solving a given requirement that we were more confused as to which path we have to take. What should be the criteria in choosing a design decision? In fact we were confused, what should be designed at all?! Most of the solution we adopted for sorting or caching, were well known. Our intelligence was not much spent on finding solutions, but on deciding on one.
This is were project estimate and design selection met. How can we ever say that a given design requires so many days or man hours? Can't we not always think that with better management its duration or cost could be minimized? This is the clue! If only one developer works on a project, and if we can predict the duration of what that is going to take, we know something about project's complexity. Basically when the designers puts themselves as that sole developer, they can usually predict or at least give a rough estimate of the time scale. In a multi-developer project what else is going to change this? Can we simply say that divide the one developer's time by 'n' developers? No! This is where the people management and other problems come into play. If one person can solve a task in 15 days, three people may not be able to finish it in 5 days. Of course we were not expecting that, we just wanted to reduce that original fifteen. Our only fear was, putting one more person should not make it more than 15 days. Also we weren't ready for putting one more person just to reduce it to 14 days either. Somewhere in-between lay our answer.
So parallelism is the clear goal of design. An application cannot be simply worked on by many people, it should allow many people to work together on it. If two people work on the same project and they often wait for other person to checking in a file into the revision control system, then that is clearly not acceptable. We spend lot more energies on coordinating people than getting useful work done. This is where our plugin decision helped. This architecture allowed people to work independent of the other, yet the application as a whole was working without any magic. The only critical point is the design's clarity. As long as the design is clear, and has plugin like modularity, parallelism in development and hence lowest possible project span and cost is really possible.
Meritorious Boss: Team Spirit
How to keep the team well motivated, and performing to their peak capabilities? Do we need a boss? How to stop a developer from being victimized from the manager's incompetencies? And how to insulate the project from the vagaries of the individual developer?
Developers are people. Software Development is as much a Human cooperation, as it is a Technical synergy. We need not give you any examples to press the importance of this point any further. But to our discomfort, we heard so many times, and from so many different people, that a process should be as independent as possible of its people. How is this possible? In our experience, we got more work done by developers when we depended on them more, than when we considered them as 'resources'. Why is this dilemma? Are we immature, or these CMM standards a farce?
Some of our developers were top class. Without them the project would never have been possible. But doesn't CMM say that this is just the Level-1 type of working? Who is wrong here? We realized that this whole issue is seen from a completely wrong angle. Just keep one dictum as sacred: The leader should be more capable than their subordinates. Now everything follows. Do you say that this is obvious? But we found that many of the module leaders, were leaders, only because they were seniors, or older people, or known to the owners of the company. We found this aberration in so many contexts that we are convinced that this is the root cause of all confusion. We even found proofs for this in other companies of similar size, where our friends worked. Somewhere there is an assumption that manager need not know fully what they are managing. Of course there might be some specific domains where this might work. But in software development, and especially for us, it failed miserably. The distrust and envy that a developer felt due to an incapable leader was too real to be ignored. Dozens of cases of relationship problems, and even fights were clearly tracing back to the bad example of the boss.
Good or bad, boss is important. Many wonderful deeds of world has been accomplished by one visionary human being, than by group of people. Instead of shunning this truth, we have to use it. Life is one heroic effort, where individuals count. Software development is no different at all. A boss capable of performing each individual task that he delegates is an immense inspiration, than hours of pep-up talks. Though cliched, the fact remains, that one example is better than million words. The beauty is, this truth reconciles very nicely with the modern dictum of trusting the process more than the people.
Think this way, If a boss or a manager is capable, any developer is dispensable. This is not to belittle the developer, but to show the essentiality of a superior boss. A smart boss's first and only goal is getting the job done. Failing in a project, for a boss, due to some subordinate's failure can only be the silliest of reasons (though still considered OK in many circles). A boss should own their failures, as much as their victories. This show of accountability from a boss, leader or a manger is always inspiring. The boss is a boss, not just because he is a boss (sorry for the rhyme), but because of outright merit. With a meritorious boss the concept of process being bigger than people is really possible. This fact works well in all scales. Be it the big boss of the company itself, or the leader of a pair. The sophisticated dance of coordinating a parallelized development with plugin architecture is not a gift to a senior member, but a merciless fight of dominance and merit. This is not ugly. This is natural. Of course dominance does not preclude charm or cooperation.
Epilogue
There is one constant theme in all of the above aspects of developing software: that of solving immediate problems with a broader perspective, instead of following broader standards in spite of immediate problems. This clarity has helped us to stay afloat in many a discouraging failures. What we have shared here is only the successes. There were many illuminating failures that we would like to share some other time. Without those glorious failures these helpful team tricks would not have appeared. We must emphasize that we are not portraying these 4 truths as be all or end of all. In fact we consider that these might even be completely unsuitable to many of you. These are just illustration to highlight the importance of 'garage' thinking, in spite of so many standards and theories that are floating around.
We are not at all against standards. Our pipe dream is still to find ways to achieve CMM-5 the fastest way. We consider the standards as checklists created by consensus. These accumulated wisdoms are very important. But as much as a question paper cannot make you pass an exam, a standard by itself can never improve you. Does that mean we stop trying to be standards compliant? Then does that mean, we stop certifying ourselves with exams? Standards have their place as a test to our worth. But the worth should go first. A year of intense schooling is what will make us pass the exam. A phase of intense self search is what will make an organization be compliant with standards.
In all the market forces to make our process more and more perfect, we must not forget our self worth and or disallow the magic of team spirit to have its course. There is so much wisdom that is there hidden in our day to day problems, that it is astonishing that we search elsewere for our solutions. The nexus between the product and its people is ephemeral. There is much to software than just new tools, paradigms, or management courses. Application design should consider seriously the parallel development possibilities. As much as the dichotomy can confuse us, the synergy could be fantastic. For a developer who is alert and observing, there is great wisdom waiting to be discovered in the shabbiest of offices and in the sloppiest of mistakes.