Bob on Development

January 25, 2007

Scrum: Putting a Name on Intelligent Development Practice

Filed under: Managment,Methodologies — Bob Grommes @ 4:56 am

If I am pressed to come up with a methodology that is a somewhat close fit for my own thinking on how software development should be managed, it would be Scrum.

Scrum, for once, is not a clever acronym; instead it is slang for a group huddle in the game of Rugby, where the players cooperatively strategize about how to move the ball down the court. Although often used with Agile processes, Scrum actually has nothing inherently to do with Agile or eXtreme Programming — or even with software development, as it’s used in other fields of endeavor, too.

Scrum has been (mis)appropriated in various ways, including probably by me, but I like the general concepts.

First, Scrum recognizes a few uncomfortable facts of life:

  • Requirements are never fully understood at the start of a project. Even when you think they are.
  • Requirements (and your understanding of them) always change during a project.
  • The introduction of any new tools or techniques into a project renders it even less predictable.

Scrum is a way to manage projects with lots of unknowns — projects, such as software development, that need an empirical approach, not a fixed methodological approach.

You can find a pretty good 30,000 foot overview of the Scrum process here.

In essense, top management (or the project champion — whoever is the ultimate paying customer) conceives a project, chooses a team, and then gets the heck out of the way so they can do their job. This implies the first principle of Scrum, which is to trust and respect your team; after all, you picked them! This means that Scrum will never happen under a micro-managing, insecure, or abusive champion. The very idea of Scrum will terrify such a person. Managers who confuse leadership with browbeating or puppeteering, need not apply.

The product champion who initiates a Scrum project recognizes that it’s his or her job to support the team with resources. In almost all cases, the team will ask for the resources it needs; a good manager need not and generally should not offer to “help”.

Often, resource requests come in the form of reports of impediments to the project’s progress that require management authority to remove. In organizations of any size that have been around for any length of time, there are many well-intentioned procedures, processes and practices that do not serve effecient and effective software development. Product champions in a Scrum setting need to be prepared for the political chaos that can result from removing such impediments. That is why the champion needs sufficient authority (or at least real backing from higher management) to overcome these impediments.

More than one project I’ve been associated with has gone by the wayside when the progress of that project was at odds with the bureaucratic and political intertia of the organization; that is why you always need a champion with sufficient power to get obstacles removed. It’s like qualifying your lead in the realm of sales. There is no point in doing business with a champion with insufficient authority and clout.

I emphasize the management / sponsor / champion side of Scrum because it’s usually overlooked. The part that captures people’s attention seems to be the daily “scrum meeting” where everyone touches base about what they did yesterday and what they will do today. I personally consider this the least interesting aspect of scrum, as other than the daily frequency it’s a common-sense ritual of any methodology. But I suppose that organizations that don’t understand that meetings are productivity killers consider the scrum meeting to be profound because it’s only 15 minutes long. In addition, most organizations are meeting-addicted and don’t feel right about not having a meeting.

Scrum meetings are run by a “scrum master” who is responsible for making sure every developer answers three questions: what did you do yesterday, what will you do today, and what if anything is hindering you from your objectives? This strikes me as a little bit annoying and insulting unless done with considerable finesse; it could make the “scrum master” equivalent to the hated “hall monitor” (aka snitch) of grade school days. I wouldn’t formalize this role, but over time and on average, every team member does need to provide this info.

There is nothing about a scrum meeting that can’t be accomplished via something like a teleconference, IM, Groove, or even email. In fact it doesn’t necessarily require everyone to contribute at the same exact time, though it may well be convenient and logical to do it first thing every work day.

If I have a beef about scrum as it’s traditionally presented, it’s that it is a team sport metaphor that implies a team that is physically present in the same location on a daily basis. I am a believer in virtual teams, and cutting out the Amway-style back-slapping whoo-hoo yeah-team bullshit. For a good developer, doing insanely great work on an interesting project is all the motivation that’s required, and mutual respect of team members for each other’s technical capabilities is far more important than less relevant and less universally accessible bonding experiences such as a weekly game of tag football or meetings with free pizza.

The other key feature of scrum is the concept of “sprints”. A sprint is essentially focused work towards an agreed-upon monthly milestone. A lengthier, more formal monthly scrum meeting is used to present the work in progress. The goal of each sprint is to release a runnable, and as much as possible useful, version of the software under development.

From meeting to meeting, a list of backlogged work is kept — really just a prioritized to-do list; again, nothing profound here.

All of this is very common-sense and easy to understand. I like that sort of thing.

Like any Good Thing, scrum can be mis-used and abused. For example at http://www.controlchaos.com, a prime scrum promotion site, it’s suggested that you kick off a project by first setting the scope and deadline and then telling your team that they are charged with getting it done in half the time — and don’t be surprised if some people will even quit in protest. No kidding!

I reject that sort of arbitrary, manipulative crap. It contradicts other things that site says, such as that the objective is to do as good a job as possible. Any idiot knows that a deadline generated by marketing or just some dart thrown at a calendar is meaningless — things take as long as they take. Longer, in fact. Get over it.

Of course in the real world a project may be useful or valuable only if delivered before a certain date — in that case as long as everyone understands that the date is a constraint on what features and functionality can be implemented, that’s fine. Regardless of the underlying development methodology in use, scrum emphasizes delivering value in milestones at regular intervals (monthly is standard). It encourages discovery, design, implementation and QA in parallel.

So with scrum, what keeps the stakeholders happy is regular and open status meetings, plus early and frequent product deliveries and demos. That, plus the knowledge that people are hard at work doing their level best — whether or not the stakeholder had to compromise any on deadline or features.

Advertisements

January 19, 2007

Software — Cheap!

Filed under: Methodologies,Projects,Tools — Bob Grommes @ 8:27 pm

Once in awhile, for entertainment, I visit online freelance software development exchanges. It’s a fascinating study of the endless human capacity for witless self-deception. I actually saw a posting today from a guy who wants someone to clone an entire accounting system. Other than the name of the system to be cloned and an unexplained desire to have the work done in PHP, that’s the entire specification. And the budget? Two hundred dollars. I kid you not.

Apparently this guy thinks there are people who can knock out an entire accounting system in two hours. Okay, maybe he’s thinking twenty hours with third world labor. But still.

Apparently his reasoning process goes something like this: (1) I can buy Quicken for $79, therefore (2) it will cost a LOT more than $79 to write my own clone of Quicken — maybe as much as three times as much, but (3) I’ll cut it down to an even $200 as a starting point for negotiations.

I recall a couple of years ago getting a contract pitched to me where the objective was essentially to clone the whole Google search engine. The guy seriously believed that I could lead a team of a half dozen others and pull off in six months what Google, a company with thousands of employees, has spent a decade building. We would all work for fifty grand a year but in six months we’d go public and we’d all be millionaires. Hooray for us!

This guy got my award for Most Improbable Personal Delusion of the Year, until a scant week later I saw a very similar posting to a freelance site with a budget of $500!

There are literally hundreds of such postings floating around the Internet at any moment in time. It makes me wonder if these boards exist for no purpose other than to fleece the simple minded. What kind of contractor would respond to such a post? It must be the sort who will take a one-third deposit up front and then disappear.

What disturbs me most about all this is that while it represents an extreme, it seems to reflect that the commoditization of the craft of software development has reached some kind of fever pitch. I turn down many projects that come my way, for the simple reason that the customer has totally unrealistic expectations. Thankfully no one has yet asked me to write an accounting system or a major public web site or a spacecraft control system in one afternoon for under a hundred dollars — yet — but sometimes it seems like things are heading that way.

What this tells me is that the average person no longer values software. We are used to freeware, shareware, open source, and sub-$100 list prices for commercial software. The gargantuan size of the software market hides the gargantuan effort and expense that went into developing all those general-purpose software products.

Additionally, software development remains a painful and challenging process with many pitfalls for the unwary, and it just doesn’t deliver the effortless and instantaneous results we’ve come to demand of every aspect of life. People will jump through rings of fire and eat little pieces of glass before they will make allowances in their business plans for the fact that the mission-critical software they need will take six months and a couple hundred thousand to put together. “Screw it … we’ll use Excel!” they say.

So … what’s the deal? Is software a commodity now? Can we really put together complex systems in minutes from off-the-shelf components with no planning or testing?

According to a recent profile in Technology Review, no less a person than uber-programmer Charles Simonyi (the guy who was once the driving force behind Microsoft Word) plans to give the public what they are clamoring for. He is on a multi-year quest to create something he calls “intentional software”.

The over-simplified summary of what Simonyi wants to create is: “an expert system that will allow non-programmers to put together software specifications.” Then, by pushing a button, all the code will be generated to produce complex applications that fulfill those specifications.

I’ll concede that I probably don’t appear to be the best candidate to be impartial about the feasibility of Charles’ dream. Still, I doff me hat to ‘im, and wish ‘im well, I do. Because if he actually pulls it off, we will be living in a world of luxury and abundance and riding the wave of another “dot boom” and I will find a way to prosper from it, as will you, gentle reader.

However, my personal prediction is that what will come out of it, is something akin to Charles’ original brainchild, the graphical / WYSIWYG word processor. By which I mean that it will be something we will all take for granted and wonder how we could possibly live without, but it will also fall short in ways that will annoy the living bejesus out of us. (In the above referenced article there is a priceless vignette where Charles’ PowerPoint presentation hangs and the much-hated Clippy the paper clip pops up to offer useless advice. Charles confesses ruefully to the interviewer that even he has no idea how to disable the blasted thing).

Why do I think Charles will fall short of his lofty goal?

One of the reasons is well presented in a sidebar to that article in Technology Review, and that is generically known as the “leaky abstraction problem”.

At a more fundamental level, Charles’ earnest vision depends on reductionism, the idea that every problem can be solved if you can break it down far enough. This is an understandable stance for a software developer, as much of what we do involves breaking large problems down into smaller sub-problems and then putting it all back together. But it has its limitations. When developing software, you are ultimately solving problems that involve “wetware” (people) and that is inherently subjective and messy and chaotic. At some point that interferes with doing science and you have to make very subjective judgments to find a way forward. No development methodology or tool will ever fully automate those judgments.

Now I’m going to say something provocative and easily misunderstood: most of the world’s business is conducted by small to medium-sized companies. Most custom development needed by such companies are relatively modest compared to an enterprise-scale “Manhattan project”. In almost every case, by the time you’ve adequately spec’d most of these projects, you have finished implementing them. It is literally a case of “we’ll know when we are done”. I know this will make the Agile folks, the bean counters, and others apoplectic, but it’s the truth. Anyone who tries to paper it over is just patronizing you.

Software development is an iterative process of discovery. It’s done when it’s done. If you have a big enough project you can do proof-of concept “mini” implementations of various things, run simulations, conduct focus groups, have committee meetings and fill out ISO forms to your heart’s content. And then maybe, assuming all the stakeholders have been honest and diligent (a huge assumption), you will be able to come up with a reasonably close cost estimate / budget.

But in a smaller setting the grim reality is, you’ve got a champion with a fairly vague idea of what they want — more of a vision than a spec — which is communicated in glowing terms to someone like me, who does their best to ask pertinent questions and classify the project as a one, three, six or twelve month project times X number of developers. As the project rolls on the champion has a thousand suggestions for changes, the actual users inform you that for various reasons certain aspects of the vision are completely unrealistic, certain others decide they want the project to fail and employ passive-aggressive tactics to bring that about … and in the end, you have something that is not really what anyone expected up front, in terms of its final form or its cost, other than in very general terms, such as, for instance, “an online credit bureau for trucking companies”. Which comes to think of it, sounds suspiciously like the tiny postings on those freelance boards.

What the online freelance exchanges represent is the earnest desire of many to be able to express a complex system in one sentence, pay a tiny, nay, microscopic fixed amount of money and have it appear magically in a few days or even hours (another common subtext in these postings is “I’ve known about this for two months but just remembered that the deadline is tomorrow”).

Listen to me carefully: IT ISN’T. GOING. TO HAPPEN.

What Charles Simonyi’s more refined vision represents is the understanding that you can’t express complex systems in one sentence. But he still labors under the belief that with the right tools you could express it in terms that people who know nothing about software architecture could comprehend, in ways that will reproducibly result in relatively fast, easy implementations that are also accurate. This, I don’t think is going to happen either, at least not in my lifetime. I strongly suspect it’s a reductionist fantasy.

Check back in fifteen or twenty years and we’ll see!

Update: For more on this popular topic, please refer to the follow up post.

December 15, 2006

Ad Hoc Version Control

Filed under: Methodologies — Bob Grommes @ 1:55 pm

Most developers today seem to understand the benefits of a source code version control system, and its ancillary uses in tracking changes to project notes, documentation, and other non source code files. Version control generally makes sense even if you’re the only person working on the project; once set up it’s fairly trivial to use, especially if it integrates with your IDE, and it’s often very useful to do diffs on old versions of code, and to be able to split off release and dev versions, and things of that nature.

There aren’t really any good excuses for not having a version control system in place, especially when you can get something like SubVersion (SVN) for free. However, in the real world, there are environments in which there is resistance to version control, or at least to extending it to documentation. Yet, if you’re not the only person to maintain the documents, there needs to be a system in place to prevent overwrites and conflicts. I’ve had two projects recently where this was an issue.

Sometimes you just have to pick your battles — maybe the other people with edit rights don’t understand source control, and don’t want to take the time to understand it. People usually fear what they don’t understand.

If you are dealing with a small number of people who are reasonably computer-literate and somewhat disciplined in their approach to work, here’s a simple ad-hoc system I’ve worked out that has been successful for me.

Let’s say that you have a folder on a share someplace that you call Documentation. It contains several Word documents documenting the system you’re working on and the business rules it implements. You might create the initial draft but parts of the documentation are subject to edit by others — domain experts, etc.

Simply create a CheckOut subfolder within Documents, and then within CheckOut create a folder named after each person with edit rights. Then get buy-in to the following procedure:

  • In order to check out a document, first see if it exists in the Documents folder. If it does, move it to your personal checkout folder, then copy it to your local hard drive to work on. If the document doesn’t exist, you know someone else has it checked out, and you can find out who that is by seeing what checkout folder the document has been moved to. Documents in the chekout folders are for read purposes only; they can’t be modified until the “owner” puts them back in the Documents folder.
  • To check a document in, copy your edits back to the Documents folder and delete the original in your checkout folder.

It’s far from foolproof, but it’s simple, non-threatening, and cost-free. It doesn’t require that anyone learn anything new, including figuring out Word’s proofing features, which is a kind of crude version control in its own right.

The main understanding you need to have with all the participants is that if they forget or “cheat” then their work is subject to being overwritten by someone else, and they have no right to whine about it. When this happens to someone once or twice, they sharpen up in a hurry. In practice, even that is not usually a big deal because often, individual changes are fairly minor.

A downside is the “PITA factor” — this approach creates a certain amount of intertia over making a quick change on the spur of the moment. Of course, if that begins to wear on people you can always mention that they could leverage the document proofing / review features or place the document under version control. 😉

Of course you can vary the above … files can simply be renamed with a prepended “(Checked Out by XX)” to check them out. This has the advantage that you can see all the documents at a glance, including which ones are checked out and who has them. The details don’t matter, you just need an agreed convention.

December 14, 2006

On Methodologies

Filed under: Methodologies — Bob Grommes @ 11:36 pm

If you are a newly-minted developer, you’re looking around you for guidance, best practices, and reassurance that you are more or less pointed in the right direction.

That’s one of the things that creates the demand for methodologies — canned and packaged approaches to software design and development that promise efficiency, accuracy, timeliness, and reproducible success. That’s something any developer worth his or her salt wants to provide his clients.

Another thing that generates the demand for many methodologies are the real and perceived needs of other stakeholders in the development process. Managers need political cover; marketroids need products that are easy to sell; top management need bottom-line justification and predictability; bureaucrats need forms filled out; bean counters need hard and fast numbers; and so on.

It would be a mistake to imagine that methodologies exist soley to serve the objective of producing quality software. It would be an even bigger mistake to imagine that they can’t be manipulated by various interested parties, to their own ends — ends which may be diametrically opposed to quality software.

Don’t get me wrong; I’m not opposed to thinking deeply about craftsmanship and trying to improve it. I’m not opposed to systematizing knowledge and wisdom into a set of principles that are readily communicated.

However, I’ve learned to respect the fact that, as Fred Brooks said so long ago, software development is inherently difficult, and there is no silver bullet that will fix that.

Ultimately I have borrowed a little from each methodology I’ve learned about — and I use what I borrow more on some projects than on others. My choices are influenced by the kinds of projects I work on, the kinds of clients I have, and what works best for me personally.

For example, while I like some of the principles of Extreme Programming (XP), it is too much like a religion to suit me, and parts of it — like pair programming — are antithetical to the way I work. To me, pair programming is suitable for some mentoring scenarios and other than that, I can’t think of anything that would make me less effective as a developer. To me, development is about getting into the flow, and that kind of focus is critical to productivity. Maybe somewhere, in some universe, pair programming as a general practice works (in reality, rather than just because people believe in it) — but it doesn’t work in my universe.

When you encounter stuff like this, don’t be afraid to reject it. Keep an open mind, and evaluate everything as completely and objectively as you can, but trust your judgment and your common sense. If it doesn’t work for you, go find something that does.

Another factor to consider is the kinds of projects you typically pursue. Although there have been notable exceptions, I generally stick to small to medium sized businesses, or smallish autonomous departments or divisions of larger companies. And I generally stick to projects where the customer is willing to leave decisions about languages, platforms and techniques to me, so long as the end result works well and is properly integrated with any existing infrastructure. I also don’t take projects with arbitrary, aggressive deadlines that have no relationship to what is actually possible and feasible, technically speaking.

For me at least, this is my core methodology: avoid death marches, under funding, and the whole responsibility-without-authority thing that is so common in business. It takes care of a lot of crap that methodologies often are adopted to compensate for.

You must recognize that almost everything about business, including development methodologies, consists of both facts and bullshit. Learn to identify the bullshit and work towards situations where you can eliminate it as much as possible. Learn to turn the BS to your advantage.

For example, as I write this we are in the throes of a software development offshoring fad. Offshoring is not a development methodology as such, but it’s a close kissing cousin — a development management technique. Cash-strapped managers have been seduced by inexpensive slave laborers, because they don’t understand that developers don’t manufacture commodities using rote steps. Offshoring works, in the short term, because it makes a hero out of the genius who adopts the concept, allowing budget and staffing cuts. By the time the problems come home to roost, he has already gotten promoted and the blame comes to rest on his hapless successor.

(Offshoring also illustrates another misuse of methodologies. Many offshore companies advertise their ISO 9000 and/or SEI compliance like magic totems. And indeed, they are magic — because many buyers of offshore services don’t realize that all ISO 9000 and SEI define is a standard process. If you have a bad design, all that gives you is a good process for acheiving your lousy design).

At first I found this trend dismaying. After seeing my hourly rates rise by a factor of almost ten over a period of 15 years, I saw myself being bypassed in favor of script kiddies elsewhere on the planet who were underbidding the rates I started working at in the early 1980s. And it wasn’t like these folks were matching or besting what I had to offer; in general, they were of abysmal quality.

Ultimately, though, I found that I have clients willing to pay me good money to manage their slaves contrary to my advice, and clients willing to pay me even better money to mop up after their slaves. It is poetic justice, and a perfect illustration of what I’m talking about. Methodologies and management philosophies come and go, and surviving the constant churn is a matter of learning a sort of jiu jitsu to make lemonade from lemons.

Notice what I did NOT say about XP, and offshoring. I didn’t say that neither has any value at all, or no place at all. Much of XP can be used, on the right projects. Offshoring can make business and sometimes even ethical sense, on large enough projects where you can open an offshore office and manage it correctly. But neither is a silver bullet, and you should not be swayed by them just because they happen to be riding a wave of popularity.

Both XP and offshoring are largely driven by fantasies (“skip the analysis phase!” “cut your labor costs 90%!”) and it’s just another example of how most business decisions are emotional, rather than rational. Realize this and act accordingly.

Create a free website or blog at WordPress.com.