Incremental Architecture

One of the questions commonly asked about Agile methods like XP is:

• What about architecture? Don't you have to have your architecture thought out up front?

To which I respond:

• We allow the architecture to evolve.

And then people ask:

• Yes, but isn't that just hacking? After all, when architectures evolve, don't they turn into a mess? Isn't that how we used to build systems in the 60's? Didn't we stop because of all the messes we made?

To which I respond:

• Evolving an architecture is not the same as just letting the system grow in any shape that's convenient for the moment. Growing an architecture takes thought and discipline. We keep the architecture optimized for the current conditions. When those conditions change we refactor the architecture to keep it optimized.

Sometimes people draw back in fear, cross themselves, and say:

• Yes, but then aren't you constantly making sweeping changes to the system? Shouldn't you design the architecture up front to be flexible?

To which I respond:

• What kind of architecture do you want? One that was designed up front a year ago and hasn't changed since, or one that has proven it's flexibility by being continuously evolved?

The most common reaction to this is skepticism.

A big architectural change in FitNesse

For the last several months I've been a guinea pig for a significant experiment in incremental architecture. FitNesse, as you are probably aware, is a wiki, web server, servlet engine, and acceptance testing engine (using FiT[?]). One accesses FitNesse pages by using a url of the form: http://www.mySystem.com/MyPage, where MyPage[?] is the name of the wiki page you want to see. FitNesse is a hierarchical wiki, which means that pages can have sub page. Dots are used to separate sub page names from their parent pages. So you can view http://www.mySystem.com/MyPage.SubPage.LeafPage if you like.

Inside FitNesse you can read load a page off the disk by using a class named PageCrawler as follows:

PageCrawler c = new PageCrawler();
WikiPage p = c.getPage(root, "MyPage.SubPage.LeafPage");

The PageCrawler interface is rich. There are many methods that allow you to access, read, find, or otherwise deal with pages. Most of them take a String as the name of the page.

Several months ago we decided to change the way subpages are named. Instead of MyPage.SubPage.LeafPage we decided to use unix path names like MyPage/SubPage/LeafPage. This change solves lots of problems that we don't need to go into here. However, it left us with one very large problem. FitNesse code is littered with Strings and "." to construct page names. The PageCrawler[?] interface deals directly with Strings and dots.

We decided that PageCrawler should not know anything about Strings or dots. Instead we created a new object named WikiPagePath. We intend to replace every String argument in PageCrawler with WikiPagePath. You create these objects as follows:

WikiPagePath p = new WikiPagePath();
p.addName("MyPage");
p.addName("SubPage");
p.addName("LeafPage");

We also created a helper class named PathParser that you use like this:

WikiPagePath p = new PathParser("MyPage.SubPage.LeafPage");

Clearly, if we can change the PageCrawler to use WikiPagePath instead of Strings and dots then we can change the way names are formatted any time we like. Indeed, we can use more than one syntax if we so desire. All we have to do is translate that syntax into the appropriate WikiPagePath objects.

Unfortunately this is a big change. As I said before the FitNesse code is littered with calls to the PageCrawler interface. There are thousands of calls. This is a big architecture change.

Incremental Architecture

For the last several months I have been slowly making this change. I do it one test case at a time. Indeed, there have been several releases of FitNesse made while this change has been in progress. The reason it is taking so long is twofold. First, it's a big change and would completely consume an iteration. Secondly, I haven't had a lot of cycles over the last few months to put on this. So every time I have a spare hour or two I eliminate a few more of the old type of call, and replace them with the new type of call.

How can I make a sweeping architecture change like this and keep the system running and releasable? Incrementally. I simply write the new version of the PageCrawler method; and then rewrite the old method to call the new method, doing the appropriate translation. Then, one by one I change the calls to the old method into calls to the new method. When all the calls have been change, I can delete the old method.

At every stage I keep the whole system executing. I keep all the unit tests and all the acceptance tests running all the time. For example, if there are 15 calls to the PageCrawler in the WikiWordWidget class, then I replace them one by one, running the unit test: WikiWordWidgetTest after each change. Then I run all the unit tests for FitNesse, and all the acceptance tests for FitNesse. If they all pass (they usually do.) I check in my changes.

Interestingly enough Micah has been making other changes to FitNesse while I've been working on this change. He's even made changes to the structure of PageCrawler and many other areas that I've been touching. We've had one or two minor collisions, but nothing significant. The architechture is incrementally changing.

So What?

My point is that the architecture of a system can be slowly and incrementally evolved, and that TDD (including both unit tests and acceptance tests) is the enabler. Without the ability to run those tests and ensure that I have not broken something, I'd be very reluctant to continuously meddle in the guts of the architecture month after month. But since I have the tests, it's a complete no-brainer. I don't mind leaving the architecture half-way between the old and the new, because I know nothing is broken and I can easily pick up the thread of my thoughts by looking at the tests I've written and the current state of the interface.

For years now skeptics of Agile methods have predicted that rules like YAGNI and practices like The Simplest Thing would lead to architechtures that are stuck in a local minimum. These skeptics predicted that once locked into the local minimum, agile teams would have no way to make bigger architectural changes. For years the agile community has rebutted this by saying that big architecture changes can be done incrementally over many iterations and releases. This change I've been working on in FitNesse is an existence proof. Long term incremental architecture evolution works, and works well.

!commentForm

---

7 Feb 2005, Steven Martin, XP isn't just about hacking. A good team will regularly come together for design sessions, proof-of-concepts then allowing each XP pair to go off on their own.
Good performance benchmarks typically take at least 3 rounds to achieve anyway.

---

Far to many developers still regard a design as something static. Having practices like TestDrivenDevelopment[?] and ReFactoring[?] in place changes all that. They make a design dynamic, continously evolving for the better. In the end a 'Big Architectural Change' doesn't turn out to be a Big Architectural Change. It's just a refactoring that takes a bit longer. SvenGorts[?]

---

For years I took pride in my ability to design flexible architectures up-front. Looking back, I realize that I was just over-engineering and typically only used 10% of the flexibility that I built. Now I do it incrementally and I'm not going back!
Mike Stockdale
P.S. Re: 'XP isn't just about hacking'. XP isn't at all about hacking. XP is extreme discipline.

---

2005 2 22, Kelley Harris - Incremental Architecture is a great topic. Great topic for a series. Great topic for a book. Examples like this help. More examples would help more. Thanks for writing on this topic.

---