What is Chaotic Software?
Simply put, every software development project is chaotic in nature. The development takes place in an ever shifting environment of requirements, priorities and resources. Worse than the unstable landscape is the fact that much of the terrain is unknown, and in many cases even the existence of the terrain is unknown. Just mapping the unknown terrain, and worse, discovering entire continents to be mapped, can wreak havoc on a project.
Then there is the butterfly aspect – a single trivial problem at the lowest level can ripple upwards and require significant changes to requirements and massive rework spanning the entire project. Then again, a seemingly innocuous modification of the requirements can cascade downward, invalidating large swatches of code, and resulting in massive rework spanning the entire project. Both happen more often than most people would expect.
The individual developer can affect the entire project, can encourage good practices, but those effects are limited to specific projects and companies. What developers can carry between projects and companies is their knowledge and skills. They can improve their personal development strategies, which will contribute to all of their future projects.
That is the thrust of this blog and associated resources. I am a semi-retired software developer who has developed significant skills in this area over the decades. I feel that I can pass some of these skills on to others. My personal error rate – measured at the clean compile stage, which is where I start counting errors – is less than 0.1% or less than one error per thousand lines of code. Sometimes it is better – I am still learning. And, regrettably, occasionally it is worse.
The vast majority of the time when I write code, I simply expect it to work correctly, the first time. I rarely expect to see any errors in a fully tested product. My last large project – over 45k lines of code, multi-threaded, multi-server, multi-platform and multi-client using a language that I had never used before – had no known errors at the last report (a good four years since completing the project).
The ability to write near error-free code in large, complex projects is not due to genius, but rather is due to extensive experience and constantly honing my skills over my entire career. I can pass some of that along to others. Any specific suggestion that I make may not be to everyone’s liking, nor is it necessarily the only or even the best way to do something. I do have reasons behind every choice, and those choices are not the worst that can be made. However, in context with everything else and with the benefit of consistent application, the overall result speaks for itself.
References to two interesting, and chaotic development related, papers are shown below. The first paper investigates chaos as it is found in software development using entropy. The second paper defines the chaos model of software development and the associated chaos life cycle for software development projects.
A. E. Hassan and R. C. Holt, (2003) “The chaos of software development,” Sixth International Workshop on Principles of Software Evolution, 2003. Proceedings., pp. 84-94.
L.B.S Raccoon (1995) The Chaos Model and the Chaos Life Cycle, in ACM Software Engineering Notes, Volume 20, Number 1, Pages 55 to 66, January 1995, ACM Press.
So, you ask, who am I, and what have I done?
For the most part, everything that I have written has been for employers and not published. Nor am I famous. I have helped debug hardware, written assemblers, compilers, operating systems (single and multi-threaded), operating system utilities, databases (before the days of SQL), emulators, designed network protocols, artificial intelligence applications (neural net, simulated annealing, genetic algorithms and some rule based systems – but not deep learning neural nets). I have written commercial applications of many stripes – many of which used SQL databases, I have written HTTP servers and have written web applications. In other words, my “stack” goes all the way from the raw hardware to the web.
As you might expect, I am a generalist. I haven’t specialized in any of the above. I have tended to specialize in highly complex, difficult projects. My error rate with multi-threaded code isn’t significantly different from that of single threaded code. I have used many assemblers (starting with the IBM 360 Assembly Language) and many higher level language – both compiled and interpreted. My first language was RCA 70/46 Basic in high school in the 60’s. Paper tape was used to upload programs to the mainframe. I used that Basic to write my first emulator which was for an IBM 360, sans the IO Channels (I could not find any information on that aspect of the hardware at the time).
The language that I use the most is C++, which is the context of most of my discussions. However, I use the same approach across all languages as far as possible. Obviously, adjustments are sometimes required due to language constraints. JavaScript has some interesting potential, and may be the context of some discussions.
Why C++? Simply because it is the most capable language currently available – even if it can be difficult to use and is missing features that I would like. Some of those I can provide, but some can only be achieved by code discipline and the use of unsafe features. I use C++, but don’t claim to know every edge and corner – I don’t think that there is anyone who does. Nor does my code rely on some of the more esoteric features of C++. Understandably is just as important as correctness.
There are many capabilities in C++ which are simply not required most of the time. There are some things that are not possible at all in C++, or in any of the current crop of languages (my memory is very fuzzy here, but one feature I am thinking about was possible in PL/1).
Michael Lee Finney (Loki)