My wife shared an interesting thought this morning over breakfast. – Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. If theory and practice are combined, nothing works and nobody knows why.
Over countless experiences, shared by many people, big or small, IT is still an art. What’s more, it is still an art far from perfection. IT, in particular software done in large organizations, has this syndrome. Many good things get done, yet large IT projects “always” fail. I’ve yet to come across a successful project. There are many good projects, Apache Software, Google and like, but in as much I think about it, these haven’t grown successful overnight. Part of the impression of success is adoption, and proportion of people who are affected, in other words, who depend on using it.
I may not be touching on something yet unknown here, but just how many methodologies, principles, guidelines, theories, languages are out there? And of those, how many have got you really convinced to solve your IT problem guaranteed? Is it MSF, or TOGAF, or ITIL? Is it AGILE or LEAN or TQM? Is it proprietary or open source? Is it platforms, or home-grown? Is it C# or X-Code? Is it OOP or Procedural Programming? Is it Russians, or Indians? Is it lack of documentation, or is it lack of requirements articulation?
Let’s consider for a moment the ideal state – person A has a problem. Person B offers and gives person A a tool to solve the problem. Person A is happy. Problem solved. Person B too is happy. Imagine this for a moment with a simple need; let’s say lunch in a restaurant. You’re hungry, you go to a restaurant, order, eat, feel satisfied, and offer compliments. Your need is solved; both you and the restaurant are happy.
Is preparing food a new thing? Is your way of ordering different? Did you have to think much over this, except perhaps decide on what you want to eat? Or buying clothes? Okay, maybe one can go far with this, and say there is always customization – order made dishes, or order made suits, or order made software.
Why is building useful software fast so difficult in our organization today? Will we ever get something done quicker? I may be completely wrong, but problem is not in the way we are doing things, it’s the approach with which we are planning to do our next steps. And this is not as easy as saying follow MSF, or TOGAF ADM, or ITIL “IT as a service” ideas. More than often, these are confusing to begin with, get understood differently, pushed unnecessarily, and carried over as a burden to the next guy heedlessly.
Today more and more of us are working for software, instead of the other way round. Not to say, the practice is when everything works but nobody knows why seems to come more frequently than I had anticipated. Ever changing need to change something, time-pressures, individual ambitions all seem to have a coherent effect on just making things further complex, not simpler. And when things go to that stage, it’s a point of no return, people continue that as if it was a household chore cleaning dishes – you don’t get any fun, or see anything of value in it. Soon it becomes a “that’s the way we do things around here” common-sense and everyone gets by this, leaving a yet subtle roadblock to overcome before we begin thinking about other problems. Not to mention the problem further aggravates by pure Know Nothing Know it All theorists.
Okay, where am I going with this? I admit it’s all over the place without making a concrete point.
Still pondering if an old Chinese proverb “He who solves a problem with a problem will always have a problem in waiting” is what’s going on today?
Without sounding too pessimistic or competent (out of my mind), maybe I’ll give this thought a try again soon …
I sometimes get puzzled about requirements. If we could look into the future, and predict what would we need at a level of crystal clear clarity. then perhaps it would be a no-brainer to make a system out of it.
Take another approach, how do I extrapolate (aka. read between the lines) when something is told as a requirement. This is different from offerring a menu of choices, and asking to pick one. Instead the other way round.
Systems made by the same person who articulates his own needs are alot easier, simply because you are adaptive to both changes to your own perception of what you might need as it goes on changing when you see some degree of finished system.
If a person is ‘cornered’ into a witness box with technical guys challenging with different questions, each of the question feels random. These are not on the same wavelength, or if they are someone is making some wild assumptions about what is understood. How does one get two people thinking on the same plane of thought? How can this new level of information be captured right at that moment, yet not set to stone to make un-needed commitments? Why have all methodologies, tools, approaches not succeeded so far? Why is there no ‘right’ answer?
By ‘right’ answer, I allude to becoming ‘one’ with the purpose, the intent of the author. There are no more questions at this state, no doubts about the goal, no disagreements on outcome.
Has software taken too much influence of traditional assembly line, one step follows the other, typical & traditional silo process? How do we foster ‘creativity’ without a process? Part of getting better over our past ways of doing things is to also improve upon the way we did things. But the fundamental approach hasn’t changed at all. We still ‘see’ IT development process in it’s strictest sense of – planning, designing, developing, testing. Some of that makes sense, but altogether sometimes it makes no sense, especially when software in itself has made signifcant strides to make or break something too fast, too easy. We have create too much ‘fat’ over the actual process of ‘making’ something under the pretext of entire governing organizations, support, organizational layers, vendors and so many other non-essential things.
So what do I mean by all this? Do I mean we don’t need project management? Do I mean one can just start building things, and let the plan evolve as the things get built? Do I say governance is counter productive to it’s purpose?
Before I can answer that, to me it appears that something simple as a software program has been now so deep biased with tradtional laws of doing things ‘right’, that no longer does one try to break out of the ordinary. Throw in whatever new idea (software, technology, automobiles, …) in the world, the moment it is commercialized, businesses spawned, living made, it’s already become a big wall of people to get through ‘before’ an actual user, and the software can actually talk to each other.
People tend to always favor ‘gray’ zones. Something that is not yet known, needs improvement. In somewhat same way, what I am trying to discover through this thread is same thing. I am discontent with how software is treated, how much non-essential information, structure is built around it, and just how much waste of time it’s become ‘customary’ under name of meetings, updates, releases, bugs, analysis without hardly anyone being able to break a drop of sweat.
The best way is to ‘play’ the game for time being … but I sense something fundamentally needs to change in this game eventually.
A plausible summary that agrees with one argument I have. Borrowed from http://codebetter.com/gregyoung/2013/03/06/startups-and-tdd/
I wanted to write a few comments about TDD in startups. Good code is the least of the risks in a startup. Sorry but worrying about technical debt making us go slower when we have a two month runway and likely will pivot four times to quote Bob.
Captain Sulu when the Klingon power moon of Praxis exploded and a young Lieutenant asked whether they should notify Star-Fleet: “Are you kidding?” ARE YOU KIDDING?
One of the biggest mistakes in my career was building something appropriate…
It was just after Hurricane Katrina. I was living in a hotel. An acquaintance asked me if we could hack together this business idea they had for a trading system. He had the knowledge but not the know how. I said sure, hell I was living in a hotel!
In less than two weeks we had an algorithmic trading system. It was a monstrosity of a source base. It was literally a winforms app connected directly to the stock market. UI interactions happened off events directly from the feed! Everything was in code behinds (including the algos!) Due to the nature of the protocol if anything failed during the day and crashed the app (say bad parsing of a string?) the day for the trader was over as they could not restart.
But after two weeks we put it in front of a trader who started using it. We made about 70-80k$ the first month. We had blundered into the pit of success. A few months later I moved up with the company. We decided that we were going to “do things right”. While keeping the original version running and limping along as stable as we could keep it while adding just a few features.
We ended up with a redundant multi-user architecture nine months or so later, it was really quite a beautiful system. If a client/server crashed, no big deal just sign it back on, multiple clients? no problem. We moved from a third party provider to a direct exchange link (faster and more information!). We had > 95% code coverage on our core stuff, integration suites including a fake stock exchange that actually sent packets over UDP so we could force various problems with retry reconnects etc/errors. We were very stable and had a proper clean architecture.
In fact you could say that we were dealing with what Bob describes in:
As time passes your estimates will grow. You’ll find it harder and harder to add new features. You will find more and more bugs accumulating. You’ll start to parse the bugs into critical and acceptable (as if any bug is acceptable!) You’ll create modules that are so fragile you won’t trust yourself, or anyone else, to modify them; so you’ll work around them. You’ll build a festering pile of code that, with every passing week, requires more and more effort just to keep running. Forward progress will slow and falter. It may even reverse as each release becomes buggier and buggier, and less and less stable. Catastrophes will become more and more common as errors, that should never have happened, create corruptions and damage that take huge traunches of time to repair.
We had built a production prototype and were suffering all the pain described by Bob. We were paying down our debt in an “intelligent” way much the way many companies that start with production prototypes do.
However this is still a naive viewpoint. What really mattered was that after our nine months of beautiful architecture and coding work we were making approximately 10k/month more than what our stupid production prototype made for all of its shortcomings.
We would have been better off making 30 new production prototypes of different strategies and “throwing shit at the wall” to see what worked than spending any time beyond a bit of stabilization of the first. How many new business opportunities would we have found?
There are some lessons here.
1) If we had started with a nine month project it never would have been done
2) A Production Prototype is common as a Minimum Viable Product. Yes testing, engineering, or properly architecting will likely slow you down on a production prototype.
3) Even if you succeed you are often better to stabilize your Production Prototype than to “build it right”. Be very careful about taking the “build it right” point of view.
4) Context is important!
Never underestimate the value of working software.