Just because we think something has value, does not necessarily mean it is valuable.
Often our perception of what has value is driven by a personal bias, such as that t-shirt from college you just can’t throw away. That t-shirt delivers value in the form of memories, however, you can’t derive anything valuable from the t-shirt outside of your memories.
There is a difference between perceived value and delivered value.
Value cannot be realized unless it can be quantified in some way.
This may sound like a semantic exercise but I think the distinction is an important one that needs to be made as it drives how we fund software development projects.
When organizations go through their annual planning cycles, they ask their functional leaders to come up with software projects that they want to do.
In this context, leaders may often turn to what they perceive will provide value, however, the value they perceive may not deliver the value the organization requires from a strategic standpoint and it often is not quantifiable.
This happens when the organization is not aligned with a shared understanding of the strategic outcomes that will deliver real value to the company.
There is often an inherent imbalance in many organizations, where you have one part of the organization working to eliminate some technology or functionality that another part is actively seeking to implement. These are real cases I’ve seen as a coach.
The way to deliver valuable outcomes (ROI) to the organization is to align your work to your strategies.
My valuation model takes your strategies and converts them into value factors that are applied across your entire software development Portfolio.
Using a portfolio investment approach we then define the organization's risk/return profile which will ensure that we are focused on investing in work that is both foundational as well as aspirational.
If you don’t have balance you will never reach your aspirational goals as your foundation can’t support it.
Software development is an expensive endeavor with annual investment running between 5%-10% of total revenue or more for most organizations.
Not knowing that your investment is delivering real value makes that cost go even higher.
Software Engineering is Not Engineering (More Like Art)
Recently I had an interesting conversation regarding whether or not software developers were engineers or not. The most compelling argument that they could provide for why Software Engineering was truly an Engineering discipline centered around the fact that Universities and Colleges have their Computer Science group in the Engineering department. That is not sufficient in my mind to by default call Software Engineers well, Engineers. Why? Consider the following
Engineers who build skyscrapers, airplanes, bridges and products that require low tolerance for defects are often operating under the laws of science and as such require that they account for many scientific elements when building their products.
They perform a great amount of 'testing' both in theory and with product before something is released for use. Their engineering practices are based upon scientific principles that go through extensive peer review and have been building in knowledge for hundreds if not thousands of years.
Now lets look at the definition of Engineering:
- The branch of science and technology concerned with the design, building, and use of engines, machines, and structures.
- The work done by, or the occupation of, an engineer.
- (Business / Professions) the profession of applying scientific principles to the design, construction, and maintenance of engines, cars, machines, etc. (mechanical engineering), buildings, bridges, roads, etc. (civil engineering), electrical machines and communication systems (electrical engineering), chemical plant and machinery (chemical engineering), or aircraft (aeronautical engineering)
Though of this I think can be applied to the science of computers, ie the hardware = Machines I don't however see much if any of this being applied to Software Development.
A paper in 1996 (William Aspray, Reinhard Keil-Slawik, David L. Parnas) talks about some of the reasons that writing code isn't associated with Engineering:
"Computer science is often characterized as an engineering discipline with the systematic study and development of software as its principal subject matter. Software Engineering, however,although combining both key words, has not become a central discipline in most computer science departments. In many respects, this discipline embodies the same ideosyncracies that can be observed within computer science as a whole such as:
• Highly innovative and rapidly changing field with no broadly recognised core of material that every practitioner must know; • Few results are supported by empirical or comparative studies; • Work within the field older than 3–4 years is rarely acknowledged or referenced; • Old problems are given new names and old solutions overlooked; • Evolution of the discipline is tightly coupled to economic and societal demands; • There is a need for interdisciplinary work comprising e.g. mathematics, psychology, business or management science, … ; • Continuing debate about whether there should be a discipline called software engineering, and if so, whether this should be treated as another discipline among the set of traditional engineering disciplines."
As you read through this I think we can start to see a clear delineation between the disciplines required of 'Engineering' and the disciplines that Software Development requires (Note - This is not an article to criticize people who write software, but rather a way for us to better understand how to manage them and how to help them deliver even better software)
Go to almost any software development team and you will find a mix of people who have ended up going down the path of writing software, they can be people who were Computer Science majors all the way over to people with a Liberal Arts degrees.
The languages that allow us to write software today have matured over the years and as such have become easier to learn and work with. This mix of skills and backgrounds is not bad, but it deviates from what an Engineering team building an airplane might look like. Full transparency here, I have never worked in an organization that does true Engineering, but I know many people who do, and I suspect that you would not find people who are designing/engineering bridges or airplanes who don't have a strong educational background in Engineering, but you will find these people in your software development teams.
So why is 'Software Engineering' more art than science?
I think that Software Development is more art than science because we must create software that we interact with on a more human basis. Creativity abounds in web development as we continue to strive to bring intuitive and easy to use interfaces to the people who are our 'users'. The people who write software must be able to synthesize multiple layers of code, from Front End to Back end and everything in between. This talent is probably more aligned to writing a symphony in that you have to deal with an entire array of players, each with different instruments (API, Legacy, etc..) and when you get it right you get beautiful music, when you get it wrong you get noise.
When writing software we have the option to try multiple paths to an end, whereas when building a bridge our ability to deviate from proven engineering practices success is much more limiting and catastrophic when we do when we don't.
One can also argue that the newer languages being created are designed to make writing software more accessible to more and more people who have not be through a computer science programs, we are striving to get our languages to ever more English like syntax.
Consider also that people who write software are not held to specific engineering practices. Software, as it has evolved, has provided people who 'develop' software product with endless ways to implement it the underlying code. The fact that there are not any set standards that every Software Engineer must use I think clearly shows that software is not a true engineering discipline, but rather a creative one. Artists embellish, they innovate, the push boundaries into new ways of performing....I think this better describes many of our software developers more than engineering.
The beauty of current software development is that you have the ability to envision new ways of implementing code that delivers functionality and in doing so create new coding processes that others can use. Software languages provide many different ways to write code to deliver a product functionality, that might be 3 lines of code or it could be 100, it depends on the background and experience of the individual writing the code.
I think when we start thinking about how to manage and support our Product Development teams in our organizations we would benefit from providing them the framework that encourages high quality through innovation, failing fast, and supports both the individual and groups ability to determine their success. That is not to say that Sr. Leadership doesn't play a part here, it certainly does, by providing clear vision of what is wanted (think of a symphony conductor, who must bring out greatness while providing their interpretation of a particular piece), an ability to accept failure as part of success and a fearless ability to have confidence that when you provide the other two components people will bring greatness as a by-product.
Artists operate in a very Agile manner, we listen and feed off of each other and in the process discover new ways to approach our art. Listen to jazz music and you see the elements of Agile and creativity evident as the small group (Scrum team) takes a song and through listening to each other can create new music, sounds or approaches to an old song.
As with Artists, Engineers don't like to be told something can't be done, that quest to make something happen that has never existed before is what an artist strives for, there is an emotional rush that comes with artistic creation and I think the same is true of software product creation.
Engineers who build skyscrapers, bridges and planes are no less artistic or innovative in their designs however they must take a much more rigorous approach to what they deliver, they simply don't have the ability to go back and redo it. If they get it wrong people die, though there is of course software that is also mission critical, but for the vast majority of us who develop software for a living, we are able to take a less rigorous approach when we create our products and we expect in Agile to move to a continuous delivery model so our work is never 'done'.
Agile Planning - I have a need a need for speed
Working at Disney a number of years ago was in so many ways transformative for me (not sure why I left) because it provided me with an opportunity to work with an organization that needed to get better at delivering software for our partners and we ended up choosing Agile as our path. Disney was the place where I had the opportunity to help build an Agile process from Requirements to Delivery and what we discovered was that we needed to develop an effective planning process that allowed us to build a solid backlog of work before we just started coding. I here that so often in organizations that are just starting to adopt Agile. I think a statement I heard recently is descriptive of organizations that just start coding - Shoot and Point.
Disney is a largely creative driven organization (Not surprising) and because of this we typically had a disconnect between our creative (UX) groups and the Product Delivery teams. The UX team primarily worked independently of the PD teams and as was the case when I arrived, UX would deliver a creative design that didn't align to our technical capabilities. This is a common issue in today's web development environment.
Our first 'Release' Planing went very poorly and after a round of retrospectives we came up with a format that at first pass you would say wasn't Agile (trust me we used that phrase a lot in the early days of our becoming Agile). But in the end this first step of Discovery ended up being what I believe is the most critical element of being able to go fast in Agile.
The basic process that we ended up with was as follows:
- Pre-Discovery - Sr level PD, PO, UX, Marketing and other Stakeholders would review a specific new feature that was being considered. The group would utilize several tools such as Mind Mapping to understand the scope, parameters and potential dependencies at a high level. If the feature work was approved then we went to the next phase.
- Discovery - Depending uponthe the size of the potential project Scrum teams and extended stakeholders would meet to go through a low-level review for that feature development. For many of our larger efforts it was not uncommon for us to sequester the teams into a room to work through the entire effort, UX to QA to Delivery.
- Process
- Kick off - Have your PM or PO along with the key stakeholder(s) of the effort describe WHY this feature is so important. We learned in our process development that it helped our PD teams to have an understanding as to why this feature was important to the organization. It helped them feel connected to the value that was being delivered and not simply code jockeys running a race.
- Competitive Review - Another great exercise was to have the entire team go out and find competitive features that we either did or didn't like and describe why. This helped the next phase of our Discovery process as we worked to define what our feature sets would ultimately look like
- UX and Story Development - This was the primary scope of our Agile workshops. Typically led by the UX lead for the project we would begin developing low-fi wireframes and discuss the issues, constraints and code complexity that the low-fi would entail. We discovered in this process that we could work through the types of issues that come much later in a typical product delivery effort.
- Outcomes -
- UX ended up with designs that they could utilize to develop prototypes that would be used in User testing prior to any significant development work being completed. This allowed changes to UX to be found at the very beginning of the PD process rather than at the end when refactoring consumes a much larger amount of time and leads to lower quality of code.
- PD ended up with a solid design at the beginning of the PD process which led to high quality code and higher levels test automation.
- QA ended upon with an ability to write higher quality acceptance criteria which lead to high quality in the delivery and higher levels of test automation (sensing a theme here?)
- User Story development was done during the Discovery phase and with it I was able to have a fairly accurate model to predict the number of stories at the beginning of the Discovery phase (typically between 100-120 higher level Epics, we strove for stories to be between 21-34 points in this phase as PD would start fairly quickly after the discovery phase 2-3 weeks) and how many that would translate into for a full project (typically 350 - 400). This provided me with input as to how many BDD acceptance criteria would come out of this as we used a marker to determine when a story should be broken down - More than 7 variables in the BDD would be an indication that it's time to think about breaking down the story and more than 14 tests in a single test scenario would also trigger the conversation of whether to add a scenario to a story or create a new story.
- Benefit - Keeping your BDD test automation in small increments makes it much easier to understand what broke, who probably broke it and what is needed to fix it.
- Outcomes -
- Process
I know this doesn't 'Sound' Agile (like the name of my company), but in my experience doing this small amount of work up front does provide teams the base to go really fast once the PD process begins.
I have used this process now for many years and when we do it right it's like writing a symphony, all of the moving pieces make beautiful music. When it isn't done right, then all you get is noise.
This process probably does work for larger and more complex organizations over small organizations, but really would you start building a house with no blueprints and no idea of what you wanted? If you had builders just show up and you told them I need a house to live in and I need it fast you will get that, but I doubt it will be anything that you want. And in reality it wouldn't be done fast as they probably wouldn't have the right materials scheduled to arrive at the right time. I have my roofing supplies but the foundation company can't some for a month, see what I am getting at?
Slow down a bit, understand what you want, how to get it and then go fast to get it.
BDD - Step by Step Process Tutorial
As you might tell from some of my blog topics I'm a huge believer in BDD, not just as a means to automation but more importantly as a process that helps us define requirements more accurately. When we move into an Agile delivery model, many individuals struggle to understand how they are supposed to document everything that they used to put in their Product Requirements documents (or a whole host of other acronyms). These documents formed the basis of our engagement with the business and with the teams that would ultimately deliver what the business thought they were asking for in the PRD.
There is an interesting game that has many flavors such as Social Telephone game, that for me, makes clear the issue with large documents that multiple groups need to review. Each of us is individually focused on the things that we think are important and we often have different interpretations of the written word. And as such as we work to deliver in our silos we move towards an uncommon understanding of what is being built. I would argue that many 'defects' that are found are actually poorly interpreted requirements.
For those of you in highly regulated industries you know what I'm talking about. The endless hours spent reading a single paragraph trying to derive the real intent and meaning behind the written word is exhausting and fraught with peril because if we get it wrong there are penalties and mad scrambles to implement a hack to make it work the 'right' way.
Software development is made hard because of these communication differences, be they cultural, language or other, they exist and they make it difficult to come to a collective understanding of what is being asked for.
User stories help break down this barrier as they define small pieces of a larger whole and place a value statement so that we understand WHY what we are being asked to build is important. Many teams writing user stories leave off that important 'so that' statement. Leaving off 'so that' leads to so what?
Behavior Driven Development, for me, changed the way that I looked at defining requirements because it removed the business language of things such as the business requires this, or the feature shall do that. The statements sound powerful yet deliver little in real context for the teams to work from.
Breaking down user stories with BDD still leaves the team to work with a non-technical domain language and then provides a clear method of defining the behavior (positive and negative) for the teams to work from.
The power of context with the respective speed and quality it delivers can't be denied. However don't be fooled that this is easy, it isn't. Agile is extremely disciplined, something many miss on their way to the Agile party. Done right it can take your delivery processes to new heights, done wrong it becomes just another new fangled process that doesn't work.
Below is a BDD tutorial that can get you started on your way to learning how to build contextually rich user stories. Good luck and have fun!
Being Agile - Say it, Do it, Prove it
I was working with a team recently and as we talked about all of the planning that Agile entails, I broke it down into very simple terms - Say it , Do it , Prove it. That is really what Agile is about. Anything else outside of these three concepts is noise to your ability to deliver product and services to internal and external customers. As Product Owners in an Agile organization, you need to understand all of the effort and dynamics involved in getting your teams to Say it, Do it, Prove it.
Delivering what you say you are going to deliver is the best way to build credibility with your stakeholders.
For Agile teams, this translates into being effective at decomposing your stories into small enough increments so that you are confident in your understanding of the user story and estimates that the team believes in.
- Say it = Release Planning and Backlog Grooming -
- Starting at a high level, the Product Owner is responsible for saying what is important to the organization from a value standpoint and beginning the process of developing a user story backlog that supports this vision.
- User story decomposition is so important to effective Agile teams and the Product Owner must start with a set of well-formed stories that provide context to the team.
- What is 'context'? Context is anything that provides definition. It is basically what the product should do from a functional standpoint. One of the biggest mistakes many teams make is writing declarative stories that start with the 'How'. This,\ in turn, puts the technical team on the defensive as they may have many different ways to implement the feature. As a Product Owner, be sure to steer clear of writing stories that define how you think the feature story should be implemented. I know that as we all become well versed in technology there is a strong desire to show off our technical chops, however, as a PO you need to provide context from a business standpoint that your tech teams can consume. I've heard time and again from engineers that they would really like to understand how what they are developing delivers business value or solves a particular pain point for the customer. The team works much more effectively when they are completely grounded in the business context of what they are building.
- Do it = Sprint execution
- An important element for teams to address once they are ready to take stories into a sprint, is that the goal during the Release Planning and Backlog Grooming activities was to begin to build out the context of 'How' the story will be implemented. It is so important for teams to understand that there is essentially a handoff from the PO to the Scrum Team and that each of them is responsible for building what I call contextually rich user stories. Gojko Adzic calls this Specifications by Example. Effective teams who deliver fast and with high quality work closely as a team.
- I believe that the combination of User driven stories and context driven specifications (examples) forms an extremely strong definition of both Ready and Done. which is why I coach my teams to utilize BDD as the basis for developing their User Story acceptance criteria. The team works together to complete BDD acceptance criteria forming a clear understanding of the boundaries of the feature. This provides the PO with a concise view of what to expect during the Demo.
- Another key benefit of writing BDD as part of your user story writing is that the test automation engineers can easily consume this as part of their code development for the automation. PO's should push to get to this level of context as it also means that your test engineers can start developing their test automation code once the story is ready for development. They can essentially perform TDD in that they can write their automation before the feature is actually developed. Once developed the automation should run cleanly and both speed and quality are attained.
- The goal of teams should be to deliver user stories that do not require much further elaboration once the sprint begins. You want your teams focused on delivery ,not on discovery.
- Prove it = Retrospective
- You have done all of the work to clearly define your user stories with high levels of context. With all of this effort, the Retrospective should be an easy affair to show the work that was defined in the stories. The PO should not have any surprises. In fact, if the team misses any test scenarios after the story has been started, the PO should consider that more of a missed requirement over anything else. Since the entire team developed the stories, there can be no finger-pointing at anyone. It was a shared miss and everyone must accept it.
It sounds really simple when broken down into these 3 basics phrases. The truth is that 'Being Agile' is much more involved than simply 'Doing Agile'.
"Being Agile" means exposing all of the inefficiencies in your product development processes. It requires that the organization be completely honest in assessing what is not working and committing to letting the teams that do the work fix these processes. You cannot top down drive the type of organizational change that is required for Agile continuous improvement. Real organizational change is fostered at the top but truly owned by the Scrum teams that form in support of any Agile adoption.
Product Discovery - Uncover your backlog in Agile
Much of the writing and learning that you gain about Agile focuses on the day to day nuts and bolts part of execution, however I think that teams are missing a much greater opportunity to improve the quality of both execution and delivery if they spent more time developing their backlog. My teams consistently hear me talk about getting contextually rich user stories as part being able to delivery quickly and with high quality.
In order to get to the point that we have contextually rich user stories teams need to spend some dedicated time building out their product backlog.
Whether you are developing an entirely new product or simply adding features to your existing product, taking the time to fully investigate and build out your User Story backlog is important.
Much is said about doing just enough documentation to get the job done, but what I've seen over the years is that teams that fail to build out user stories that have a specific set of information in them will spend more time trying to figure out what they are doing IN the sprint which negatively impacts quality and the teams velocity.
There is nothing wrong with spending time before a set of sprints building a strong understanding of what the team is going to do.
You wouldn't try to build a house without a certain level of planning ahead of time. Agile teams often arrive at the start of their Sprints with nothing more than a set of ideas. If builders showed up at a job site to start building your home with only a rough idea of what you wanted, they couldn't go very quickly nor would they build something of quality that met your needs and expectations. You might get what you want, but probably not.
Agile teams need to take this approach a bit more, spending time with their Product Managers to not just flush out the features that are wanted, but more importantly the lower level details needed to provide the team an ability to plan more effectively and deliver high quality products.
If a builder shows up and wanted to pour a basement, they need to know more than that. They need to know the size of the basement (ie how much concrete do I need) , if the site has been prepared, has drainage been considered, etc.... Building anything of quality requires planning before you can deliver.
Agile I think too often believes that you can go lite on the planning and still delivery quality. If you think that refactoring is the golden ticket for lack of planning, it's not. Rarely do teams have time to do the amount of refactoring they require to keep their applications clean and scalable. This leads to applications that grow in complexity requiring even more time to add features, all the while thinking that we can go lite on planning.
As you consider Agile as a development process, don't forget that Product Discovery is an important element of delivering high quality product quickly.
Agile, Change and the things we can do
We home school our children and a few years ago my wife decided to utilize stories and a Kanban board to manage the kids work flow every day. Doing this provided her the ability to build a backlog of work tasks that our children were going to work on (no we didn't have them perform estimations or commitments :) ) Each day she puts up the work that they need to complete in the Not Started column and then the kids can decide which ones that they want to work on and pull the ticket into the In Progress column.
Once they are done with a ticket my wife reviews the work and if accepted moves the ticket to Accepted. This approach provided our kids with some level of control over their work flow and also an understanding that they weren't done until Mom signed off on their work. This way of managing their work provide them visibility to how much is left for the day and more importantly the topics that cause them more trouble.
Now that my daughter is working on her first business/website (she's 13) we are using an Agile Discovery process where we are building a User Story backlog (she's learning VOC) and developing low-fi wireframes (she's learning design) for her website where she will be selling her customized jewelry made out of recycled products, photography of the animals she has seen across the country and raising awareness of animal conservation. She's now comfortable with an Agile process because we laid the groundwork during school. I think in many ways teachers could utilize this process which would provide visibility to high performing individuals and those that are struggling. Taking a page from Scrum perhaps the high performing individuals can provide support to the ones that may be struggling, just like we would expect our Scrum teams to do when someone is struggling with a particularly gnarly code problem.
Agile is most commonly associated with Software Development, but really the iterative continuous improvement concepts apply to anything you want to focus on.
Agile can be applied to changing your habits. For example if you want to lose weight, write an Epic stating that value of losing weight, then write several thematic user stories with your intermediate goals and then write stories for your sprints with specific activities you need to accomplish (exercise, etc...) put it on the wall for your Kanban tracking. If necessary have a standup with people who are supporting you so you can report on your progress, impediments (Ben and Jerry's is right next to my office...)
An underlying principle of Agile is the need to change behavior, not just of the teams building your product but those that want them built. You can't have a smooth running organization if you don't address the intrinsic behavior of people in general. Agile can provide the framework for helping make this change by providing a shared language.
When we work in an organization we may think we are all working towards an end goal (ie making great products, making money, etc...) but as soon as an organization begins to grow people who don't know each other and who may have no shared experiences need to start communicating.
I recently completed McCarthy's Core Protocol training and one of the things I came away from it was that getting people to have a shared framework of communication is the most important thing that high performing teams and organizations should focus on, the rest will come as a result of this effort. If you deliver this you deliver the ability for these teams to produce great product anywhere/anytime.
Think about it this way - We are all organic systems, who behave based upon the experiences that we take in over our life time. Since no two humans are organically the same ( not even your own family) we end up with an ineffective communication driven by our life experiences and trained responses.
Core Protocols and in my mind Agile, provide a framework to establish a common language, just as we do with disparate technical systems (isn't that what a service based architecture is all about?), we write code that provides a common communication protocol between them, removing the impediment their different languages pose to having them communication effectively.
As you begin to start to adopt Agile understand that though the practices you read about, such a Release Planning, etc... are extremely important, the need to provide your teams and organizations a shared communication framework are equally important to long-term success.
Sound Agile - Aligning Agile with Soundness
As I indicated in my last post I renamed my blog to Sound Agile and I'll be writing about Agile from the perspective of Soundness and how organizations should be evaluating their Agile adoption. I spent quite a bit of time trying to decide what I wanted to name my consulting business and blog (both are called Sound Agile) and I ultimately wanted to convey what I have seen as successful with respect to the teams and organizations I have worked with over the past 10 years.
Thanks to my wife for providing me a word that when I looked at it provided me with the right context to what I had been trying to convey.
Sound:
adjective
1. free from injury, damage, defect, disease, etc.; in good condition; healthy; robust: a sound heart; a sound mind.
Supporting Manifesto - Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
a. Free from Defect - Speaks to Quality that we strive for in Agile. b. Healthy - Speaks to your employees health and well-being c. Robust - Speaks to building scalable products. d. Sound Mind - Speaks to teams keeping a clear and open mind to improve
2. financially strong, secure, or reliable: a sound business; sound investments.
Supporting Manifesto - Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
a. Sound business/investment - Speaks to the value proposition that Agile focuses on. b. Financially strong, secure and reliable - Speaks to the ability of Agile teams to delivery products that are foundational to your organizations long term health and security.
3. Competent, sensible, or valid: sound judgment - This speaks to trusting that the people that you are hire are compentent and when following Agile ceremonies such as Standups, Retrospectives and Planning are able to deliver product that is built on solid decisions.
4. Having no defect as to truth, justice, wisdom, or reason: sound advice. - This speaks to the continuous improvement and Retrospectives that drive this effort.
5. Of substantial or enduring character: sound moral values - Speaks to team commitments, believing and supporting themselves, the foundation of Scrum.
We spend so much time writing and talking about things that we should, could, might do to deliver Agile, We hear people talk about how 'Agile' are you, tell people that doesn't 'Sound Agile' and I think we have muddied the waters of what Agile should be.
When you hear Sound Agile you may be inclined to add an additional adjective 'Sounds Like Agile' and to some degree that is expected. My focus however will be on the soundness of Agile from the perspective of organizational change an area that isn't really talked about as much as the other elements of day to day Scrum.
I look forward to comments and input to my blog as my writings will form the basis for the types of coaching and consulting I hope to do in the future.
Technology isn't important, Product is
We are so enamored with technology today, what it can do, the excitement it brings to us as technologists that we have lost sight of the fact that technology isn't the end game for a business. Why? Because if your organization could figure out a way to run their business so that it wasn't dependent upon technology they would do it in a heartbeat. The people who own and run businesses don't care about the underlying technology, even those who classify themselves as technology companies. At the end of the day technology enables the business to develop product. They aren't in the business of developing technology as end product and I think that goes for companies who deliver 'technical' products.
Many teams I've worked on in the past 10 years have isolated themselves from the organization, instead telling stakeholders they were focusing on technical excellence. Leaving the stakeholders wondering why they weren't delivering what they wanted all in the name of technical excellence.
I worked on one project that was so late that the business ending up exiting the market that they were in. In response the Development manager said if only the stakeholders knew how cool the technology was that they had worked on they would change their mind. Really? They could care less, all they wanted was a product that would keep them in the market and making money.
Though the amount of information we can process has exploded we aren't really solving really new problems, they are just much larger in scale and are required with much higher availability. At the end of the day business runs on information that drives product decisions that build value. Like any other part of the organization we are responsible for delivering value, not technology.
If we could deliver data to our business without all of this cool technology like hadoop, etc....they would be just as satisfied.
We should spend much more time on how we can deliver what we need for the business with less over building systems that continually build in complexity because we aren't product focused. Think about it, organizations that deliver at scale, such as car manufacturers continue to refine their processes so that complexity is removed, whereas we seem to have to add complexity in order to deliver value. They add technology where it makes sense not just because it's the latest thing.
Technology isn't important but our technical product is. We have architects, tech leads, dev managers, however I don't often see them actually defining the technical product that the business needs, rather they evaluate specific pieces of it, SQL server over Oracle, Java over .Net, etc.....and with each iteration individual personalities can influence the type of technology being implemented, often times in direct competition with the documented tech stack that company operates in.
What we end up with is a technology product that doesn't scale well and adds cost to the organization in both resources and revenue delays. When confronted with the mess we've made we tell the organization that can solve the problem with a new platform. Unfortunately we often fail to fully deliver on that promise for the same reasons that we got into the situation in the first place.
This is not to say that we don't try to build systems right. Executives also need to bear responsibility for not providing the Engineers you have hired the support and time necessary to actually build technical products that deliver value to your business product, they are one and the same in today's world.
When your technical team is forced to meet dates, the inherent compromises that need to be made typically result in short cuts to your technical product in order to deliver the most for your business product. You can do this in the short term, but if you don't provide your teams time to shore up these short cuts your technical debt continues to grow. And like any debt you eventually have to pay it. Pay it at the end of every release like a credit card you won't incur finance charges. Continue to pay just the minimum and you eventually can go bankrupt.