Top Down or Bottom Up – Large Scale Agile Adoption

So I’ve worked in both large and small organizations where we have gone through an Agile adoption or the phrase of the day, Transformation.

Having seen both sides of the coin I started realizing that you have two paths to take when considering moving your organization to an Agile delivery methodology.

I use the phrase delivery, because I think at the end of the day that is what we are talking about.  Strip away the manifesto’s goals of conversation over documentation, accepting change, etc…What are are really talking about is moving an organization from a Project delivery methodology to one that is Product delivery oriented.

What is the difference?  From a Management perspective actually quite large.

  1. Project Delivery – These have very strong controls which move people to a new project.  Budgets are set up for the specific time period of the project and then up front requirements and design are completed in order to ensure that the project is fully ready to be engaged.  Project Managers manage all facets of the project via extensive project planning and plans.  Management receives up dates as to the project progress on a regular schedule, usually weekly. Resources are assigned either in full or in part, yet no one actually monitors nor can they really manage whether or not someone is working 25% on a particular project.  Projects tend to focus on reporting and there is high pressure to ensure that individual projects are green, which drives teams to deliver on the easiest and often less valuable parts of the project first and only at the end of the project is the hard work tackled, which reveals itself as budget overruns or timeline delays (or worse delivery of reduced scope).  Project reporting is elaborate and management receives project reports that are often sanitized.  Value is typically not delivered to the organization until the end of the project.
  2. Product Delivery – The work that is done for a Product is centered around a value stream it delivers and the work is ongoing.  Teams are funded as a whole and are kept together long-term in order to maximize their productivity.  Work is planned out in short increments called Release Plans that span anywhere from 6-12 weeks, with 2 week sprints.  Management receives regular updates (2 weeks) but can access information radiators at anytime, transparency is the key and goal with Agile Product Delivery.  Teams commit to work in 2 week sprints and their commitment is key to building trust with Management.  As time goes by management can trust that both a teams abilities and productivity can be counted on.  Teams are focused on delivering high quality code to production every two weeks which brings value to the organizations investment in them along with the increased value in terms of new sales, reduced costs, etc…Feedback loops via Product Demonstrations provides management the ability to assess where they are going with the product and deliver not what was asked for up front (Project Delivery) but rather what is actually needed (Product Delivery).

So what does the difference between Project and Product delivery approaches have to do with Agile adoption, well everything.

Most Agile adoptions begin at the bottom of the organization with the teams tasked with developing new software.  These efforts are borne, as the Agile manifesto was, out of frustration with how software was being developed in their respective organizations.  Often management is aware of the issues these teams face but are unable or unwilling to make any changes to how things are currently working and why would they?  You learn very early in your career that rocking the boat is not something that goes over well with organizations, my first boss told me I couldn’t by computers that weren’t from IBM because you don’t get fired if you buy IBM. The message is that if anything goes wrong you need to point to well-known names, processes, etc.. with which to blame or use as support.  To think that this doesn’t go on today is to place your head in the proverbial sand.

Though we can have great success with bottom up Agile adoptions with respect to improved productivity within small groups/teams, the overall Project oriented organization is typically still in place.  Management still wants to see project plans, have things ‘planned’ out for up to an entire year, they aren’t comfortable with the fuzzy feel of product roadmaps.  They want commitments, even false ones, so that when things fail they can point to the fact that they had all of their planning in place.

For Agile to really take hold, Sr. Leaders need to change the way that they manage both their people and the work.  It starts first I think in understanding that we have not learned how to speak to management very well yet from an Agile/Scrum perspective.

We need to understand what management is really concerned about and then center our product delivery efforts around that.  One of the problems that we face with some of our leaders is that they themselves don’t always know what to be focused on, they are looking at multiple balls in the air but at the end of the day as a Sr. Leader I think I have just a few things I should be focused on:

  1. Growth – This is often related to sales, market and revenue.
  2. Profits – Tightly aligned with the first item, our ability to make a consistent profit is what helps us continue to reinvest in our company.  Ever increasing revenue or sales without corresponding profits will eventually lead a company into bankruptcy, money isn’t free and it is not endlessly available, in spite of what we think we see with new technology organizations.
  3. Organizational Excellence – Because none of the above can happen unless you have a great organization.

Agile actually addresses all of the above, yet we spend more time talking about how we will improve our individual work efforts which causes us to  fail to tie this to the needs of management.  Management on the other hand often views the improvements that come from the bottom up approach as more of an anomaly rather than an organizational improvement worth adopting.

Trust is the missing component when it comes to conveying how Agile will make the entire organization better.

Agile isn’t easy and it requires skills that frankly many of our Sr. Leaders lack or don’t fully utilize and the politics of most organizations reward behavior that doesn’t align with Agile principles such as transparency, open honest conversation and openly questioning the status quo.  We have people in power who got there by way of the non-Agile status quo and changing that means they have to learn how the new game is played in order to stay on top, it’s much easier to keep things they way they are over learning how to navigate the new.

So how do we speak to our Sr. Leaders with respect to Agile?

  1. Better ROI – Talk to any Finance executive about what they look at when purchasing a piece of equipment that will deliver revenue and you will hear them talk about Net Present Value of the investment, Positive Cash Flow and Depreciation costs.
    1. We improve ROI in Agile due to our focus on only the most valuable items.  In non-Agile project work often are working on features/functionality that may be important to someone inside the organization yet will bring little or no value to the organization.
    2. When we are able to start talking about the value streams of our organization, be they revenue, cost reduction or improving our brand image we begin to be able to have a better ROI conversation with management.
    3. We also positively impact ROI via higher levels of productivity gained with dedicated teams.
  2. Flexibility – One of the most important elements that 21st century organizations require is the ability to be flexible enough to react to market forces or reactions.  Financing large projects far out into the future with the expectation of some level of return and no we don’t really have great track records of predicting future ROI out very far into the future.  With Agile we provide the framework to identify the most valuable work for the business in small planning windows.
    1. Sr. Management needs to understand that this flexibility comes with an obligation to have consistent short term review windows as the team progresses so that we deliver what is actually needed and not what we thought we wanted.  You may have thought you needed a Ferrari when in fact what you needed was a mini-van, we provide the framework to course correct via Sprint reviews every two weeks.  If you plan all of your project up front for the Ferrari, we’ll certainly try to make that happen, but in reality as the end of the project nears you will probably get the car but with a lawnmower engine and no brakes, it may look like a Ferrari but it won’t operate like one nor will it provide the value the organization really needed.
  3. Predictability – Another key element that we deliver with Agile is predictability and accountability.  Your teams will be much more accurate in planning and delivering in short-term 2 week sprints with a planning horizon of 6-8 weeks.  What management needs to look for is consistent delivery of the committed work that the team makes, commitment is everything.  What Wall Street analysts look for is a business that can provide a solid ROI, react responsibly to their competitors or even better be a market leaders and provide predictable results year in and year out.

So the question at hand is what is better a Top Down or Bottom Up adoption?  My money for long-term success is on the organization that can consume what Agile really means, not just to their development teams but the organization as a whole.  You can’t BE Agile if you don’t make the paradigm shift from command and control to one of collaborative and collective delivery.

BDD Adoption – Taking time to get it right

Recently I’ve had the opportunity to speak with a broad range of Agile coaches and consultants who have consistently told me that they have not seen any organization successfully adopt BDD as part of their standard process of elaborating their User Stories.

I’ve had the opportunity to work with several organizations that were able to successful adopt BDD and believe strongly in how this process can transform how you build your products.

I think most people hear the word BDD and they immediately think that it is an automation centric effort, though that his a great value add for BDD, I’ve adopted BDD without automation and it led to improved testing and quality.

Adopting BDD is so much more than automation (which I wrote about here) it’s really about clearly defining the expected behavior of your system.

Why is this so important? Because as humans we all interact differently with the systems that we encounter.  As people who develop these systems we deliver based upon our experiences and understanding of the system we are building.

Language (Business Requirements) has been the manner and method that we have used to convey what we want a system to do to.  However with an international work force we don’t all have a shared language.  In addition to this constraint we also interpret what we read differently based upon our primary language, life experiences and culture.  Given this, it’s not surprising that when we build a system, it has many different interpretations attached to it, which negatively impacts both the functionality AND quality of our system.

BDD is a way to talk about how our system should behave not what it should do.  Yes we want our systems to have specific functions and features, but talking about how each of these will behave brings about a much richer conversation of what can happen in many functional situations the system and the people will encounter, not just the happy path.

There are two primary components of BDD:

  1. The Given/When/Then test setup
  2. The Example Table

Getting the first part of your BDD is the most important component because it defines the size and scope of what the User Story.  We do that by decomposing an individual user story into Test Scenarios.  One quick way to determine if your user story is perhaps to large or complicated is to evaluate the number of Test Scenarios you can define.  Keep your Test Scenarios to under 4 per each individual User Story so that you can more easily understand the expected behavior for that part of your feature or functionality.

Keeping the number of test scenarios is also important once you move your BDD to an automation framework in that your tests are small and when something breaks it is much easier to find out where you need to address a fix.  Remember development teams should be consistently refactoring their code to improve code quality.  With high levels of automation we can quickly catch when refactoring unexpectedly changes the underlying behavior of our system.  This is where Quality is kept front and center in our organization.

The second part of building effective BDD is the Example table.  My training focuses on getting the first part clear and clean because the example table is an easier effort of filling in the expected results in each of your parameter columns.  As with the first section you can use the number of parameter columns as a means of determining if your test scenario is too large.  Try to keep the number of parameters to between 4-8, anymore will result in more brittle tests and more time debugging your automation code.

Adopting BDD as your primary means of decomposing stories may on the surface seem like a lot of work, but remember that you are only doing this level of detail for each 2 week sprint so the number of stories that you build out context driven BDD is relatively small.  Doing it every sprint means you get better at it every time and you will refine your ability to generate BDD much more quickly.

We should also remember that this is a TEAM oriented approach, don’t relegate this to your QA team, you will miss getting all of the context needed to delivery high quality code and functionality.

Management and Agile – To Succeed or Not to Succeed

As more and more larger organizations look to Agile as a means of delivering software to their customers the one thing that keeps coming back to me is that for any of this to work there has to be transparency and acceptance that a move to Agile will change your organization, not understanding this will court almost inevitable failure.

Agile in itself is an aspirational desire to change the way that we deliver software, one that does away with the project processes that evolved over the years from Waterfall.  I know that waterfall feels sound and safe with all of its up front business analysis, project planning, Steering Committees and that all important Change Management process, but in reality it really is more of a facade than foundation.

Having managed work and projects in both worlds I have seen how it all works.  Recently I was told (in an Agile organization) that Project Managers are responsible for delivery and it was at that point that my thoughts crystallized around my own journey, Project Managers don’t deliver, Teams do.

At it’s heart Agile is about everyone doing their part to make our product delivery better, whatever that looks like for your organization in that moment in time.  Agile by itself is not prescriptive, the Scrum/Lean techniques and processes that have evolved from our Agile journey may be a bit more prescriptive and become more so when we add things like Scrum certifications to people’s palmares.  We need remember that one of the key reasons that the manifesto came into being was an intense desire to find better ways to deliver software, which means the journey has no end and certifications and such are merely ways for us to have a shared language.  Let me repeat, once you commence on your Agile ‘journey’ it doesn’t have an end, you will always be evaluating what you can do better.

So back to the question at hand, to succeed or not to succeed in Agile, what needs to happen?

  1. Trust – First and foremost we need to begin to build trust between Sr. Management and our Product Delivery teams.  If we have a history of delivering late, with fewer features and cost overruns it is really hard for that same management team to flip the switch once you say you are Agile and trust the very teams who haven’t delivered in the past.  Trust is a two-way street and the great thing about Agile is that once you begin to master the techniques and methods that successful teams utilize trust is almost a by-product of that.
    1. Commitments – In Waterfall we are making a ‘commitment’ to deliver a set of value/features in a specified period of time based upon a business requirements document as guidance for what the business/customer is asking for.  When we make a commitment farther out into the future we become less and less accurate with our plans, it is the nature of the unknown unknowns in life.  Things change, they always do, so to expect that our business and software development teams, in today’s ever-changing world, can predict 9-12 months in the future exactly what they are going to deliver is simply living in delusion.  Manage reality or it will manage you.  Commitments in Agile are much shorter in time, basically every 2 weeks.  These commitments however are based upon the Vision that YOU management need to provide.  You say you can’t plan for the future every two weeks I would argue you can’t plan much further out.  By planning and committing in shorter delivery increments, management get an opportunity to change direction without causing massive pain from already planned out work.  We need to be able to change direction or refocus efforts on the things that are most valuable to our business, not what we made big plans for last year that we thought we needed.  Locking in feature development that doesn’t meet customer needs, simply wastes money and loses market share.  When teams make and keep their commitments to you, they gain confidence and you gain trust.
      1.  Context – In most waterfall projects we end up asking for absolutely everything we think we might need, when in reality sometimes 70% of what we asked for (or even less) is more than enough to meet the needs we were trying to address.  Focus on the most Valuable things your customer wants and move on to the next highest value work, not diminishing returns on things customers may not value as much as we might.  Teams want to work on what brings the most value to the organization, development teams when provided the transparency of why we need to do something can do amazing things.
    2. Self Organizing Teams – This one really causes I think the most concern for management.  You in essence are saying that the teams that you currently manage are better able to make decisions about how to delivery our products.  Guess what, they are.  These are people who work in the trenches every single day, know every single issue, impediment, risk, etc… associated with your current product delivery processes.  And every one of them has probably said something to the effect of, ‘If I was in charge I’d to do X to fix this’.  I’ve been a Sr. Manager for many years and have built several high performing teams and one of the best things I can do for them is to provide guidance and vision for what I’m looking for and letting them solve the issues that deliver the solution.  I’m a smart guy but I don’t have all of the answers and if you are the type of manager who believes that in order to be ‘respected’ you have to be the one to manage how Agile will change your product delivery processes you will fail personally and the organization will suffer as a whole.  Teams of people can solve major problems much more easily than one person can, so let them have the ability to self organize and empower them with making the necessary changes to improve your product delivery.
      1. Context – With great power comes great responsibility.  By ceding some level of daily control to your teams you are also doing so with the expectation that they deliver on what they commit to..  If they don’t then they must provide a game plan based upon the Retrospective on how they will solve their inability to hit their commitments.
  2. Investment – Agile won’t come without an investment cost associated with it.  If like many large organizations you have a mess of legacy code mixed with attempts at migrating to newer technology stacks, business requirements and rules imbedded in code with tribal knowledge scattered through the organization.  Agile requires speed in your product development processes which translates into several investment areas:
    1. Automation – When we say automate we mean across the entire spectrum of the organization, if it can be automated you should probably evaluate whether it can/should be.  More specifically we want to automate:
      1. Unit Tests – Developers should be writing unit tests for everything they build, preferably using XP techniques such as TDD (Test Driven Development).  These are not really hard processes but if you are starting from scratch there is both discipline and framework that needs to be built-in order to get to a mature state for test automation.  Unit tests need to be executed with every build, because with that comes fast feedback if something broke, fix it early and you increase speed and profitability.
      2. Integration Tests – Probably one of the harder areas to get high levels of automation in, primarily because the organization hasn’t invested in the appropriate product like test environments.  Be prepared in your Agile journey to spend heavily on getting the right environments in place and highly available.  Testing the performance of your system right before you deliver is a recipe for disaster and delays (remember time is money).
      3. Functional Tests – These are the tests management is probably most familiar with and may even have reviewed at one point or another.  These are typically the manual tests that QA will execute at the end of your waterfall project, where we are not baking in Quality but testing out defects.  Building high levels of automated testing at the functional level gets to what I call, Progressive Regression.  Instead of running a final full regression at the end of a 6-9 month project, why not do it every single night?  You will need to again invest in physical environments but also in people training as many in the QA world are not equipped to handle the new role of a Software Engineer in Test.
        1. Word of Caution – Don’t rely only on Test Automation in your functional testing efforts, you still need real people with hands on testing capabilities because at the end of the day automated testing cannot always tell you when something is bad from an experience or product flow perspective.
      4. Deployments – One of the hardest things that teams fail at is planning for deployments to their environments.  Making your deployment process as frictionless as possible is a high value target for your organization.  Many organizations don’t have a fully functional DevOps org and many in this field are still struggling to figure out how to operate in an Agile world.  Let them figure it out and provide them with the tools that will support automation of critical deployment processes and hold them accountable to doing it.  To many times we purchase tools and then never make the time to actually use them, invest and utilize, that is the key to your ROI.
      5. None of the above work comes without an investment in both hardware and people.  Current requirements processes will change substantially as you move to an Agile cadence.  People will struggle to find their way, some level of productivity reduction is expected in the beginning of an Agile transformation. You as a Leader need to set a clear vision of why the organization needs to change, make it clear that the teams have accountability to deliver the work that they commit to and that you as a leader have accountability to provide them with space to learn, fail and finally improve.  Successful Agile includes failure because without it we aren’t really learning from our mistakes, rather hiding the truth.  Agile done right makes everything visible, especially who is accountable for what.
  3. Patience – Nothing great was ever built-in a day or a week.  Odds are good that this will take more time than you thought it would, but also make it clear to the organization that an Agile death march is not something that you are taking on either.  Agile is about accountability and commitment, use these values to your benefit and identify those that simply can’t or won’t work in an Agile environment.

From a management perspective you need to understand your role in the success of any Agile transformation, it must also change the way you look at and manage your business.

Agile isn’t Easy

Over the years I have seen many teams and organizations who start on their journey towards Agile product delivery make the mistake of thinking that Agile is easy, promotes freely changing direction and worse will fix all of your issues and make everything better quickly and easily.  The truth couldn’t be further from the reality.

There is no such thing as a perfect software development delivery process.  Unlike production lines for things such as automobiles where you have the same pieces going to together each and every time and each piece has a specific functionality, tolerance and timing, software development is the exact opposite.

Software development is about meeting changing needs across the dynamic nature of business  For example – You wouldn’t see an automobile company add anew feature to a car on their assembly line over a 2-4 week time period.  They need time to design the entire process and ensure that the production line is capable of accepting the new change.

Traditional software development tends to align a bit more to the automobile example and in fact there are times where this type of rigid, pragmatic approach to product delivery is actually the correct process (Agile isn’t for everyone nor for every situation).

However for those that are going to move to Agile you need understand that the type of discipline that you might use in the automobile example actually needs to exist in your Agile processes, seriously you ask? Yes – You need to be able to deliver high quality, well-tested and fully functional pieces of software every two weeks. Now are you seeing how difficult Agile can be?

If you think Agile is easy, then you are already on the path to failure and unmet expectations.

It is very common for teams who are moving to Agile to take their interpretation of the Agile Manifesto to the unhealthy extremes, for example:

  • Working Software Over Comprehensive Documentation

Many teams I’ve worked with take this to mean NO documentation and that couldn’t be further from the truth.  We value working software over the need for documentation but I’ve never believed that you can have long-term success with your product without delivering some levels of documentation.  Without documentation your software knowledge becomes tribal and when your tribe leaves the team or your organization, well so does their knowledge.

There are way for teams to build documentation as part of their daily Sprint development work.  Using the combination of user stories and Behavior Driven Development (BDD) acceptance criteria as the foundational elements of your work you are creating your documentation as part of the work needed to deliver quickly.  The great value in BDD is that the acceptance criteria is written in English syntax and then translated into test automation.

This process of writing stories with BDD is supported by Gojko Adzics book, Specification by Example and allows us to deliver light weight documentation during our sprints.  I often see teams adding a story to a future sprint to handle their ‘documentation’ requirements of their previous work but I haven’t seen this work long-term.  Functional development, dealing with bugs, etc… will ultimately push these stories down into the backlog, never to be seen again.

The process described above is not easy, but it can be done and the teams that can get to this level of capability will succeed in driving value to your organization every two weeks.

One of the things that I consistently tell organizations moving to Agile is that it will highlight every current weakness of your product/software delivery methods.  Agile is a game changer, it requires a mind shift in how you look at your product, the work that supports it and how you see the visualize the value your product delivers.  If you are a leader who is going to be uncomfortable finding out truths about your organizations inefficient manner of product delivery then you need to think twice about moving to Agile.

Do you have TITL (To Important to Lose) people in your organization?  If you do, then you need to really look at how they influence any changes in your organization.  Do you need to get their approval, gain their support, etc….?  If so you will find more often than not that Agile will scare the hell out of them.  Successful Agile teams/organizations understand that self organizing teams take away the need for many of the day-to-day management decisions our middle management layer makes.  Agile speed comes when you remove the friction of management layers and provide teams with a clear vision of what you want them to deliver.

You must be prepared for the resistance that you will face from your product delivery teams, not everyone wants to go to Agile  We become comfortable with what we know and do in our daily work life and in that comfort comes stasis.  If you are not prepared to lose people, especially your TITL people, then you need to question if Agile is really for you.

I know it sounds heartless, but I’ve been working for over 30 years and one of the first things I learned right out of college was that technology was disruptive and if you weren’t on board with how it changes how you work you would be left behind. At one company I worked for many years ago, I saw Regional sales managers who had been with the company for over 30 years and when we rolled out a sales force tracking/marketing tool (which I led) there were several who refused to even turn on the computer, they were all let go within months.  As employees we have the obligation to continue to grow and learn and continue to make ourselves valuable to our organization and if we don’t, then you as a leader have an obligation to make tough decisions that ensure that the organization is continuing to grow and not stagnate with old processes.  It’s not personal it’s business.

As you move to Agile you also need to understand the investment that needs to be made in your technology stack.  Many organizations have decades old technology stacks which have been shoe horned into the future and though you get by you won’t be able to become Agile until you have a strong Continuous Integration framework, high levels of Unit, Integration and Functional test automation.  Getting to this will take time (and using the stories and BDD disciplines mentioned above will help you get there).  You simply can’t go fast if you don’t have the technology backbone that supports it.

Agile isn’t easy by any stretch of the imagination, it requires thoughtful introspection, focus on continuous improvement, disciplined delivery and a tenacity for value and quality that is never satisfied.

Baking In Quality with Agile

One of the things that I love about Agile and especially the related techniques of BDD and Test Automation is that if done correctly your teams are essentially baking Quality into your products AS they develop, not after.

Traditional SDLC (read waterfall) took a very linear approach to project delivery which in turn translated into a similar approach to how we developed and delivered our software products.

In our traditional delivery methods, quality was not ‘baked’ in but tested out and we never ever got to the point where we are able to test out all of the ‘defects’ that are found, instead we create a bug database where ‘bugs’ go to die.  Every organization typically has some form of a bug database with bugs that were ‘found’ sometimes years ago and were never fixed (of course begging the question were they ever bugs in the first place?)  In Agile we really don’t want to see a bug database because we should be instilling a zero defect policy for each sprint, meaning that we should never be introducing new tech debt into our product.

As I progressed on my Agile journey I came to realize that there are actually two types of ‘bugs’ that we encounter when we develop software:

  1. Bugs as Missed or Undefined Requirements
  2. Bugs as True Bugs

— Bugs as Missed or Undefined Requirements – I will argue (and in a book that I am starting to write about this topic) that a majority of the bugs that we find and document aren’t really bugs at all, but rather functionality that has been misinterpreted based upon the requirements definition that comes out of segregated development processes, ie Write Requirements, Develop Software, Test Software and Deploy Software.

Language is such an imprecise way of communicating that it almost virtually guarantees that if you have more than 1 person developing the software for your product you will get different interpretations of how to implement the requirement functionally.

Remember the old ‘The System Shall’ statement? that is commonly used in writing Business Requirements documents?  I always thought that this missed the scope of the requirement, what about what the System Shall Not Do?.  We focus so much on happy path for our functional development that we miss large segments of functionality based upon what an application shouldn’t be allowed to do.

Let’s take an example of an English word to convey what I’m meaning:

What do you think of when you see the word – BASS

Do you think of this:

Bass_Guitar

OR this?

Bass_Fish

These have the same spelling in the English language yet they have two entirely different meanings. Our life experiences become a prism for how we interpret what we read and how we react to it. (PS, I’m a musician so I think of the instrument before the fish)

Teams that rely on written requirements documents that are reviewed and worked on independently, meaning developers develop the code/functionality and then pass it on to testers will inherently have issues/bugs related to how something was interpreted and then developed.  In the above example the developers may have thought they were delivering a bass guitar, but the testers were expecting a bass fish (yeah extreme I know) but I believe this is the root of many of our issues when trying to deliver what the business expected in the first place.

— Bugs as True Bugs – I believe that true bugs are more technical than functional (or should be).  Bugs related to how integration happens are very common because although we can describe the behavior of our feature we can’t always anticipate issues related to how independent systems will work together.  Many ‘true’ bugs in Agile are caught in the moment and fixed before they ever make their way to production.  For the Finance people out there this is a tremendous cost saving that has been proven time and again.  ROI is greatly enhanced when you deal with tech debt up front rather over time.  And please don’t ever think really that it is more important to get ‘something’ to production over making sure that the product is operationally sound.

So what to do?

To address the  inherent limitations with our  communication, we need to abstract our thinking into more concrete descriptions of behavior over broad-based statements such as the System Shall.

User Stories and the corresponding BDD acceptance criteria are a great way to do this as a BDD example table clearly defines the behavior of our product functionality via outcome based upon inputs.  The ‘language’ of BDD is unique so that everyone can begin to have a shared understanding of what the story and behavior of the product(aka system) will do.  BDD abstracts our communication and removes individual interpretation.

In Agile we start by ensuring that the teams understand that they OWN the quality of their delivery. One of the things that I absolutely love about high performing Agile team is that there is no finger-pointing, if a Sprint fails to deliver what the team committed to then everyone shares the blame, not just an individual or functional group.

How we bake quality into our products is by understanding how to write User Stories that provide context without the ability to misinterpret the meaning of the requirements.

To do this we must first ensure that we have a well written user story, what does that look like?

A good user story needs to identify the What and then the Value statement.  Many teams that I have worked with start to write stories that only identify the actor and What but leave out the value statement, which is really the proof that what we are working on is of sufficient value to devote our resources to.  A good user story should never have any Creative or Technical design conveyed, I know that many people like to show their technical knowledge by writing stories that convey what they think the design or system will need to utilize, but all that does is start the team down a path before exploring all options.

Behavior over language interpretation is what you are striving for when writing contextually rich user stories.

BDD with its accompanied Example statements takes an otherwise basic user story and brings it to life.  Much like we do when we take basic ingredients for cookies and then bring them together in the right amounts to deliver awesomeness every time.

Software quality is much like baking, you need:

  • The right ingredients – Good individual team members, honest communication, commitment to quality
  • The right process – Write good user stories, add quality BDD acceptance criteria, code and test in parallel and then deliver, involve the entire team in writing BDD, involve the entire team in the estimation process.

By taking the time to build contextually rich user stories and define the story with BDD acceptance you move towards a shared understanding of the ‘behavior of your product’ over one that is driven by functional requirements.  Requirements tend to convey to little of behavior and focus more on the big win that is being conveyed to Sr. Management regarding what is being delivered.

Agile is a very disciplined delivery process and in order to bake the quality into your product you need to develop efficient processes that keep the User Story/BDD train running smoothly.  If you are entering a sprint and then writing your BDD then you are already behind, you need to develop a process by which teams are working on current sprint development AND building context for the next one.  It can be done and when it is you get what I call progressive regression with the automation that comes out of your BDD work.

Agile is very disciplined and to think that going Agile will make your current life easier, well guess again.  What Agile will do is highlight EVERY current weakness you have in your current product delivery process and then focus your attention on finding ways of improving on them.

For those of you looking for workshops regarding User Story/BDD techniques, please reach out to me at soundagile@gmail.com.

Software Engineering is Not Engineering (More Like Art)

Recently I had an interesting conversation regarding whether or not software developers were engineers or not. The most compelling argument that they could provide for why Software Engineering was truly an Engineering discipline centered around the fact that Universities and Colleges have their Computer Science group in the Engineering department.  That is not sufficient in my mind to by default call Software Engineers well, Engineers.

Why? Consider the following

Engineers who build skyscrapers, airplanes, bridges and products that require low tolerance for defects are often operating under the laws of science and as such require that they account for many scientific elements when building their products.

They perform a great amount of ‘testing’ both in theory and with product before something is released for use.  Their engineering practices are based upon scientific principles that go through extensive peer review and have been building in knowledge for hundreds if not thousands of years.

Now lets look at the definition of Engineering:

  1. The branch of science and technology concerned with the design, building, and use of engines, machines, and structures.
  2. The work done by, or the occupation of, an engineer.
  3. (Business / Professions) the profession of applying scientific principles to the design, construction, and maintenance of engines, cars, machines, etc. (mechanical engineering), buildings, bridges, roads, etc. (civil engineering), electrical machines and communication systems (electrical engineering), chemical plant and machinery (chemical engineering), or aircraft (aeronautical engineering)

Though of this I think can be applied to the science of computers, ie the hardware = Machines I don’t however see much if any of this being applied to Software Development.

A paper in 1996 (William Aspray, Reinhard Keil-Slawik, David L. Parnas) talks about some of the reasons that writing code isn’t associated with Engineering:

Computer science is often characterized as an engineering discipline with the systematic study and development of software as its principal subject matter. Software Engineering, however,although combining both key words, has not become a central discipline in most computer science departments.  In many respects, this discipline embodies the same ideosyncracies that can be observed within computer science as a whole such as:

• Highly innovative and rapidly changing field with no broadly recognised core of
material that every practitioner must know;
• Few results are supported by empirical or comparative studies;
• Work within the field older than 3–4 years is rarely acknowledged or referenced;
• Old problems are given new names and old solutions overlooked;
• Evolution of the discipline is tightly coupled to economic and societal demands;
• There is a need for interdisciplinary work comprising e.g. mathematics,
psychology, business or management science, … ;
• Continuing debate about whether there should be a discipline called software
engineering, and if so, whether this should be treated as another discipline among
the set of traditional engineering disciplines.”

As you read through this I think we can start to see a clear delineation between the disciplines required of ‘Engineering’ and the disciplines that Software Development requires (Note – This is not an article to criticize people who write software, but rather a way for us to better understand how to manage them and how to help them deliver even better software)

Go to almost any software development team and you will find a mix of people who have ended up going down the path of writing software, they can be people who were Computer Science majors all the way over to people with a Liberal Arts degrees.

The languages that allow us to write software today have matured over the years and as such have become easier to learn and work with.   This mix of skills and backgrounds is not bad, but it deviates from what an Engineering team building an airplane might look like.  Full transparency here, I have never worked in an organization that does true Engineering, but I know many people who do, and I suspect that you would not find people who are designing/engineering bridges or airplanes who don’t have a strong educational background in Engineering, but you will find these people in your software development teams.

So why is ‘Software Engineering’ more art than science?

I think that Software Development is more art than science because we must create software that we interact with on a more human basis.  Creativity abounds in web development as we continue to strive to bring intuitive and easy to use interfaces to the people who are our ‘users’.  The people who write software must be able to synthesize multiple layers of code, from Front End to Back end and everything in between. This talent is probably more aligned to writing a symphony in that you have to deal with an entire array of players, each with different instruments (API, Legacy, etc..) and when you get it right you get beautiful music, when you get it wrong you get noise.

When writing software we have the option to try multiple paths to an end, whereas when building a bridge our ability to deviate from proven engineering practices success is much more limiting and catastrophic when we do when we don’t.

One can also argue that the newer languages being created are designed to make writing software more accessible to more and more people who have not be through a computer science programs, we are striving to get our languages to ever more English like syntax.

Consider also that people who write software are not held to specific engineering practices.  Software, as it has evolved, has provided people who ‘develop’ software product with endless ways to implement it the underlying code.  The fact that there are not any set standards that every Software Engineer must use I think clearly shows that software is not a true engineering discipline, but rather a creative one.  Artists embellish, they innovate, the push boundaries into new ways of performing….I think this better describes many of our software developers more than engineering.

The beauty of current software development is that you have the ability to envision new ways of implementing code that delivers functionality and in doing so create new coding processes that others can use.  Software languages provide many different ways to write code to deliver a product functionality, that might be 3 lines of code or it could be 100, it depends on the background and experience of the individual writing the code.

I think when we start thinking about how to manage and support our Product Development teams in our organizations we would benefit from providing them the framework that encourages high quality through innovation, failing fast, and supports both the individual and groups ability to determine their success.  That is not to say that Sr. Leadership doesn’t play a part here, it certainly does, by providing clear vision of what is wanted (think of a symphony conductor, who must bring out greatness while providing their interpretation of a particular piece), an ability to accept failure as part of success and a fearless ability to have confidence that when you provide the other two components people will bring greatness as a by-product.

Artists operate in a very Agile manner, we listen and feed off of each other and in the process discover new ways to approach our art. Listen to jazz music and you see the elements of Agile and creativity evident as the small group (Scrum team) takes a song and through listening to each other can create new music, sounds or approaches to an old song.

As with Artists, Engineers don’t like to be told something can’t be done, that quest to make something happen that has never existed before is what an artist strives for, there is an emotional rush that comes with artistic creation and I think the same is true of software product creation.

Engineers who build skyscrapers, bridges and planes are no less artistic or innovative in their designs however they must take a much more rigorous approach to what they deliver, they simply don’t have the ability to go back and redo it.  If they get it wrong people die, though there is of course software that is also mission critical, but for the vast majority of us who develop software for a living, we are able to take a less rigorous approach when we create our products and we expect in Agile to move to a continuous delivery model so our work is never ‘done’.

Delivery Management vs Project Management

In Agile we have evolved from a ‘project’ oriented mindset to a product one.

That is not to say that we don’t undertake a something that looks like a project to build or enhance our products.  However in Product Management we are more focused on the ongoing nature of how our product will unfold, in shorter durations and without the end in site.

I believe successful Agile Project Managers need to focus on Delivery Management over Project Management.

What is the difference?

  1. Delivery Management – Is the management of work in an iterative fashion that focuses on delivering product that delights our customers.  It is not focused on Time, Scope and Budget, but rather on customer experience, high quality code and low defects. Delivery management focused on long-term value and benefits over planned short-term objectives.
  2. Project Management – is the application of processes, methods, knowledge, skills and experience to achieve the project objectives. A project is a unique, transient endeavour, undertaken to achieve planned objectives, which could be defined in terms of outputs, outcomes or benefits.

Having been a project manager in a waterfall/PMI world before my move to Agile some of the mental challenges I faced included:

  1. Lack of a plan – One of the first things I did as I was moving to Agile was to track the amount of ‘change’ that happened in one of my waterfall projects.  I created a Project Change Request for everything that was not originally identified in the Business Requirements and Technical Design Documents, assumptions that changed, scope, etc… This process drove my team absolutely crazy because as with all waterfall teams, we fear change requests, they are bad news to management.  However what this effort did was provide visibility to the type of change that happens in every single software development project (and these changes were never socialized outside of the team so they had nothing to worry about).
    1. Realization – 
      1. My project plans provided no real ability to know if a project was going to be successful, they looked good but didn’t tell the real story of what was happening with the project team and what they were developing.
      2. Agile provided me with much more visibility to the real issues that a project team faced and that my project plan was really just a place holder for future Sprints.  I only needed to know that the Sprints were planned and then communicate what the team was committing to.  Once I had that information THEN I could hold them accountable.
  2. Story Points over Time bases estimates – I had a really hard time initially wrapping my head around Story points over time and spent many an hour mentally putting the points back into hours.
    1. Realization –
      1. I finally stopped thinking in time once my team started ‘delivering’ consistent points and work for every sprint.  At that point all I ended up needing to know was the story points of work begin committed to.  The point here is as a Project Manager you need to instill good estimation behaviors with your team and hold them accountable, that IS your job.
      2. Velocity is the primary data point you want to focus on, consistent velocity = consistent delivery.  It’s not your job to determine the What for the product but it is your job to ensure that what they commit to does get Delivered (though if you work to be a valued member of the team, you should definitely be able to provide input).
  3. Status Reporting – Ah the bane of our existence as project managers and something that I had to deal with recently, the dreaded status report.  Honestly I hadn’t been a project manager for many years and as a Manager I never needed one with Agile teams.  All I needed was a dashboard (Jira or Rally are the ones I’m most experienced with) to observe a teams backlog, Velocity and Progress towards a Roadmap they were working on.
    1. Realization – The status report will not every go away but I think you need to ensure that your reporting is consistent with the rest of your Agile processes.  Tools such as Jira and Rally provide you with abilities to manage Roadmaps, teams, budgets, etc… Everything is still there but your reporting needs to take advantage of the data elements within Agile, don’t translate your Agile into a waterfall type status report.  This will only ensure that Sr. Management stays disconnected from Agile and the entire organization needs to be plugged into Agile and walk the walk.

If you are a traditional PMI type project manager and are moving to Agile here are some things you should consider:

  1. You are not responsible for the success of a project/ delivery – Yeah I know it sounds wrong, but in Agile it is the TEAM that is responsible for delivery.  You need to plug into your team, get to know what they do, the challenges that they face, become a sounding board for ideas, get your hands dirty, learn how to test, etc…Becoming a valued member of the team is what moved me out of Project Management and into Quality Assurance. 9
  2. Be an Agile advocate – Embrace agile so that you do more than go through the motions. Read, learn, join a meet up group, do more than just the minimum.  As a Project Manager you can help your organization get better at Being Agile, you have the connections and understanding of the organization that many in the technical side may not.
  3. Don’t be Defensive- As you will learn, Retrospectives should be frank and open conversations about what is working and more importantly not working in your processes.  Encourage your team to openly talk about issues, but protect them so that they can continue to work effectively as a team, remember attack the problem not the person.  For example at Disney where I both QA Manager and individual contributor my duties as QA Manager were getting in the way of my ability to test and give good feedback to the team.  They rightly called me out in a retrospective as being the issue for them not moving as fast as they could.  I was not defensive but understanding, that is what you want in Agile, because they were absolutely correct, I was the problem at that point.

So as you look at what you manage in traditional project management, understand that it is not the Project that you are delivering but a Product and Products have much longer life spans than projects.

You need to keep your teams focused in consistent and disciplined delivery that brings real value to your organization. If you are working on something that doesn’t have value you should be questioning and challenging your team as to why.

Agile isn’t easy, though I think it is often thought of as easy.  No Agile is very disciplined and when you undertake Agile it will highlight every inefficiency and poor process that exists in your organization today.  You simply cannot go fast until you address these issues.  As a Project Manager you can help drive this change.