Agile Metrics

One of the hardest things that we struggle with in Agile is with is how to report how we are doing with respect to our projects.  Agile or otherwise we still primarily think of our software development in a project orientation.

All of our historical metrics from our waterfall days talk about headcount, resourcing, man hours (days), project milestones, etc…..

In Agile we don’t produce the type of metrics that management has historically used as progress tools.

You hear quite often that in Agile velocity is king (kind of like cash) and in a real sense that is true for me.

Velocity represents the amount of value that a single Scrum team can deliver.  We know that each Scrum team has a specific set fixed cost and the value that they ‘consistently’ deliver reflects a key metric for management to key on.

  • Velocity – What can you glean from a single number?
    • Effective Estimation – A Scrum team that doesn’t deliver accurate or confident estimates they will see their velocity roller coaster from sprint to sprint.
      • What to look for? –
        • If a team consistently has to move a story to the next sprint because it couldn’t be completed that is an indicator for estimation improvement.  This has the effect of reducing the points for that sprint as they can’t accept an incomplete story.
        • If a team consistently has large stories entering their sprint, ie larger than 8 points, this is an indication that they haven’t decomposed their stories enough.  Lack of story breakdown results in more unknowns leading to inaccurate sprint commitments.
  • Planned vs Actual – Another flavor of identifying estimation issues.
    • What to track? – Once a team develops a consistent velocity (usually takes 3-5 sprints) then that is what we should expect each iteration, providing people aren’t on vacation, etc…A team that commits to varying levels of story points each sprint typically hasn’t performed enough story development or they lack a well-groomed backlog.
    • Burndown – I specifically like this as it reflects the above issues very nicely.
      • What to look for? – Pretty simple really.  A team that has performed effective story decomposition, planning and estimation should expect to see their work being delivered and completed throughout the entire sprint.
      • What to look for? – Teams who have a cliff dive burndown where all of the story points in development hit QA at the end of the sprint.  (Solution? – QA needs to set up a working agreement so that they will only test ‘x’ number of points in any given day.  If the team delivers 12 points on Friday and the agreement is 8 points for testing, then 4 story points will not get accepted, regardless if ‘development is done’ (cold uh?).
    • Sprint changes – Though harder to track with most of the tools out there (Rally has an app but I’m not sure it provides information that is easily derived regarding which stories actually left the sprint and which ones came in).  Regardless of how you track it, this one metric will tell you a lot about an organization and it’s overall planning effectiveness.  Remember – Agile accepts that change is inevitable, but in the real world you can’t have unmanaged change, teams can’t work effectively or productively in that manner.
    • Sprint Plan vs Actual – As teams improve in their planning and estimation skills, they should also get better at providing accurate estimates as to the number of Sprints it will take to complete a feature (aka project).  One of the things I had a VP say to me recently is that he wasn’t sure Agile was the way because at the end of the day he still had to make commitments to Sr. Mgt.  The notion that Agile can’t provide plans that have accuracy is wrong, however you  but need to focus on Discovery, Planning and Estimation efforts.  And we need to understand that there are places and times where we need to take time to do this work.  Thinking that we should always just be coding is a sure way to build something, but probably not the right something.

Succeeding to Fail in Agile

In an Agile environment everyone needs to understand that ‘being Agile’ is about taking the framework of tools, processes and methodologies and applying them to your specific organizational needs.

At Disney where I was part of a group that moved into Agile, our first foray into Agile Discovery and Planning was, well an epic fail. We we didn’t produce any work product that would allow us to enter a Sprint.  We developed a lot of ‘stuff’ but as it was at is it turned out, not the ‘right’ stuff.

But from that initial failure we evaluated what did work (group Discovery) and what didn’t (PO not taking ownership of story development).  From there we were able to begin to understand the ‘what’ that was important and over the course of around six months we developed a pretty solid delivery model that allowed the organization to feel confident in our delivery abilities.

Now, what worked for us at Disney more than likely won’t work for your organization.  The only way to find out is to try something, anything and see how it goes.

If you want to be a virtuosos at any musical instrument you need to practice, hour upon hour of repetition.  Along this long road of preparation you will find some efforts easy, others hard.  You will fail at mastering some part of a technique that just seems so difficult, you feel you will never get past that point and then you will try something else and BAM, you got it.

From our failures comes learning, the type of learning that becomes part of your DNA. Your mind takes over and can put thinking in the background as you play your instrument.

To be great in Agile, your organization has to change its very DNA.  If you want to succeed, you have to fail and you have to fail often and quickly.

You here that a lot in Agile when we are talking about our test automation, let it fail early and often. We know that fixing issues while still in development is significantly cheaper than fixing them in Production.

Make sure you instill in your organization, management support for succeeding to fail.

Baseball players get into the Hall of Fame for succeeding to fail at a 70% clip, if you bat over .300 for your career you have a good chance of making it to Cooperstown.

As your teams embark on their Agile journey, keep these concepts in mind to ensure you are driving towards success:

  • Self-organization – We want this to happen and for teams to be empowered, because they know more than anyone, what the real pain points are to their daily work.  Let them try something to see if it works no matter how crazy you may think it sounds.
  • Retrospectives – Make sure that a team that is self organized and is empowered to try new approaches is having regular post Sprint retrospectives.  This continuous inspect and adapt cycle ensures that we are always evaluating what works and what doesn’t.

In the end if things aren’t working – TRY SOMETHING. What do you have to lose?

Investment Portfolio Management Applied to Product Management

I’ve been telling myself that I needed to attempt to apply formal investment portfolio management techniques to how we value, prioritize and manage our portfolio of product development efforts, so here goes (definitely still a work in progress)
Back in 1950′s Dr. Harry Markowitz created an investment model called the ‘Efficient Portfolio’.  Markowitz stipulated with his theory that an “Efficient Portfolio is one where no added diversification can lower the portfolio’s risk for a given return expectation (alternately, no additional expected return can be gained without increasing the risk of the portfolio)”.
The Markowitz Efficient Frontier is the set of all portfolios that will give the highest expected return for each given level of risk.
 
This model set the framework for how current money managers build a portfolio of securities that provide range of investment returns to meet each investors risk profile.  Providing investors with a broad range of risk/return choices allows individual investors to build an investment portfolio that meets their specific risk threshold with respect to a given rate of return.
An efficient portfolio looks like this:
 Efficient Frontier
When you are younger and have time to take chances your risk/return profile might be higher on the frontier, whereas at retirement you will slide down that scale as you are more interested in protecting your total investment.  One thing to note is that if your risk/return data falls above the efficient frontier, then you are accepting a level of risk that is not in line with the rate of return you might receive.  Pushing past the efficient frontier can open you up to unexpectedly high returns but conversely you can also expect very high negative returns due to the risk you are taking on.
Product Development organizations can utilize the Frontier as well, for example, a young startup will have a much higher appetite for risk as they understand that to take market share from competitors they need to take risks with speed to market.  However there are specific elements of risk that need to be considered as you speed your product to market.  Ensuring that Usability has been considered, Prototypes have been developed, code quality is considered and test automation all need to play part when you are building your risk/return Efficient Frontier for your Product Portfolio.
If we were to apply the Efficient Frontier to how we manage our Software Development investments we could build a risk/return profile that is easy to understand and align with the organizations risk/return profile.   There are many software projects that at inception are known to be risky, however a lack of empirical data often means that the projects will get the green light and then fail miserably.  The organizations inability to accurately asses risk/return at any time with their software development investments is a huge blind spot and keeps us from consistently delivering the value that the organization needs to stay competitive.
Agile addresses the value (return) part of what the Efficient Frontier speaks to however it talks nothing of Risk overtly.  Risk is more implied with the notion that we manage it by delivering in short increments and focus on shipping value consistently.  However Risk is more quantifiable as I mentioned earlier.
Building an Efficient Frontier in the investment world is a data intensive effort, which our current product/software development processes doesn’t easily support.  However I believe that we can use the formula that Markowitz created to generate an Efficient Frontier for Product and Software Development organizations.
For this effort we will make some assumptions with respect to the Frontier model and changes to Markowitz’s formula so that it works with our limited data set:
  1. Portfolio = Product Development
    1. An organization can have several Products in their Portfolio -
      1. Consumer Facing
      2. Internal Facing
      3. Infrastructure
      4. Research and Development
  2. A security is equivalent to a Scrum Team.
    1. These would equate to the individual securities that Markowitz speaks to in his model.  Where an investment portfolio consists of many securities, each with their own risk/return profiles so to does an organizations product development portfolio consist of the same.  Each team is a security that can on its own provide return that comes with an associated risk.
    2. Though we don’t think of investment securities as having dependencies (as software development teams have) in fact a diversified investment portfolio consists of a range of investments that will perform a certain way based upon the dependency that business has to the market that they operate in, so in this case the notion of a portfolio still holds as a viable means to build a Product Development Efficient Frontier.
  3. Potential Risk Parameters:
    1. Development Lifecycle – Waterfall, Agile, RUP, XP, Blended (use at macro level). You could equate this potentially to Bonds, Stocks or other investment instruments.
    2. Experience of Team
    3. Number of Scrum Teams
    4. % Test Automation
    5. Code Complexity
    6. Speed to Market
    7. Roadmap volatility

In my next post I’ll provide some supporting ways we can ‘build’ an efficient frontier for Agile Product Portfolio analysis that both Product and Program Managers can utilize to assess priorities for the entire organizational backlog.

Monetizing Agile Projects

Coming from a finance background my education and experience is grounded in ensuring that money invested returns ROI.  We think of things such as ROI, IRR Cash Flow, etc…

For example when a manufacturing company decides to make a hardware purchase for machinery (any kind) to produce some type of ‘goods/product’ they perform financial analysis regarding the cost of the equipment, the rate of return that it will generate, how quickly they can amortize it and ultimately what is the net profit that the machine is expected to generate over its anticipated lifespan.

For most of my career we start on projects that have a goal in mind, potentially new revenue or cost savings, some have gone so far as to try to determine the ROI, but in general I haven’t seen the type of due diligence that manufacturing type companies perform, applied universally to software development.

This may be an underlying cause of many of the software development projects never seeing the light of day or failing to deliver the expected outcome because we never performed effective financial plans that would establish the scope and speed necessary to deliver a software product so that it warrants the investment.

In Agile I think we have ways of providing some level of financial analysis that can provide us with an understanding if an idea is worth pursuing.

  1. Using the following as a guideline we can begin to estimate costs:
    1. Team Size – 5
    2. Blended Rate – $125 an hour
    3. Hourly team rate – $625
    4. Cost of Sprint(2 weeks) – $50,000
  2. Let’s assume that the feature that we want the team to work on has come back with an estimate of 5 Sprints.
    1. Estimated Development Cost – $250,000
  3. Let’s now assume that this investment is expected to yield an additional $1,000,000 in annual revenue.
  4. Expected ROI in the first year – 300%

Before we engage our teams we need to be sure that the $250k investment will return an appropriate level of return.   In this simple example its a no brainer.

But our world of software development isn’t always so clear-cut, we often don’t know what the expected outcome will be until we release the product into the wild.  There is often additional cost for refactoring before the product hits the mark with your consumers.

But using some of these simple ways of working through anticipated costs we can easily modify the example to reflect additional Sprints for refactoring once the product is released.

In this example let’s assume that the team requires an additional 8 sprints to move from MVP to final product.

The final cost of the product would climb to $625k and our return would drop significantly to 54%.  Still not bad but not the eye-popping number we initially thought it would be.

Factor in sustainment costs into this over the life of the product and you begin to see that your investments in your software absolutely need to go through the same level of analysis as other types of large infrastructure purchases that non-software organizations go through.

I’m currently working on creating a lightweight model based upon Markowitz’s Efficient Frontier investment model that money managers use today to ensure that your risk and return threshold is aligned.  I’ll be posting my initial thoughts and approach in a coming blog.

Agile Planning and Delivery

Agile is often viewed as the way that small organizations and teams can build product quickly and in fact I believe that to be true.

Being quick and nimble allows smaller/growing organizations the ability to get into a market quickly and deliver features that larger organizations haven’t delivered yet or haven’t thought of.

This very ability to deliver new features to a market that desires them is the very reason that larger organizations need to change the way that they deliver their product.  As an organization becomes larger, the entropy that comes from that growth causes the organization to stop being a market leader and start being a market follower.

In the technological world we live in, you can go from market leader to out of business in just a few short years.  I read an article recently that suggests that Walmart is starting to show signs that its dominance in retail may be coming to an end.  I wouldn’t be surprised, as I think they have lost sight of the fact that low prices don’t always solve the things we as consumers value.  Walmart isn’t a market leader anymore, they may just not have come to that realization.

It’s this lag in reality when big companies start falling behind  Somewhere along the way almost every organization will lose its way, lose site of the very thing that caused them to be great in the first place.

When you look around in the Agile learning space you see many of the original creators of Agile and Scrum trying to figure out how to help large organizations get past their own entropy.  I don’t think that Agile can help, it can highlight where your planning and delivery processes are inefficient but it doesn’t provide the language that senior management needs to hear in order to effect real change.

If we want to help organizations transform and be ‘agile’ then we need to speak in the langauge of management, which is money.  I’ve yet to have a cogent conversation with management regarding how their lack of focus costs them money and more importantly fails to deliver the speed to market that they need in order to stay market leaders.

If you think that Agile provides you with a means to quickly change product direction in order to play defense or catchup when new features that you have been planning on are released by a competitor, then you are missing the point.  Agile does accept that change will happen, but we expect that the change will take the form of measured change in how we can make the product more valuable, not wholesale changes in direction.

Changing direction within a Sprint is something I’ve seen several times and it’s almost always driven by management.  Whether you have too many idea’s/opportunities or are just trying to stay relevant give you team time to work on one thing at a time.  Context switching reduces productivity and that you can translate into money.

The example I’ve been using recently is this:

If you managed your Sprints the way you manage your money you would go broke.  Take this example for consideration (note this is what good Agile teams should be forcing into the planning conversation).

Consider a team that has planned 20 points for the current sprint and someone (Product Owner, Sr. Management) has decided that something else is more important, like completing a feature for a key client.  The appropriate response from your Agile team is ‘Great, let the team review the request and provide an estimate’.  Right there you have probably lost 2-3 points in analysis and estimation by the team.  Once they have completed their estimation, they come back to you and tell you that it’s a 5 point effort.

You think to yourself great we can do that in this sprint,  let’s do it.  A good Agile team will tell you yes we can do it but you need to remove 2-3 points for the estimation effort AND 5 points for the new work that you want us to take on in the current sprint.  So you have LOST 7-8 points of productivityIaka value) to get 5 points of value.

If you invested 8 dollars into a business and only were able to get 5 dollars back you would quickly realize that you can’t sustain that monetarily?

Put this to your executives next time you have a priority conversation.  I didn’t even bring in the  technical debt we typically take on to hack something together and the lack of testing that will happen because we weren’t ready to take on the work (read no automation).

As a Finance guy I would say that this is a bad investment.  Yes I want to satisfy my customer but if they are a good customer you should be able to have an ability to manage their expectations so that the team can take on the work in the next Sprint.  Depending on when the request came in they may only have to wait 2 weeks.

Effective teams I’ve worked with use this process as a first line of defense against changing priorities within a Sprint.

Scaling Governance in Agile

Large organizations who move towards and Agile delivery model may struggle with how to scale any governance model.

I think if you polled most Agilists about governance in general, you would tend to get a shutter and an ugly stare.  Indeed we want to address technical excellence during our sprints but what we often fail to understand is that software engineers are not omniscient when it comes to all of the areas of technical acumen related to performance, security and other compliance type efforts.
As an organization grows and becomes more and more global it finds itself dealing with many different governing bodies that each have different requirements for how we deal with data protection, which in turn relates to how we develop our software.
Having run PCI compliance for a large organization I know first hand that providing specific information to developers with respect to what to look for just in a code review is an important first step.
Governance is needed for most organizations as a first defense when something happens with respect to security/privacy.  Organizations need processes in place to provide evidence to governing bodies, courts and legal inquires regarding how we protect our product our customers information.
Governance can include both standards and artifacts.  For example PCI compliance requires that we provide evidence of compliance with respect to their 13 security requirements.   From a software development standpoint these include such activities as code reviews that are focused on security, software development cycle, evidence of secure development and test practices in addition other efforts such as  Penetration testing and other monitoring capabilities.
In Agile we need to think of this in a light weight manner.  At one organization we implemented a set of standardized stories, non-functional, that were to be included into a teams feature development sprints.  For teams that are working on several sprint efforts before going into production this can work very well as the team is addressing this as part of the planning and estimation.  When the code is delivered to production the completed and accepted user stories (which have context as to what was completed) serve as the artifact for the PCI compliance (or other) auditors.
The engineers liked this because they were creating artifacts as they developed and from an organization standpoint we gained confidence in the fact that our code was being developed against high quality standards.
For Governance activities that relate to more ongoing product sustainment efforts the same standards need to apply however the manner in which we apply them within each sprint may be different that new product type efforts.  Automated testing and monitoring form a larger basis of maintaining the standard levels that we established in earlier cycles.
Recently I heard a statistic that said 90% of attacks can be mitigated by the basic block and tackle work that teams can do on a regular basis.
Governance doesn’t have to be a bad name in the Agile space, we need to view it as part of our technical excellence efforts that form the basis of high performing Agile teams.
I think Governance is an area in Agile that is ripe for development especially at scale.

BDD – Step by Step Process Tutorial

As you might tell from some of my blog topics I’m a huge believer in BDD, not just as a means to automation but more importantly as a process that helps us define requirements more accurately.

When we move into an Agile delivery model, many individuals struggle to understand how they are supposed to document everything that they used to put in their Product Requirements documents (or a whole host of other acronyms).  These documents formed the basis of our engagement with the business and with the teams that would ultimately deliver what the business thought they were asking for in the PRD.

There is an interesting game that has many flavors such as Social Telephone game, that for me, makes clear the issue with large documents that multiple groups need to review.  Each of us is individually focused on the things that we think are important and we often have different interpretations of the written word.  And as such as we work to deliver in our silos we move towards an uncommon understanding of what is being built.  I would argue that many ‘defects’ that are found are actually poorly interpreted requirements.

For those of you in highly regulated industries you know what I’m talking about.  The endless hours spent reading a single paragraph trying to derive the real intent and meaning behind the written word is exhausting and fraught with peril because if we get it wrong there are penalties and mad scrambles to implement a hack to make it work the ‘right’ way.

Software development is made hard because of these communication differences, be they cultural, language or other, they exist and they make it difficult to come to a collective understanding of what is being asked for.

User stories help break down this barrier as they define small pieces of a larger whole and place a value statement so that we understand WHY what we are being asked to build is important.  Many teams writing user stories leave off that important ‘so that’ statement.  Leaving off ‘so that’ leads to so what?

Behavior Driven Development, for me, changed the way that I looked at defining requirements because it removed the business language of things such as the business requires this, or the feature shall do that.  The statements sound powerful yet deliver little in real context for the teams to work from.

Breaking down user stories with BDD still leaves the team to work with a non-technical domain language and then provides a clear method of defining the behavior (positive and negative) for the teams to work from.

The power of context with the respective speed and quality it delivers can’t be denied.  However don’t be fooled that this is easy, it isn’t.  Agile is extremely disciplined, something many miss on their way to the Agile party.  Done right it can take your delivery processes to new heights, done wrong it becomes just another new fangled process that doesn’t work.

Below is a BDD tutorial that can get you started on your way to learning how to build contextually rich user stories.  Good luck and have fun!

BDD Training