A good post by Chris Hoff at Rational Survivability about the "DevOps Disconnect," discussing some of the incongruities in the most popular characterizations of DevOps. As usual, the discussion in comments is among the most valuable parts of the post.

This echoes many of my own concerns and thoughts about DevOps, which have been raised elsewhere by Andi Mann, and Hoff's entry makes three, which reassures me that those of us with this perspective aren't simply those who proponents claim "don't get it" but instead have a valid point that real supporters might take time to listen to and incorporate rather than simply rejecting it out of hand.

Dennis Stevens posts "Kanban and when will this be done?" talking about how to estimate feature completion in agile environments.

Stevens focuses on the Kanban pull system that has been making inroads in agile development methodologies recently, but the techniques he describes are sufficiently generic to make use of in almost any agile system.

Scheduling in an iterative, short-cycle environment can be a significant challenge for recent adopters, but in fact it's no more inherently difficult (and may actually be easier) than scheduling in more traditional environments. The major difference is in experience. Most CIOs and IT managers have a career of experience estimating traditional waterfall-managed projects and have developed some level of skill scheduling in that environment... not enough, unfortunately, to make up for the other issues befouling waterfall management in IT. Stevens has the answer for picking up that experience in a kanban system, however: measure and analyze!

By measuring cycle time required for various work items and comparing it to the "size" or complexity of the items, Stevens' method lays out an adequate rule of thumb for producing scheduling estimates of similar items. I'll refer you to his original post for specifics.

The most important point he makes, however, is that planning and scheduling should be decoupled. To whatever extent possible, the specific work items should be divorced from delivery dates, at least as far as the team working on them is concerned. This helps to avoid a whole host of negative behaviors that are associated with deadlines. It does not, however, preclude managers from using the estimating methods to produce their own timelines for scheduling purposes.

You're unlikely to ever be lucky enough to work in an environment where planning can be entirely divorced from time. Indeed, it's a vital factor in any cost/benefit analysis to be able to state when a cost is going to result in a benefit. Committing resources to a project that will result in a 50% reduction in costs within two months is a better call than committing them to a project that will result in a 75% reduction in two years. At the management level, you have to be able to make some ballpark estimates that are sufficient for business planning purposes. What you don't have to do, and what Stevens makes a good case against, is to attempt to hold your team to those numbers. Accountability is necessary, but the better way to achieve it in Agile is by putting good, trustworthy people on your team, and getting them to buy into self-enforcing standards of accountability.

The other great coping feature in the agile toolset is the iterative release concept. If you miss an estimate on a feature release, it's not as bad if you are putting out your updates once a week as if you were doing them only quarterly. You may miss, but you won't miss by much.

That was fast! It seems like I still spend ninety percent of my time explaining to people what exactly Agile Operations is, but already the City College of Plymouth (UK) is offering a Certificate of Professional Development in Agile Operations for the subject.

I'm not up to speed with British higher education standards and lingo, but from the course description it sounds like this is primarily intended to be offered to organizations as a continuing ed course for their staff. It's not exactly AgileOps in the IT sense, either; it's more business-oriented. I'm increasingly coming around to the view that business-oriented is a more coherent way of looking at Agile Operations, however, with IT serving as a vital enabling factor, so maybe they are just ahead of the curve on me there.

The course covers Lean Business, Managing Activities, and Effective Communication (if it's not effective, is it actually communicating?) in a flexible course schedule. I'm not sure how long it's been available or who teaches it; I'd be interested in hearing from anyone on that side of the pond with any more information about it, or similar courses.

It's not news that users are frequently dis-satisfied with the delivery of IT services for their job functions. In any company, there always seem to be some people that just hate IT, know how they could do it better, and constantly try to subvert systems that are often there for good reasons they don't easily comprehend in their quest for perfection. Of course there are plenty of legitimate user complaints as well, people who are frustrated with things that certainly could be done otherwise, but aren't for reasons of cost or inertia.

But this historic background level of dissatisfaction may be masking a new wave of frustration with corporate IT, one that is both more soundly rooted and more easily addressed. CIOs who fail to deal with it are missing a dangerous warning sign about both users and the future of the IT industry. So it's worth looking at the impacts of new technologies on work and corporate IT not from the ivory tower, as we are wont to do, but from the perspective of the users, as this Wall Street Journal article attempts to do.

IT consumerization is a factor that is on the radar screen for just about every CIO by now, but a standard and workable response to it has not yet been popularized. Reactions vary from grudging acceptance to outright hostility to studied indifference. Few and far between are organizations that embrace the trend as a component of their corporate strategy; yet that is the level to which companies must integrate this societal sea-change in order to profit from it.
Continue reading "The User Perspective"

I wrote a couple of weeks ago on the role of governance in Agile Operations, but a recent podcast put together by ZDNet's Dana Gardner has kept the concept on my front burner this week. Read excerpts, or download the audio, at Gardner's blog here.

Gardner and his guests, Jeff Papows (President and CEO of WebLayers) and Kerrie Holley (IBM fellow and Chief Technology Officer for IBM’s SOA Center of Excellence) are more interested in discussing SOA and their relationship to governance, but on the whole the conversation is a good fit with the broader conversation about Agile approaches to governance (I've already put up some of my takes on SOA as it relates to AgileOps).

Both Holley and Papows have books coming out later this summer; Holley's, co-authored with Ali Arsanjani, is 100 SOA Questions: Asked and Answered; Papow's is the more broadly themed Glitch: The Hidden Impact of Faulty Software. Since neither of them are out yet, I will warn you that my take on Holley and Papow's arguments may be poorly informed. Nonetheless, while I find myself disagreeing with some of the ways they get there, I couldn't be more on board with their conclusions.

Continue reading "More on Agile IT governance"

I don't talk a lot about specific technologies used in Agile Operations here, mostly because it's an approach that doesn't necessarily require specific technologies (and anyway, the DevOps crowd covers the useful tactical details pretty well) but it's sort of an assumption that Service Oriented Architectures (SOA) will figure prominently into many AgileOps plans.

SOA isn't really a technology, of course, it's a set of principles that happen to be complimentary to the AgileOps tenets and can provide powerful enablers for agility. It's not yet clear that the SOA community has grasped this alignment, or the idea that AgileOps can be as strong a selling point for SOA implementation as the reverse. But Joe McKendrick asks an old question in a new venue when he discusses taking bottom-up or top-down approaches to SOA implementation and as it happens, Agile Operations already has an answer of sorts.

The answer, of course, is that it's a false choice, and something of an anathema to agile principles in general. The reason people have such vigorous debates over bottom-up versus top-down is that there are excellent merits, and terrible faults, in both approaches, and it's easy to pick and choose which to champion based on your preferences (or, more likely, your position at the top or bottom of the organization in question) and discount the other. Agile, though, emphasizes communication, cooperation, and alignment... all things that don't come from either top or bottom, but both.

Oddly enough, McKendrick mentions this "middle-out" approach but then fails to compare it to the others. Instead, he makes points for both top-down and bottom-up:

• Top-Down: Provides governance, business-level SLAs, is active rather than re-active


• Bottom-up: Avoids rigidity, long-term investments, and detached perspectives on tactical implementation issues

From where I'm sitting, these are all good points... and none really contradict the others. There's no reason you can't have the advantages of both without the disadvantages of either. It just requires communication and coordination... things that Agile is pretty strong on.

So, if you're looking at bringing in an SOA in your organization, think hard about moving toward the use of Agile Operations at the same time. Don't force yourself to make the false choice, picking bottom-up or top-down and then finding yourself having to reconcile the mess in eighteen months. Start middle-out in the beginning and make it easier and more effective for everyone.

Government IT gets a bad rap as being slow, outdated, and encumbered with an even more egregious project failure rate than the already horrific ~50% we face in the private sector. A parade of news stories highlighting these failures, or unveiling the antiquated machinery and processes that government workers are forced to make do with even when the systems are working, does nothing to improve the image. Innovative IT thinkers like new US CIO Vivek Kundra and GSA CIO Casey Coleman are getting some good press, but on the whole, the agencies they represent are still seen as mired in the past and resistant to change.

Turns out, though, those stodgy old bureaucrats may be more hip and cutting edge than their private sector counterparts. Agile Operations is gaining traction in federal IT shops with more strength and greater commitment than can be found anywhere in the private sector just yet.

It's entirely possible that all the lean years and antiquated systems are exactly what is causing this role reversal. Agile approaches in IT management are among the only effective answers for improving ROI and implementation time when you don't have buckets of money to throw at projects. With the private sector continuing to ratchet up expectations, public sector CIOs may have had little recourse but to adapt to more agile management techniques to make up the gap.

Though there have been, for the past couple years, and will continue to be financial pressures in private sector IT, the real motivation for private IT organizations probably will come from relevance and effectiveness. There are as many benefits to the agile philosophy for business as for government, but it's only slowly occurring to business and technology leaders what those are... and many, recognizing the possibilities, still don't have a clear understanding of the path leading to them.

Much of the discussion of agile operations occurs at the grass roots level, a result of the emphasis on practicality and the mechanics of getting things done. But agile practices are sufficiently different from what has come before that they have significant impacts at the organizational management level as well. This conflict has proven a headache for both CIOs and for agile proponents in the trenches, neither of whom are able to proceed unfettered toward their goals while the other is not onboard.

Fortunately, when we get to the operational aspects, we can borrow much from the consideration that others have already given the problem from the similar situation agile developers found when first bringing their own model into play in large organizations. Even better, we can draw from much of the original thinking of lean proponents, who were focused largely on management aspects in the first place, and whose techniques and conclusions are often directly applicable in the operations environment.

Establishing the right governance model is as critical as any other factor in implementing agile operations in your organization. It's broadly recognized, if still poorly reflected in actual practice, that management support for almost any significant initiative is a critical component for success. Governance represents the effort to establish that support by interaction between the IT department and business leadership on a functional level (this is opposed to the more theoretical interaction typical between CIOs, CEOs, and CFOs, where much detail is often absent from strategic and operational discussions).

This idea often seems an anathema to dedicated agile proponents, as "governance" brings up thoughts of mindless paperwork, rubberstamping, and endless unproductive meetings that are precisely the reason they prefer agile to waterfall in the first place. But if every crisis represents an opportunity, it is also true that every new governance structure represents a way to get governance right. Bringing agile thoughts and theories to governance can transform that process as radically as it has transformed software engineering.


Continue reading "Agility and governance"

The thought that the CIO role and the state of the IT organization as a stand-alone entity in the modern corporation are due for a dramatic revision soon has not exactly hit the mainstream yet, but it's on the way. Since late 2007, there have been indications that the CIO is losing what little respect and autonomy the role ever had, and the forces leading to that diminution and the corresponding fading of the IT department as a whole in the role of corporate technology provider.

The latest missive in this vein comes from the Corporate Executive Board (CEB) in their whitepaper "The Future of Corporate IT." (PDF) The CEB are projecting a 75% drop in IT headcount in the next five years in many corporations, and with the current paradigm of "big headcount = big honcho" the equation does not favor the head of such a diminished department.

CIOs seem bound to make this worse than it has to be by stalling, digging in their heels on traditional initiatives, and working to preserve their headcount and budget even in the face of solutions that could both improve their efficiency and reduce their expenses at the same time. This may work for a while--I certainly think that the 75% reduction in five year prediction is aggressive considering the corporate infighting capabilities many IT organizations have developed--but when the time comes, it's going to be a harder and more permanent fall from grace than is really necessary.
Continue reading "Agile Operations and the CIO"

One of the ancient and original objections to implementing agile was the predictable "But if we release more frequently, we won't have time to make sure we're not releasing crap!" This is probably a pretty tired argument in the ears of the agile development community, but it's starting to come out again now that agile is moving into the operations sphere.

Joe McKendrick at ZDNet is the latest to register this objection in his post "Opinion: SOA and Agile may short-change application quality." In it, he notes that "One of the concerns about automation is that in many cases, it just makes a crappy business process a much faster crappy business process" and wonders if service-oriented architectures that are pushed out with agile methods are putting the business at risk by exposing it to poorly designed processes. The solution, Joe says, is to make sure that "Application quality needs to be addressed as early in the process as possible."

This is a fine sentiment, but of course it's essentially describing the paradigm that agile is largely a reaction to: the waterfall model. Plan everything out, in detail, implement slowly and completely, deploy only after you are sure you have all the kinks worked out. All of which are great ideas... in theory. In practice, in information technology, they turn out to not always work so well, for a variety of reasons that have been well-explored elsewhere.

While the worries that McKendrick has are very real, there are also a host of concepts in the agile philosophy designed to address them. Increased communication, rigorous testing and automation designed prior to coding, minimalist and modular changes that don't necessarily implicate other parts of the system even if they do contain errors... these factors and others are specifically designed to address the risks of going faster.

The unspoken assumption remains, however, that slower and more methodical is less risky. If we can acknowledge that faster releases carry a certain risk, we should also be honest enough to acknowledge that slower has never really proven to carry any less risk. Feel free to consult the IT project failure rate study of your choosing at this point. In fact, waterfall has always carried a tremendous risk, and it's a risk that agile addresses directly precisely because it is faster: the sunk cost problem.

More methodical waterfall approaches have not proven any better at detecting and eliminating process errors of the sort that Joe fears. What they have done, however, is ensured that those process errors in many cases have become deeply and expensively embedded in systems before being detected. The cost of resolving such failures is enormous. While agile methodologies may not do any better at finding those problems ahead of deployment, what they can guarantee is that you will find out about them in production much sooner, and have an opportunity to address them more quickly and at less cost.

There is a sort of circular logic to these objections that is insidious and will likely continue to bubble up for years. Agile is itself a rebuttal to the sort of cautionary approach that Joe proposes, but it's clear that the point has not penetrated past the sort of intuitive, but incorrect, surface logic that obviously going slowly and planning ahead will lead to better results than moving quickly and not planning in detail. That's such an obvious position that it's difficult to even force people to justify it. But forcing them to confront the numbers and the complexity of technology may be the only way to move past this debate.

It's a necessary rite of passage for new ideas in technology to pass through the Big Three analyst Gauntlet of Fire, and with respect to agile operations, Gartner seems to be faster on the uptake than Forrester or IDC. Novel approaches to provisioning and support don't seem to have phased them, and now at least one analyst has taken up the mantle of IT infrastructure and operations reform in the guise of agile operations. The question is, is that mantle the same one existing proponents are trying to hoist?

Cameron Haight posts on The Need for IT I&O "Reform" on his Gartner blog, peppering the call with provocative statements like:

At the end of the day, I believe that it is no longer tenable to simply layer an increasingly complex operations management tool, organization and process infrastructure on top of an already complex software and hardware foundation and to expect satisfying results.

And:

I like the agile construct though because it is focused first and foremost on people (i.e., individuals and interactions over processes and tools) who at the end of the day are the most valuable components within the IT service delivery system.

So far, AO seems to be holding up fairly well in the analyst's estimation, then.

What I find interesting in Haight's take is the emphasis on the aspects of agile that push decision making down to "the coal face" as another commentator put it. It strikes me that many of the contentious discussions that have occurred lately over the applicability of frameworks such as ITIL in the context of agile operations ultimately boil down to this central tension, a tension that I have largely ignored since it seems to me to comprise the essential "magic" in implementing agile operations. Articles can be written (and are) all day long about what tools you might use and what techniques have been beneficial, but ultimately someone in the organization has to be fundamentally good at deciding where people can freelance and where stricter controls are required. There is no real way to describe that process; it's different in every organization, as staff and the confluence of requirements are unique. This is, ultimately, what it's all about, and you're not going to get it all out of a book or analyst's report; experience and internal knowledge is where it has to come from.

Haight acknowledges as much, saying:

I’m not saying to do away with operations processes in a wholesale fashion, but rather be more selective in which ones will provide the highest value and focus on optimizing these (significantly fewer) disciplines.

But this acknowledgment hasn't prevented a strong reaction from Bernard Goldin in Network World, who points out that there are solid, time-tested reasons that controls of the sort implied by ITIL have evolved. Not all IT staff are capable of reacting appropriately in all situations absent some standardized framework; many of those that are cannot always be counted on to also do so in a manner capable of being followed or replicated by co-workers or successors.

At first blush, I might seem to favor Haight; I have always advocated a pick-and-choose approach to ITIL, only adopting such tenets as make sense in the adopting environment. There are times and places for heavy process, just as there are places where process only serves to waste resources. But, interestingly, I find that I side more with Goldin when it comes to internal IT operations.

Higher costs come from non-standardized operations. Toyota, grandfather of the lean principles fueling our discussions of agile in the IT world, preaches flexibility on the assembly line but certainly doesn't encourage every worker to pick a different size bolt for the engine mounts. Standardization at some level is a perfectly acceptable part of agile approaches. The idea that "agile" has to mean "individualized" and "anti-methodology" is just plain wrong. The Agile Manifesto says that tools and processes are less important than individuals and interactions; it doesn't say that they are unimportant, as some of the commenters in the ITIL/DevOps thread (a minority view, however) seem to want it to say.

This may be a position influenced strongly by my own experience working with IT operations in small organizations where far too much freelancing has left service delivery in a precarious place. Most of my own conceptions of pushing decision authority down the hill have actually revolved around pushing it outside the IT department entirely. After all, when you are talking about the coal face where work is actually happening to meet customer needs and drive sales, in most companies, that's still happening far from the IT department. I see more damage being done by IT staff, at all levels, who are making decisions that might be better resolved (in cooperation, of course) with actual business staff.

So is this debate simply about deck chairs on the Titanic? If so, will it sideline much of the good an agile approach can have in connecting the business with the right technology? Are we missing the real point of agile principles when we restrict our considerations to what is happening strictly inside the IT department? I rather think so. Of course, "DevOps" as both Goldin and Haight have primarily characterized it, lends itself to that interpretation... another reason to avoid using the term in conversation with business executives outside the IT department. But if our discussions stop at the IT doorstep, if we're trying to decide whether or not ITIL or any element of process makes sense simply based on the character of the IT department itself, we're simply repeating a rather banal and terrible error that has lead IT into the position it already occupies, vilified and facing drastic cuts in both resources and authority as technology spending moves outside the department and CIOs are increasingly marginalized in the executive suite. Did business/IT alignment get turned into a buzzword so quickly that everyone forgot why it came up in the first place? Anyone who is trying to implement agile operations in their IT department without involving the rest of the business in the discussion over how much or how little process is necessary is making a dramatic mistake.

Cloud computing and agile IT are often bandied about in the same conversations, but to what extent are the advantages of cloud computing inherent or beneficial to agile IT operations?

Christina Torode wonders "Does IT scalability translate into business agility?" over at TechTarget and concludes that "something is missing" between the two.

Of course there is something missing; a tool is not a methodology and although you should try to select the right tools for the job, simply selecting the right tools doesn't inevitably lead to the success of the project. Indeed, tools are rarely the most important factor, and different tools may be the "right" ones given the same objective but differing teams.

On the other hand, good tools can make adherence to a set of beneficial practices much easier than without them. Cloud computing, and more specifically, virtualization in general, represent killer apps for agile administrators and CIOs.

Best practices for operations have always involved developing a certain degree of portability to systems. Hardware being eminently fallible and migrations for scaling purposes all-too-common, images became a popular solution for rapidly and systematically replacing or deploying new systems in a way that allowed thorough testing and predictable behavior. Increasingly advanced imaging and scripting tools made deployment and configuration a matter of routine in well-run IT departments.

But the messy bit, the hardware itself, was still a prominent factor in the equation. In the early days of automobiles, most drivers could not simply own a car and drive to their destination unhampered by mechanical difficulties as we do today; every driver had to be a mechanic first, the degree of reliability of the car being constantly suspect, and there existing little or no infrastructure in the form of wreckers, gas stations, and repair shops to turn to with problems. In short, it was messy. Cloud computing does not remove that messiness, but it does abstract it by turning it all into somebody else's problem, just as has happened with cars. We don't all have to be mechanics now, we can just get in the vehicle and go.

On a purely technical basis, it's interesting to look at the circular nature of agile ops and cloud computing. James Urquhart at Cnet has a three part series on cloud computing and 'devops' which is as much about how agile methodologies are enabling cloud service providers as how cloud providers enable agile for the rest of us.

On a functional basis, however, cloud-based services and applications are a shortcut to most of what operations teams have been trying to achieve for years with hard-won redundancy and virtualization features. The difficulties of running the infrastructure are dealt with by specialists, and the primary challenge for the operations crew becomes allocating and monitoring the services themselves. This puts some much-needed focus on the service deliverables without the distraction of the hardware level... it allows the ops team to stop playing at being mechanics and instead just get in the car and go.

Just as with your favorite repair shop, you'll want to do due diligence on your cloud computing provider... it's not that the hardware becomes unimportant, after all, it's simply abstracted and someone still has to know how to work with what is under the hood. But past that, an ongoing understanding of how the services are provided isn't necessary, at least by most of the team. They can instead focus on services, provisioning, and reliability at the functional delivery stage. More attention there will not go amiss these days when the utility of IT departments everywhere is being called into question and most are being asked to provide the same services with fewer staff and resources.

At the end of the day, cloud computing is just another level of virtualization, so if you have already done the hard work of convincing corporate authorities that virtualization is a winner (which it is) then you don't have far to go with cloud-based services. The costs in most instances are even more attractive than with in-house virtualization, and security issues, while real, are likely to be less a concern than the computing press has so far made them out to be. The scalability of external virtualization will far exceed what most businesses can accomplish with in-house resources... and the agility the solution offers makes it the right tool for IT organizations moving toward Agile Operations.

In case you were interested, Andi Mann seems to have stumbled into the same general problem that I blundered through last month with respect to the applications of "DevOps" outside developers interacting with operations staff.

I don't have anything further to say on the subject beyond what I came up with in my summation post, so I will leave it at that and hope that Andi comes to similar conclusions about the utility of the philosophies behind DevOps, if not the terms or practices themselves.

Just a quick link to a recent ComputerWorld article on how Toyota, the birthplace of the lean manufacturing movement that has in turn fed into the agile development and agile operations movements, itself handles information technology. There has been relatively little published about how the company fits IT into the Toyota Way; the eponymous book on the process says of technology only that the company prefers to adopt and stick with proven and tested solutions, a reasonable enough stance from an operations perspective, but surprisingly tame from the constant improvement perspective.

From the article, it's possible to extrapolate the the organization itself has had difficulty adapting the Toyota Way to IT until recently; it sounds as if, at least in the plant described in the article, real progress on that front wasn't made until after 2007, despite the process having been adopted company-wide in 2001 (and practiced less formally for two decades or more prior to that). So, if you're having trouble working your own IT organizations toward an agile stance, don't feel too bad.

Of course, the timing of the adoption in particular of agile development methods just as the Prius was coming into its own and the subsequent accusations of faulty software in the braking systems may prove eventually to be a black mark associated with the concept. But the inappropriate application of a methodology isn't a curse on the methodology itself, if in fact it was even involved in the development of that particular software (I'm not sure I would choose an iterative development model for that application; if there were ever a case for hardcore waterfall development, life-safety code might be it), or if the software is even actually at fault in those accidents.

The article is extremely light on detail, so if anyone has run across on other resources documenting the Toyota Way as it's applied to the company's IT department, I'd love to hear about it. Just because Toyota came up with the idea doesn't make their adoption definitive, of course, but it's always helpful to see how the originator realizes a concept.

Fragile Support is the practice of reducing the internal support function of the business in favor of simplified or externalized resources.

The basis for fragile support rests on two premises: One, it demands that internal systems be designed to function as robustly and intuitively as possible, so that they will never require much support; Two, it emphasizes pushing the support function back outside the business or onto such resources internally as can be maintained without any specific support role designated.

The key to fulfilling both those premises is not in some wondrous technology or managerial expertise, but rather in providing the appropriate incentives to both users and IT staff.


Continue reading "Fragile Support"