Case Study – Wisconsin’s $100 Million Loss

Note: The promised piece on ASP and SaaS availability standards has been postponed for one week.


It has been said that IT projects are always late and over budget. The most commonly given reasons are:

• Vendor errors;
• Vendor misconduct;
• Changed circumstances;
• Unforeseen obstacles.

Regarding “vendor misconduct,” it is easy to find IT executives willing to believe that vendors deliberately under estimate cost and time requirements to win contracts, only to have the true cost become apparent as the project unfolds. Such “low balling” undoubtedly occurs, but more often vendor and customer must share responsibility for a troubled project. If “blame” is to be assigned, perhaps it should fall upon customers who ignore the ancient maxim: “caveat emptor.”

Remarkable examples of “how projects go wrong” are provided by the State of Wisconsin, which in late 2006 and early 2007 confronted a number of troubled projects, with an exposure of some $100 million. The figure is all the more significant when one considers that the State’s annual IT budget is roughly $400 million. A quick survey of the troubled projects will help set the stage:

• In 2006 the University of Wisconsin discontinued a program to update its salary and benefits system, after spending five years and $26 million.
• As of Spring, 2007, a new system to track unemployment taxes was 3.5 years late and $12.2 million over budget.
• A new voter registration system was more than two years late (i.e. ready two years after the deadline set by Federal law).
  
• New sales and use tax software required a $5.7 million fix.
• Work on a new system to track unemployment claims was suspended in February, 2007, after an expenditure of $23.6 million.
• A new system to handle automobile registrations was abandoned. It was at least a year late, $9.4 million over budget and had significantly increased wait times at the DMV offices.

This string of difficulties prompted members of the State legislature to request a formal audit by the Legislative Audit Bureau (Wisconsin’s equivalent to the Federal General Accounting Office). According to the Bureau, these projects had much in common:

• Failure to appreciate the complexity of the projects;
• Failure to define the final function of the various systems;
• Failure to observe contracting best practices, such as standard procedures and incentive/penalty clauses.
• Lack of leadership and oversight.

Significantly, ALL of these variables were within the control of the State.

As a case study, Wisconsin’s difficulties are rich in lessons, for it appears the State made virtually every mistake in the proverbial book. It made many of them repeatedly:

• Failure to realistically allocate resources;
• Failure to plan in detail;
• Failure to plan with an end in mind.

Of these, perhaps the most damaging was the failure to plan with an end in mind. Without a clear goal in mind, it is not possible to set meaningful specifications. In turn, an absence of meaningful specifications opens the door to “mission creep” as each user tacks on items from his or her wish list. Cost and time requirements can and probably will increase dramatically. Even without mission creep, ambiguous specs are a recipe for failure. Vendor may believe that customer wants and expects “X,” while customer is expecting something quite different. Unfortunately, the misunderstanding might not be discovered until late in the project, after much time and cash has been invested. Also, without clear specs, it is not possible to develop reliable cost estimates or time tables. In a very real sense, the various Wisconsin projects were not over due or over budget, for they had never been properly planned or budgeted. Quite simply, the State did not know what it was getting into, and paid the price.

Taken as a whole, the Wisconsin projects suggest an IT unit in disarray:

• Lack of necessary expertise;
• Lack of leadership willing to set reasonable expectations;
• Lack of uniform contract forms and contracting procedures to guide personnel who might not be versed contractual matters;
• Lack of oversight.

The lack of oversight bears special consideration. The various projects discussed here fell primarily within the responsibility of the Department of Administration (aptly known as “DOA”).  That department failed in its responsibilities because it was pre-occupied with two troubled tech projects of its own. A second level of supervision is, in theory, provided by committees of the State legislature. Those committees, however, have been inactive since at least 2005.

Wisconsin’s misadventures offer a sobering example of the many ways IT projects may go wrong:

• Failed leadership, leading to unreasonable commitments. These commitments led, in turn, to unreasonable expectations and burdens on personnel.
• Lack of a core of personnel trained in the art and science of managing large projects (not to mention negotiating large contracts). As a result, individual units were left to fend for themselves, with unfortunate and expensive results.
• Absence of standard forms and procedures to guide users when they were indeed forced to fend for themselves.

A word should also be said about the broader consequences of these difficulties. The responsible CIO has returned to the private sector and the State has represented that all is not lost – that portions of these troubled projects can still be put to productive use. In a private company, the results might have been more dramatic. Such losses, combined with the evidence of failed oversight and leadership, might well cause a wholesale dismissal of management, not to mention the possibility of shareholder derivative suits.
Whatever the political or economic consequences, Wisconsin illustrates the importance of professional, systematic management of IT contracts (indeed, of all contracts) and the importance careful, detailed planning by trained and experienced personnel.

Towards a Successful ASP Agreement, Pt. 2

Where Has All My Data Gone?

Microsoft and T Mobile are currently receiving lots of bad press relating to the loss of data from users of T Mobile's Sidekick device. Because Microsoft was essentially acting as an ASP for those users – storing data for them – this event raises some interesting questions.

What happens if my ASP loses my data?

That depends on whether the data is critical to the operation of your business. If you were using thousands of Sidekicks to run a world-wide enterprise, you would probably be out of business. In a more likely scenario, if you had out-sourced complex calculations or processing of a large volume of information, you might still face a serious business interruption. Orders may not be filled correctly, or on time, trading (or reporting) deadlines might slip past or payroll checks might not issue on time.

Of course “What happens if my data is lost?” is the obvious question. A better one is “What is the appropriate response to such a loss?” While the answer is obvious – restore the data – it begs a number of questions:

Who is responsible for the restoration?
Who will pay the cost?
Where will the information for the restoration come from?

According to media reports, in the Sidekick case both the main and backup databases were lost due to a server malfunction. From that it would appear Microsoft observed one of the main commandments of computing: Back up your data. Whether the back ups were made in an appropriate manner is not clear from published reports. For example, were the main and back up files stored on the same server, meaning that one failure would interrupt access to both?

User difficulties were apparently compounded by the fact they did not have copies of their data on their PCs or their Sidekick devices. They could not simply update Sidekick from PC (or vice versa) and continue with their lives while Microsoft and T Mobile sorted things out. They had put “all their eggs in one basket,” and lost the basket.

The lessons are not new:

Back up your data
Observe appropriate back up protocols (e.g. off-site storage of back up data)
Keep your own copies, just in case

What safeguards should I observe?

The first safeguard has just been mentioned – keep your own copies. The second is to require the provider to keep its own back ups, and to observe appropriate processes and protocols. At a minimum the contract should require back ups, and specify the protocols to be observed. Failure of provider to meet these obligations should be defined as a material breach of the agreement, constituting grounds for subscriber to terminate the contract. But mere words on the page may be insufficient, particularly if the data is critical, sensitive or protected by law. After all, the ink on the page will not rise up and compel provider to perform. Even a court order enforcing the written words might come only after significant, if not irreparable, harm, has been done to your business. The more important the information is to your business, therefore, the more important it is to verify provider's data protection practices before the contract is signed, and to inspect provider's facility periodically to ensure compliance during the contract term. To avoid any doubt or dispute on the question, the right to inspect should be written into the agreement.

Yet the ASP model contains an exposure not presented by the Sidekick occurrence – loss of functionality. Users could still access the various functions of their devices, but could not retrieve their stored data. But what if the software that enabled each device to communicate with the servers had failed? What if software used by your payroll processor abruptly doubles everyone's salary?

Mitigating this exposure in the ASP context requires a bit of care. Should subscribers require their providers to have off-site back up facilities, able to begin processing in the event a disaster strikes the primary site? Absolutely, if the provider is providing services critical to subscriber's business. Again, the need for careful drafting by counsel and due diligence by business personnel rises in direct proportion to the importance of the services. If provider cannot, or will not, provide such a mirror back up site, consider creating one of your own, complete with copies of the necessary software, licensed for use in the event of catastrophic failure by provider. (Care would also be needed to define what constitutes a “catastrophic failure”.) But if subscriber is compelled to create its own emergency back up site, is the ASP investment cost effective?

What are my remedies?

“What are my remedies?” is simply a polite way of asking “What does my provider owe me?” Assuming the contract has been properly researched, drafted and enforced, the answer should be “Nothing.” A failure at provider's facilities should pass virtually unnoticed by subscriber, certainly without loss of critical information or functionality. That would be an excellent example of a contract that works. After all, even if a court eventually orders provider to hand over a sack of money equal to the value of the interrupted services, it may be too little and too late to pay for your attorneys, lost business and loss of goodwill.

COMING SOON: Part 3 - When is a warranty of 99.7% availability a bad deal?

Steps Towards a Successful ASP, Part 1

The Application Service Provider ("ASP") model has many attractions:

• Someone else pays for the hardware, software, maintenance, updates, etc.
• Subscriber pays a fraction of the total cost of implementation.
  
• Competition gives each provider an incentive to continually improve their offerings.

But there are also potential down sides. Someone else controls the:

• Hardware
• Software
• Personnel
• Security
• Back up
• Disaster recovery

If your ASP crashes, will it take along your critical processes?

Setting aside the worst case, there are other reasons an ASP may be unsuitable.

• A third part will be given access to subscriber's data.
• The services offered may not precisely meet subscriber's business needs or processes. Because many ASP business models are based on standardization and volume, provider may be unwilling to offer custom solutions for individual subscribers. In that event, is it cost effective for subscriber to revise its policies and practices to accommodate the proposed application?
  
• Provider, not subscriber, controls when and how changes to the application are made. If changes effect the "look and feel" of the application, subscriber may encounter unexpected costs and delays for re-training personnel. Alternatively, subscriber may not want the added features, and extra cost, of Version X +1, while provider may not want to continue to support Version X.
  
• In general, neither provider nor subscriber controls the transmission lines, creating an exposure - interruption - that neither party can fully control.
• Quitting a provider can be problematic. For example, will the provider simply block subscriber's access to the application or provide transition assistance. If assistance is provided, what will it cost?

Some of these concerns can be addressed through careful contracting, some cannot. If granting provider access to subscriber's data violates privacy laws, or if provider's business model does not allow for needed customization, no amount of brilliant lawyering will make the deal work.

What then, are the foundations of a successful ASP project? A full answer would require a book (See shamless plug below), but I can address some highlights here.

1. Due Diligence

As mentioned, if provider's offerings do not meet subscriber's needs, the relationship will be rocky, at best. But diligence must extend beyond matching needs to services.

• Is provider financially sound?
• Are its personnel adequately trained?
• Does vendor have all the necessary permits and licenses?
• Does vendor implement appropriate physical and logical security?
• Are appropriate disaster recovery measures in place?

It is important that subscriber verify this information before signing the contract. Afterwards, one can always sue if, for example, there is a disaster and provider does not have in place the recovery safeguards required by the contract. Subscriber might well recover money damages, but those will be cold comfort if he/she has been put out of business.

2. Interim Remedies

In fiction, at least, the appropriate response to any business disagreement appears to be "So Sue Me!" A better approach might be to build in "safety valves" to permit discrete disagreements to be resolved without derailing the entire project. Examples would be meet and confer requirements, permission to withhold payment of disputed balances and a narrow list of circumstances in which provider may suspend or terminate services.

3. Objective Standards

When will the application be available?
What processing speeds will be provided?
What response times will be accepted?
In what format will subscriber's data be stored?
In what format will reports and other output be delivered to subscriber?

To protect both sides, it is important that these and other standards be stated clearly and be subject to objective measurement. Without objective standards, subjective judgments ("It's too slow!") could put the project at risk.

4. Testing

We will purchase an auto after a 20 minute test drive and a house after a walk-through and home inspection (and a little haggling). Outsourcing business functions, however, merits more detailed examination. The more critical the processes, the more important it is to thoroughly test the application in the most realistic setting possible.

5. Business Continuity

If the provider's servers fail, will subscriber's business come to halt? Thus the need to verify that provider observes proper security, back up and recovery procedures. For truly critical processes, consider requiring a source code escrow (which must be kept current) or maintenance of a mirror site that could easily be taken over by subscriber.

We will explore these and other features of a successful ASP contract in greater detail in future postings.


This article is derived from "Application Service Provider and Software As A Service Agreements Line by Line," Kelly L. Frey, Sr. and Thomas J. Hall. Published by Aspatore Press, 2007.

Note - my friend and colleague Kelly Frey had no part in drafting this post. Any errors in it are mine alone.

"B" is for "Breach"

The term “material breach” is a familiar one. It appears routinely in contracts, generally in the termination provisions:Ether party may terminate this agreement if the other commits a material breach of any term hereof, and fails to cure such breach within thirty days of receiving written notice of the existence thereof.” At first reading, this provision appears to be clear, fair and easily enforced. It is mutual – it protects either party. It provides notice and an opportunity to cure, in the event the breach was inadvertent. It permits termination only for serious - “material” - breaches. But what is a “material breach”?

In the legal world, a breach is failure to fulfill an obligation set forth in a contract. A “material breach” is a failure so severe that it threatens the value of the entire contract. For example, if a customer orders one ton of steel, she will probably not want to terminate the contract if the vendor delivers only 1, 998 pounds, rather than the 2,000 expected. Vendor might issue a credit or refund or promise to deliver the missing material immediately. Or customer might overlook the missing two pounds as inconsequential. In contrast, if the vendor delivers one ton of brass rather than steel, customer may wish to cancel the order or terminate the supply contract. If we assume the customer needs the steel for an office tower, the brass simply will not suffice; it is simply not strong enough. Clearly a material breach.

Or is it?

Assume the contract says “metal,” rather than “steel.” Brass is a metal.

Assume customer wants to terminate the contract because she needs the steel immediately, and lacks the time to wait for vendor to deliver the correct product. Is timely and accurate delivery a condition of the contract? Is it a MATERIAL condition of the contract? Put another way, did vendor know that the contract required him to deliver the right product, at the right time?

What if the contract simply calls for “steel”? Does it matter whether vendor delivers the latest space-age alloy or a truckload of rusting auto parts?

Let's change industries. Customer orders a “computer.”

• Does it matter that the new device processes 16 million instructions per section, when the industry standard is 25 MIPS?

• Does it matter if customer paid a discount price?

• Does it matter if customer paid a premium price?

• Does it matter if the product is delivered “a little” late?

• That is “slightly” over budget? What is “slightly”?

• That it “doesn't quite” work? What is “doesn't quite”?

• That it runs fine as a stand alone, but won't interface with customer's systems?

• That it won't run customer's software?

• Does it matter whether that software is incidental or critical to customer's operations?

Lawyers have a method for finding answers to questions such as these. They call it “discovery.” It is one of the more expensive and time consuming parts of a lawsuit. If there is enough money at stake, vendor's lawyers will leave no file untouched, and no employee not-interviewed, in an effort to show that the alleged breach is not material – that the failure (assuming there was one) did not cause real and substantive harm to customer. Alternatively, vendor's lawyers will argue that the product or service complained about meets the standards set forth in the contract, or that customer never disclosed that requirement X would be central to the deal. Vendor's lawyers will suggest that, at best, customer is mistaken or confused; at worst they will suggest that customer is making a dishonest attempt to escape the contract, for whatever reason.

Which delivers us to a quandary: What is a drafter to do if the officially sanctioned term “material breach” is simply an invitation to dispute and litigation?

Change the definition.

The problem is not the term itself, but the meaning given that term by the law. But, in commercial contracts, laws, regulations, and legal definitions are generally DEFAULT provisions – they apply only if the parties do not set their own rules or definitions. (Within limits. A contract to commit a crime is still a crime, and unenforceable.)

Which of these provisions would you prefer to administer and enforce?

“Either party may terminate this agreement if the other commits a material breach of any term hereof, and fails to cure such breach within thirty days of receiving written notice of the existence thereof.”

OR

“Either party may terminate this agreement if the other commits a material breach of any term hereof, and fails to cure such breach within thirty days of receiving written notice of the existence thereof.

“For the purposes of this provision, 'material breach' shall mean....”

Admittedly, the latter is more difficult to complete. Each party must ask itself “What would cause me to want to call off this deal?” Then they must persuade the other party to include those provisions in the agreement. Both steps run counter to the common understanding of the deal process - “Get it done” and “Be positive.” A more realistic rule is probably “Be thorough.” The more time spent up-front spelling out the details of a deal – and identifying the key parts of the deal – the less time will be spent arguing about perceived failings.

Or, as our parents always taught us: “Get it right the first time.”

Copyright 2006, Thomas J. Hall. All rights reserved.

ACCEPTANCE TESTING

The project is complete. The hardware, bright, shiny and new is in place, with glowing lights and drives humming with new software. The vendor rep performs the initial log in, demonstrates a few features, pockets the last check and leaves. Over the next few days, problems begin to manifest:

• The new system is painfully slow to boot;
• The automatic backup feature will not run reliably;
• The system will not interface with two of your legacy systems;
• The system incorrectly calculates withholding taxes when printing paychecks.
  

The system commits yet another error and remits an incorrect amount of withholding taxes to the IRS;It completely corrupts the accounts receivable file.

Vendor sends in a technician, who announces that all is well with the system and attributes the difficulties to user error. He leaves behind a large bill for his time.

Could this situation - which is deliberately exaggerated - have been avoided?

• Certain questions come immediately to mind:
• What did vendor commit to deliver"
• What did customer expect to receive?
• What does the contract say customer was to receive?
• What performance standard does the contract specify?

In this instance, the contract refers only to “published specifications” and contains no testing procedures. Of all the safeguards that may be built into a contract, acceptance testing is perhaps the most important. They are the best way to ensure the deliverables:

• Performance promised; and
• Meet customer's needs.

To be effective, the performance standards and testing procedures must be:

• Mutually agreed; and,
• Objectively measurable.

Such acceptance procedures protect customer and vendor. Vendor is protected against a customer who, for whatever reason, wants to escape the contract. Customer is protected against a product that does not perform as customer needs and wants it to.

Vendors occasionally balk at acceptance testing. Not because they object to demonstrating the quality of their work, but because, typically, customers will withhold final payment until acceptance is successfully completed. Vendors, like everyone else, prefer to be paid in full, and as quickly as possible. Clear, objective standards can do much to overcome this concern. If the customer cannot withhold payment arbitrarily, then timely payment depends only on the quality of vendor’s work.

Acceptance testing will not ensure that a project will be on time or within budget, but it will do much to ensure that the final product will perform as promised, and as needed.