Delivery vs. Installation vs. Successful Acceptance Testing

When should Customer be expected to pay for the new system?  (For this discussion, let's assume the system is a somewhat customized mix of hardware and software.)

When the system is ordered?  Perhaps, particularly if vendor is a reseller who must pay suppliers for the components.  Vendors may balk at fronting lots of money for customers.  The same analysis applies to paying upon delivery.  

But customer is left with a significant exposure.  What if some of the software disks are corrupted, or the hardware contains defective components?  Such deficiencies may not be detected until the system is assembled, installed and turned on.  Let's assume that when the power switch is thrown, something goes ZAP!! and a cloud of smoke rises from the system cabinet.  In an ideal world we would expect vendor to step up, identify the problem and make it right.  But we do not live in an ideal world and vendor may have moved on to the next project.  Requests for assistance might be met with suggestions that the problem resulted from user error, or recommendations to take up the question with the manufacturer.  (In fairness to vendors, we must acknowledge that some errors or failures may indeed be caused by the users.)  If vendor and manufacturers have been paid in full, customer has little leverage, little opportunity to compel them to pay attention. 

Which leads us to payments on the "installment plan": a portion upon order, a portion upon delivery and the bulk upon successful acceptance testing.  If a substantial portion of the contract price is held back until testing is complete, vendor will not lose interest in the project. 

Of course, vendors will not agree to large hold-backs if customer may arbitrarily decide not to pay.  It would not be good business to allow customer to refuse to pay simply because customer doesn't like the final product or now believes it "doesn't work."  As a result, to be successful, hold backs must be tied to testing standards that are mutually agreed and which can be objectively measured.  In that way, vendor and customer are both protected - customer can withhold payment if the system does not "work," while vendor must be paid if it does.

Case Study – Wisconsin’s $100 Million Loss

Note: The promised piece on ASP and SaaS availability standards has been postponed for one week.


It has been said that IT projects are always late and over budget. The most commonly given reasons are:

• Vendor errors;
• Vendor misconduct;
• Changed circumstances;
• Unforeseen obstacles.

Regarding “vendor misconduct,” it is easy to find IT executives willing to believe that vendors deliberately under estimate cost and time requirements to win contracts, only to have the true cost become apparent as the project unfolds. Such “low balling” undoubtedly occurs, but more often vendor and customer must share responsibility for a troubled project. If “blame” is to be assigned, perhaps it should fall upon customers who ignore the ancient maxim: “caveat emptor.”

Remarkable examples of “how projects go wrong” are provided by the State of Wisconsin, which in late 2006 and early 2007 confronted a number of troubled projects, with an exposure of some $100 million. The figure is all the more significant when one considers that the State’s annual IT budget is roughly $400 million. A quick survey of the troubled projects will help set the stage:

• In 2006 the University of Wisconsin discontinued a program to update its salary and benefits system, after spending five years and $26 million.
• As of Spring, 2007, a new system to track unemployment taxes was 3.5 years late and $12.2 million over budget.
• A new voter registration system was more than two years late (i.e. ready two years after the deadline set by Federal law).
  
• New sales and use tax software required a $5.7 million fix.
• Work on a new system to track unemployment claims was suspended in February, 2007, after an expenditure of $23.6 million.
• A new system to handle automobile registrations was abandoned. It was at least a year late, $9.4 million over budget and had significantly increased wait times at the DMV offices.

This string of difficulties prompted members of the State legislature to request a formal audit by the Legislative Audit Bureau (Wisconsin’s equivalent to the Federal General Accounting Office). According to the Bureau, these projects had much in common:

• Failure to appreciate the complexity of the projects;
• Failure to define the final function of the various systems;
• Failure to observe contracting best practices, such as standard procedures and incentive/penalty clauses.
• Lack of leadership and oversight.

Significantly, ALL of these variables were within the control of the State.

As a case study, Wisconsin’s difficulties are rich in lessons, for it appears the State made virtually every mistake in the proverbial book. It made many of them repeatedly:

• Failure to realistically allocate resources;
• Failure to plan in detail;
• Failure to plan with an end in mind.

Of these, perhaps the most damaging was the failure to plan with an end in mind. Without a clear goal in mind, it is not possible to set meaningful specifications. In turn, an absence of meaningful specifications opens the door to “mission creep” as each user tacks on items from his or her wish list. Cost and time requirements can and probably will increase dramatically. Even without mission creep, ambiguous specs are a recipe for failure. Vendor may believe that customer wants and expects “X,” while customer is expecting something quite different. Unfortunately, the misunderstanding might not be discovered until late in the project, after much time and cash has been invested. Also, without clear specs, it is not possible to develop reliable cost estimates or time tables. In a very real sense, the various Wisconsin projects were not over due or over budget, for they had never been properly planned or budgeted. Quite simply, the State did not know what it was getting into, and paid the price.

Taken as a whole, the Wisconsin projects suggest an IT unit in disarray:

• Lack of necessary expertise;
• Lack of leadership willing to set reasonable expectations;
• Lack of uniform contract forms and contracting procedures to guide personnel who might not be versed contractual matters;
• Lack of oversight.

The lack of oversight bears special consideration. The various projects discussed here fell primarily within the responsibility of the Department of Administration (aptly known as “DOA”).  That department failed in its responsibilities because it was pre-occupied with two troubled tech projects of its own. A second level of supervision is, in theory, provided by committees of the State legislature. Those committees, however, have been inactive since at least 2005.

Wisconsin’s misadventures offer a sobering example of the many ways IT projects may go wrong:

• Failed leadership, leading to unreasonable commitments. These commitments led, in turn, to unreasonable expectations and burdens on personnel.
• Lack of a core of personnel trained in the art and science of managing large projects (not to mention negotiating large contracts). As a result, individual units were left to fend for themselves, with unfortunate and expensive results.
• Absence of standard forms and procedures to guide users when they were indeed forced to fend for themselves.

A word should also be said about the broader consequences of these difficulties. The responsible CIO has returned to the private sector and the State has represented that all is not lost – that portions of these troubled projects can still be put to productive use. In a private company, the results might have been more dramatic. Such losses, combined with the evidence of failed oversight and leadership, might well cause a wholesale dismissal of management, not to mention the possibility of shareholder derivative suits.
Whatever the political or economic consequences, Wisconsin illustrates the importance of professional, systematic management of IT contracts (indeed, of all contracts) and the importance careful, detailed planning by trained and experienced personnel.

Towards a Successful ASP Agreement, Pt. 2

Where Has All My Data Gone?

Microsoft and T Mobile are currently receiving lots of bad press relating to the loss of data from users of T Mobile's Sidekick device. Because Microsoft was essentially acting as an ASP for those users – storing data for them – this event raises some interesting questions.

What happens if my ASP loses my data?

That depends on whether the data is critical to the operation of your business. If you were using thousands of Sidekicks to run a world-wide enterprise, you would probably be out of business. In a more likely scenario, if you had out-sourced complex calculations or processing of a large volume of information, you might still face a serious business interruption. Orders may not be filled correctly, or on time, trading (or reporting) deadlines might slip past or payroll checks might not issue on time.

Of course “What happens if my data is lost?” is the obvious question. A better one is “What is the appropriate response to such a loss?” While the answer is obvious – restore the data – it begs a number of questions:

Who is responsible for the restoration?
Who will pay the cost?
Where will the information for the restoration come from?

According to media reports, in the Sidekick case both the main and backup databases were lost due to a server malfunction. From that it would appear Microsoft observed one of the main commandments of computing: Back up your data. Whether the back ups were made in an appropriate manner is not clear from published reports. For example, were the main and back up files stored on the same server, meaning that one failure would interrupt access to both?

User difficulties were apparently compounded by the fact they did not have copies of their data on their PCs or their Sidekick devices. They could not simply update Sidekick from PC (or vice versa) and continue with their lives while Microsoft and T Mobile sorted things out. They had put “all their eggs in one basket,” and lost the basket.

The lessons are not new:

Back up your data
Observe appropriate back up protocols (e.g. off-site storage of back up data)
Keep your own copies, just in case

What safeguards should I observe?

The first safeguard has just been mentioned – keep your own copies. The second is to require the provider to keep its own back ups, and to observe appropriate processes and protocols. At a minimum the contract should require back ups, and specify the protocols to be observed. Failure of provider to meet these obligations should be defined as a material breach of the agreement, constituting grounds for subscriber to terminate the contract. But mere words on the page may be insufficient, particularly if the data is critical, sensitive or protected by law. After all, the ink on the page will not rise up and compel provider to perform. Even a court order enforcing the written words might come only after significant, if not irreparable, harm, has been done to your business. The more important the information is to your business, therefore, the more important it is to verify provider's data protection practices before the contract is signed, and to inspect provider's facility periodically to ensure compliance during the contract term. To avoid any doubt or dispute on the question, the right to inspect should be written into the agreement.

Yet the ASP model contains an exposure not presented by the Sidekick occurrence – loss of functionality. Users could still access the various functions of their devices, but could not retrieve their stored data. But what if the software that enabled each device to communicate with the servers had failed? What if software used by your payroll processor abruptly doubles everyone's salary?

Mitigating this exposure in the ASP context requires a bit of care. Should subscribers require their providers to have off-site back up facilities, able to begin processing in the event a disaster strikes the primary site? Absolutely, if the provider is providing services critical to subscriber's business. Again, the need for careful drafting by counsel and due diligence by business personnel rises in direct proportion to the importance of the services. If provider cannot, or will not, provide such a mirror back up site, consider creating one of your own, complete with copies of the necessary software, licensed for use in the event of catastrophic failure by provider. (Care would also be needed to define what constitutes a “catastrophic failure”.) But if subscriber is compelled to create its own emergency back up site, is the ASP investment cost effective?

What are my remedies?

“What are my remedies?” is simply a polite way of asking “What does my provider owe me?” Assuming the contract has been properly researched, drafted and enforced, the answer should be “Nothing.” A failure at provider's facilities should pass virtually unnoticed by subscriber, certainly without loss of critical information or functionality. That would be an excellent example of a contract that works. After all, even if a court eventually orders provider to hand over a sack of money equal to the value of the interrupted services, it may be too little and too late to pay for your attorneys, lost business and loss of goodwill.

COMING SOON: Part 3 - When is a warranty of 99.7% availability a bad deal?