Archiving101.com

Source: http://www.washingtonpost.com/wp-dyn/content/article/2008/08/20/AR2008082002617.html?hpid=moreheadlines

WASHINGTON — The White House is missing as many as 225 days of e-mail dating back to 2003 and there is little if any likelihood a recovery effort will be completed by the time the Bush administration leaves office, according to an internal White House draft document obtained by The Associated Press.

The nine-page outline of the White House’s e-mail problems invites companies to bid on a project to recover the missing electronic messages.

The work would be carried out through April 19, 2009, according to the Office of Administration request for contractors’ proposals, which was dated June 20.

Last week, the White House declined to comment on the document.

On Wednesday, the White House refused to talk about internal White House contracting procedures, but said the information is “outdated and seriously inaccurate.” It would not elaborate. The White House also declined to say whether it has hired a contractor for the work yet.

“With an eye on the clock, the White House continues to drag its feet and do everything possible to postpone public access to the records of this presidency,” said Anne Weismann, chief counsel to Citizens for Responsibility and Ethics in Washington, a private watchdog group.

The draft document outlines a process in which private contractors would attempt to retrieve lost e-mail from 35,000 disaster recovery backup tapes dating back to October 2003, a period covering such events as growing violence in Iraq, the Abu Ghraib prison scandal and the criminal probe into the disclosure that Valerie Plame had worked for the CIA.

The recovery project would not use backup tapes going back to March 2003, according to the draft document, even though an earlier White House assessment suggested e-mails were missing from that period as well.

Industry experts point out that relying on the backup system to ensure accurate retention, preservation and retrieval of all e-mails is problematic because it does not take into account deleted e-mails.

“A backup system isn’t designed to be a 100 percent complete inventory of all e-mails,” says William P. Lyons, chairman and chief executive of AXS-One, a provider of records compliance management solutions.

“It’s designed to make a copy of data at a specific point-in-time,” said Lyons. “Data is backed up on a daily, weekly and monthly basis as part of a disaster recovery strategy, to ensure to protect the organization from data loss.”

The White House draft document says that the number of days of missing e-mail ranges from 25 to 225, a range that industry experts say would make it difficult to bid on a recovery project.

“Generally, when the scope of the work is expected to fluctuate by a factor of nearly ten, I can only take you so seriously,” said Steve Schooner, co-director of the Government Procurement program at George Washington University.

“Contractors cannot accurately plan for or staff based on such an estimate,” said Schooner.

At a hearing on Capitol Hill in February, the White House told Congress it was trying to determine how many e-mails were missing. An earlier analysis from 2005 estimated the number of days of missing e-mails at 473 over a period of 20 months.

While the higher number would appear to suggest the White House has found a large amount of previously missing e-mail, that may not necessarily be the case. Industry experts say it is unclear from the brief description in the draft document whether the missing-day measurements in that document and those in the earlier analysis can be compared.

“We will continue to work with members of Congress and the National Archives and will communicate the results of our accounting effort at an appropriate time,” White House spokeswoman Emily Lawrimore said.

Rep. Henry Waxman, D-Calif., and chairman of the House Committee on Oversight and Government Reform, has said the White House’s failure to properly archive e-mails violated the Presidential Records Act. The top lawyer for the National Archives has expressed disappointment the White House did not have a formal records management system in place.

On Wednesday, House Democratic Caucus chairman Rahm Emmanuel of Illinois criticized how the problem has been handled, saying, “The White House that wants to keep track of all your e-mail and phone records can’t even keep track of their own.”

Kinda taking this off topic in a sense, but interesting to read non the less.  This is from the highly valuable Active Directory discussion list where Don Hacherl, who was the Lead Development Manager at Microsoft for AD, posted this story:

 The oldest traceable part of AD started life at 3Com in 1988 or 1989.  This was an (incomplete!) X.500-ish directory with custom communication protocols, built on top of a C-Tree database, running under 16-bit OS/2.  By 1990 3Com had abandoned its network software efforts and the directory code moved to Microsoft as part of some complicated deal.  The LanMan group planned to include the directory service in LanMan 3.0 and immediately started porting it to the JET Blue ISAM and building an RPC front end compliant with the X/Open XDS API.

IT history has more interesting stories .. I bet not many know the background of “Squeaky Lobster” in Exchange Server.

I’ve seen a few companies use this phrase now and the more I think about it, the more it really makes sense to explain the value of an archiving system to a business this way.  Instead of showing off with all the laws and regulations that my unfortunate brain has had to consume, relating it to something that everyone understands is refreshing.

Why an insurance policy?  Well .. in a sense it kinda works the same.  An archiving solution simply will save you money in case of an eDiscovery request. Its like driving a car without car insurance.

Having been in the archiving industry for many years, one of the things I have noticed is that almost every vendor comes up with its own terminology about the same technology.   The term ‘archiving’ is sometimes used to refer to the physical capturing of the information non-journaling .. but I’ve also seen it being used to refer to stubbing, shortcutting, extending or stripping of items that already have been captured.

That brings me to the whole stubbing thing.  Stubbing, Archiving, Extending, Stripping, Shortcutting … if we as vendors already have a problem with understanding our own terminology .. what do you think the prospects think?   However .. for some reason the industry has come to a realization that the stripping, stubbing, shortcutting of data that is stored on a File Server is best to be referred to as File System Archiving .. almost every vendor uses that phrase.

Can’t we just come to an agreement to use a single ‘name’ for things ?

I’ve started to receive quite a few comments in the last few days from people who purchased their archiving product but are finding that support is not responding adequately to their problems.     Not sure what is going on here .. and I would be interested to hear from you if you have heard this from other sources as well.

So lately I’ve been reading quite a bit on the usability of products. With this I’m not really referring to what the administrator sees when he opens up his console on Monday morning sipping a cup of coffee, but the stuff that the end users see and work with.

The success of an archiving solution partially (I’m not saying entirely here on purpose)  depends on if users can comfortably use the application to their benefit.  Important to understand is that a solution or feature that is too complicated to understand or use is simply going to be ignored by the end users.  A properly designed solution ensures that users optimally leverage the functionality given to them, resulting actually in a benefit for the employer (that could be increased productivity for instance).

Now .. if I talk to Product Managers from competing vendors they all agree that usability is important and they tout that theirs has the ‘best of class’ or ‘best in the industry’ … or whatever this weeks hot description is.  If that is the case .. why does my head hurt sometimes when I see screenshots of the user interface?  If it hurts me .. it probably hurts many other brains as well.

A solution should not only be technical capable .. but also usable.  There are tons of reports out there that probably give you some rough number or idea on ‘extra savings’ a proper designed user interface will give you so I’m not going to quote those (Google them if you wish).  In the last Gartner Magic Quadrant … ‘ease of use’ was seen as a big bonus point … and I’m glad that that is starting to be recognized.  The fact that archiving solutions solve a very complex problem doesn’t automatically mean that these solutions have to look complex or be complex to use.

Remember .. your end users will love you

Historically consolidating your archiving systems or moving to a new archiving product has been challenging in many occasion.  The fact that customers choose another archiving product isn’t always because something else is better (it is a large percentage however).  Companies get acquired and there might be another product in use already that will be made standard.  Or … customers opt to move away from an on premise solution to a hosted archive.  In my years in this industry I’ve seen it all … most often these migrations were painful, time consuming and costly.  The easiest way was to export all the data to PST (if that option was even available), and then import it back into the new system.

However you now can buy an off-the-shelf product that can help you migrate your archived data to a new system.  The folks at Essential in the UK have written it.  Right now it supports only a few archiving vendors, but I suspect more will follow. More details are at http://www.essential.co.uk/product/transvault/index.asp

Looks like people are finally listening and seeing that stubbing/shortcutting truly is ‘evil’ and that it has negative impact on an Exchange environment.  Quoting from:

 http://technet.microsoft.com/en-us/library/cc671168(EXCHG.80).aspx

Third-party archiving solutions have become popular as corporate compliance requirements and mailbox quota management have gained importance. Many of these archiving solutions offer the ability to leave a small stub file in place of the archived message that can be used by end users to retrieve archived messages from the archival system. Some organizations use the stub file solution as a workaround to offering large mailboxes. One of the goals of stub archiving solutions is to reduce the aggregate mailbox and database size, thereby reducing recovery time objectives (RTOs). On the surface, this appears to be a good idea. However, stub-based archiving solutions have the following technical problems:

• Server performance   Removing the message bodies and attachments from Exchange reduces the mailbox size, but it does not significantly change the server performance for users accessing Exchange via Outlook in online mode and Outlook Web Access. Item counts are the primary performance driver for the Exchange store, and not aggregate size. For example, server performance with a folder containing 100 KB of full e-mail message items is similar to a folder containing 100 KB of stub files.
• Client complexity   Because the use of stub files with a third-party archiving solution requires the deployment and use of Outlook add-ins, a significant amount of time must be spent by administrators to deploy and manage these add-ins. Administrator time is also required to assist end users with technical difficulties using the add-ins. Not deploying stub files removes all of this additional administrative work that must be performed, thereby allowing more time to administrators and end users.

Today I raise a glass and toast ….  for the non-believers .. time to change opinion.  Archiving vendors need to change their ways to capture information.

This week I’ll be spending some time at LegalTech. My employer has a booth at this conference and during todays setup I kinda walked around and chatted a bit with the competition in the space (even though one might be a competitor, you should still be able to have a nice talk as two grown adults right?).    With only a few archiving vendors actually having booths, the focus at the conference goes beyond just archiving and it will be nice to hopefully see some new and interesting technology. Expect to see a few press releases this week (like Mimosa Systems announcing their new SDK availability for NearPoint)

In other news .. not all US States apparently have the same records rentention laws. In Hawaii, government records that must be made available under open records law include agency meeting minutes and documents such as government contracts and purchasing information. The law requires government agencies to make their records available to ensure an honest and open government.

However, the law does not require state agencies to maintain e-mails and other documents, even those that rise to the standard of a record. So, while people have a right to inspect public records, including e-mails, the law doesn’t provide any guarantee that they will exist.  More at: http://www.kpua.net/news.php?id=15198