Free Administrative Record (Supplement) - District Court of Federal Claims - federal

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED Filed 05/25/2007 Page 1 of 34 COPY MAY BE MADE PUBLIC

IN THE UNITED STATES COURT OF FEDERAL CLAIMS CHE CONSULTING, INC., Plaintiff, v. THE UNITED STATES, Defendant, and STORAGE TECHNOLOGY CORPORATION, Defendant-Intervenor.

DECLARATION OF BRENDA SPENCE

I, Brenda Spence, pursuant to 28 U.S.C. § 1746, do hereby declare: 1. I am Senior Contracting Officer with the United States General Services Administration

(GSA). My duty assignment is in support of GSA services at Stennis Space Center, MS. I have served in this position with GSA from February 2004 until today, and have served as a warranted Contracting Officer for over 24 years. 2. This declaration supplements the CO Contracting Officer's Statement of Fact and

Position that I prepared previously in December 2006 and is a part of the record in this case. This declaration addresses the following matters: Summary of the Supplemental Market Research Findings Original Market Research Additional Market Research Small Business Capability to Meet Combined Requirements FAR 7.107, Acquisitions Involving Bundling, Inapplicability

3. Supplemental market research findings are as follows:

AR 385

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED Filed 05/25/2007 Page 2 of 34 COPY MAY BE MADE PUBLIC

a. Agencies that are acquiring hardware and software services separately have systems that are not as complex or mission critical as the systems and missions supported at the Naval Oceanographic Office (NAVOCEANO) Major Shared Resource Center (MSRC) , (i.e. National Aeronautical Space Agency (NASA), Air Force Material Command Air Force Flight Test Center (AFMC/AFFTC), Department of Treasury). Temporary system outages do not have any significant negative impact on the programs/missions these agencies support. In fact, some agencies maintain their systems by using in house technical staff and any repairs that cannot be resolved at their level would be redirected to the original equipment manufacturer (OEM) or OEM service provider on a case by case basis. b. Agencies with complex and/or mission critical systems (i.e. United States (US) Army Research Laboratory (ARL) MSRC, Aeronautical Systems Center (ASC) MSRC and US Army Corp of Engineers (USACE) Engineering Research Development Center (ERDC) MSRC, NITC, MSS), acquire hardware and software as a total solution from one maintenance provider who has an established agreement with the original equipment manufacturer (OEM). c. Agencies with complex and/or mission critical systems agree a one-maintenanceprovider approach is critical to ensure continuity of operations. The risks associated with separating hardware and software maintenance is considered to be too great. d. CHE may have misrepresented the maintenance strategy of DISA by stating this agency splits STK hardware and software maintenance requirements. The DISA maintenance requirement includes support of software that is "integral and essential" to the operation of the hardware and requires the service provider to have an established relationship with

2

AR 386

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED Filed 05/25/2007 Page 3 of 34 COPY MAY BE MADE PUBLIC

the OEM.

DISA does have a separate order for support for the production software the

Automated Cartridge System Library Software (ACSLS) with STK. e. Based solely on information obtained from STK, CHE's claim of having 12% of the market appears to be unsubstantiated. f. There are multiple small business third party maintenance providers with established relationships with the OEM that can and do perform both hardware and software maintenance. Many of these vendors have established federal supply schedule (FSS) contracts with General Services Administration (GSA) to provide hardware and software maintenance on Storage Technology (STK)/Sun equipment. Competitive quotes have been received from a small business in response to solicitations 4THG17078067 and 4THG17078068. This vendor has successfully performed contracts of the same size and complexity for other Department of Defense (DOD) and federal agencies.

g. Federal Acquisition Regulation Subpart 7.107, Acquisitions Involving Bundling, is inapplicable.

4. The original market research performed was based on the guidance in Federal Acquisition Regulation (FAR) Part 10 which states agencies must conduct market research appropriate to the circumstances before soliciting offers for acquisitions. Prior to the issuance of solicitations 4THG17078067, 4THG17078068, 4THG17072001 and 4THG1702002, Federal Supply Schedules (FSS) were researched to determine the availability of the requirements under schedule. Multiple vendors offering hardware and software maintenance for Storage Technology (STK)/Sun equipment were identified. In addition, requirements were issued via E-Buy for the

3

AR 387

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED Filed 05/25/2007 Page 4 of 34 COPY MAY BE MADE PUBLIC

purpose of identifying companies that would have an interest in submitting a quote. Multiple sources responded to the E-Buy notification and the solicitations were issued to multiple sources. Based on the market response to the E-Buy notification, the number of small business sources available on the FSS, and the commercial nature of the services required, it was determined sufficient market research had been conducted and the solicitations were issued. Total value of the requirement for a three year period is less than $1.5 million.

5. Additional market research was conducted during the period April 5, 2007 through May 3, 2007. Approximately 40 phone calls and emails were issued to various sources for information. The following synopsis provides the results of that effort:

a. Other DOD MSRCs. The other three DOD MSRC's were contacted to verify their maintenance strategy for STK/Sun equipment. The MSRC's located at ARL, ASC, and USACE ERDC confirmed the maintenance strategy is and has always been one service provider for both hardware and software. There are two other smaller centers located in Alaska and Hawaii but these centers were not contacted since they have limited STK maintenance requirements. Specific MSRC responses are as follows:

i. Greg Rottman, USACE ERDC, stated "In short we utilize one contract for both. I want one throat to choke." The systems at USACE ERDC include SL8500 Library, Powderhorn 930 Silos, 9840C/9940B/T10000 Drives, ACSLS software and a Sun E15K Server. The current structure for support includes a single Sun 4-Walls support covering both hardware and software through an 8(a)

4

AR 388

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED Filed 05/25/2007 Page 5 of 34 COPY MAY BE MADE PUBLIC

reseller. This maintenance strategy has been in place for approximately 4 years. Prior to that date hardware and software maintenance was done by the OEM. The decision to use the current maintenance structure was based on cost savings to the agency. There was a reduction in the cost of maintenance and a reduction in the cost of administration. Their systems support research and development projects/programs. Some of the issues/problems with this approach have been recurring administrative issues with the 8(a) reseller facilitating OEM recognition of items added to the contract. In addition, there has been some finger-pointing between the reseller and OEM issues.

ii. Karl Streed, ASC, stated "We contract with CSC who subcontracts with STK (SUN) for both hardware and software maintenance. CSC has looked at breaking out SUN maintenance several years ago but found that SUN would not allow anyone but SUN to do software maintenance. It was determined that the risk of having separate hardware and software maintainers was too high for critical storage systems. It was felt that SUN would be reluctant to provide proprietary information to any third party maintainer which could result in difficulty diagnosing hardware/software failures which would result in extended downtime."

iii. Debbie Thompson, ARL, stated "We currently contract all our maintenance through Raytheon. They contract with OEMs/third party vendors for the Sun, EMC, etc. equipment. As for STK (Sun) equipment, they obtain the maintenance

5

AR 389

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED Filed 05/25/2007 Page 6 of 34 COPY MAY BE MADE PUBLIC

for the hardware/software from Sun directly. They do not breakout the hardware/software separately".

b. GSA and Navy representatives contacted several federal agencies that had listed requirements with the Federal Business Opportunity website (FEDBUSOPS) in order to verify the complexity and scope of their requirements. i. Lisa Gossett, Department of Treasury Bureau of the Public Department, issues procurements on behalf of Department of Homeland Security. Lisa Gossett has a small customer base and the systems are being maintained by various software maintenance providers. All hardware and software maintenance requirements are on separate contracts and the services required are considered to be non-complex. There are no operational requirements that would impact the acquisition strategy of maintenance support. Their equipment is supported by internal resources and when there is a requirement for parts, a subcontractor would be called. Downtime does not have any negative impact on their agency.

ii.

Pamela Diliberto, Minerals Management Service (MMS)-SASC Procurement Office, stated MMS usually combines their hardware and software maintenance requirements. There is a requirement for maintenance providers to utilize only OEM products in executing repairs. Pamela Diliberto's technical point of contact is Kelly Griggs, an information technology specialist at MMS. Griggs stated the degree of system complexity for systems they maintained ranges from simple to highly complex. The hardware and software maintenance is segregated by

6

AR 390

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED Filed 05/25/2007 Page 7 of 34 COPY MAY BE MADE PUBLIC

function, type, and manufacturer. The current methodology is to combined hardware and software maintenance which has been the practice for the past eight years. The market research/factors considered by MSS in making their decision to use that structure included "vendor specific maintenance availability." Griggs stated "If a third party is able to provide the service without loss of warranty services or under a cooperative agreement between the third party vendor and the manufacturer then the third party vendors are considered in the competition. Third party vendors are considered for all contracts where the vendor is the reseller of the maintenance services by the manufacturer. Third party vendors are not considered where the manufacturer does not recognize the vendor as certified or approved for the warranty service being provided." Other factors considered when making an award determination include past performance and cost. However, "The primary factors are the ability of the vendor to provide OEM supported warranty and maintenance services, the ability of the vendor to provide the service in the time frames requested, and the ability of the vendor to provide additional consulting services as options to the maintenance contract". MSS depends upon the equipment and criticality of the function provided by the equipment. Some servers and functions require 99% uptime with a response time under 2 hours and others, not as critical, have a lower uptime and response time. The lowest requirement is 95% uptime with next business day response.

iii.

Griggs stated in the past MSS had third party vendors with no relation to the OEM. In these cases MSS found that if a component was determined to be

7

AR 391

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED Filed 05/25/2007 Page 8 of 34 COPY MAY BE MADE PUBLIC

defective and the third party vendor had performed any work on the box prior to the failure of the component, then the manufacturer often questioned the status of the part and delayed or denied replacement.

iv.

Christine Benavides of National Aeronautical Space Agency (NASA), stated she provides procurement support to the Ames Research Center for software maintenance. The initial procurement included both hardware and software, however, the agency elected not to maintain the hardware on a monthly basis. If the hardware is not functioning properly, a purchase requisition is issued for a vendor to come in and service the problem on a case by case basis. There is no significant impact to their agency operations if a particular piece of equipment or software is not functioning properly.

v.

Leslie Ervan with the Air Force Material Command Air Force Flight Test Center (AFMC/AFFTC) at Edwards Air Force Base (AFB), California stated their hardware and software maintenance are usually separated because they run systems which are defined as non complex. Though the requirements are separate, they typically go to the manufacturer unless there are authorized resellers who can provide OEM support. If the systems being maintained experience temporary downtime, it does not have a negative impact on the center. Air Force Flight Test Center has JT3 personnel, technical staff, who sometimes work with the vendors to provide maintenance on the various systems. For example, the vendor may coordinate a troubleshooting effort from an off-site

8

AR 392

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED Filed 05/25/2007 Page 9 of 34 COPY MAY BE MADE PUBLIC

location with the JT3 personnel executing diagnostics on-site. If the determination is made on-site repairs are required, the vendor's technician, not the JT3 personnel, would perform the repair.

c. GSA contacted several Sun Microsystems authorized Government System Providers (a.k.a. authorized Sun Resellers) in an effort to locate other government agencies that would have a requirement for STK/Sun hardware and/or software maintenance. Lists of federal agency points of contact were requested from Dynamic Systems Inc., Force 3 Inc., PSI Technology (Petrosys), and Red River Corporation. As of May 2, 2007, the points of contact were provided by Force 3, Incorporated and Dynamic Systems Incorporated. Several attempts were made to reach the identified points of contact, but individuals were either not the correct person, out of the office or unwilling to provide any information. Only one response was received from a Dynamic Systems Incorporated customer, Defense Information Systems Agency, Kimberly Rice.

i. Kimberly Rice is the Chief for Global Command & Control System ­ Joint (GCCS-J), Defense Information Systems Agency (DISA). They acquire SUN hardware and maintenance as a total package. The GCCS-J system is a Command, Control, Communications, Computer, and Intelligence (C4I) system, consisting of hardware, software, procedures, standards, and interfaces that provide a robust, seamless C2 capability. The system uses the Defense Information Systems Network (DISN) and

9

AR 393

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 10 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

must work over tactical communication systems to ensure connectivity with deployed forces in the tactical environment. ii. Kimberly Rice is also familiar with the Global Combat Support System (GCSS) which provides commanders with web-based access to selected Service and Agency authoritative/preferred logistics and transportation databases. Both are considered to be highly complex systems which include fairly significant tape libraries and uses one of the Defense Enterprise Computing Centers (DECCs) and its' configuration management repository. Both programs currently use an established Blanket Purchase Agreements (BPA) for maintenance support. BPA DCA200-20-A-5011 with Dynamic Systems offers extended warranty, maintenance, education and professional services for all Sun products. This would include maintenance for SUN hardware and software. Dynamic Systems is the prime contractor and OEM provides support as a subcontractor. Both programs have used the BPA since it was established in/around FY 2000 timeframe. Structure for the actual maintenance is complex, as both systems are classified to at least the secret level and require 24X7 support at locations around the world to include the DISA Central Command. Their service requirement is for 24X7, 2-4 hour on-site response time, platinum service. iii. The factors considered in deciding on the maintenance methodology was ease of use (both contractually and service related) and ability to meet the unique security requirements for both systems. Both programs have a

10

AR 394

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 11 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

long history of using Sun products; prior to the BPA being set up with Dynamic Systems the past performance for Sun and/or their reseller for these type of services was poor. Dynamic Systems has done an outstanding job of fixing the issues, establishing processes to address service, maintenance, tracking, etc., that has made the area of support easier. Cost was competitive to what DISA had experienced before and the BPA continues to be the best price/service for this type of support. iv. Contact was made with Lisa Odonnell, Contract Specialist, DISA Computing Services Directorate (CSD). Since approximately 2000, DISA has separated hardware and software maintenance on STK equipment. Exhibit A solicitation provided by CHE indicated DISA solicited their requirements for hardware maintenance only. However, section C.1. includes the requirement for software support if the software is an "integral and essential" part of providing hardware maintenance. Section C7 further states "hardware/operating system's software shall be maintained in good operating condition as defined by the standards established by the OEM". The NAVOCEANO MSRC requirement is for support of software that is considered to be "integral and essential" for operation of the hardware. The vendor is required to obtain and install replacement microcode, microcode engineering changes, microcode updates, and microcode upgrades. These allow the service provider to support the software integral to the system.

11

AR 395

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 12 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

v. Lisa Odonnell indicated CHE Consulting, Incorporate, had performed hardware maintenance on some of their IBM systems, however, to date, they have not performed maintenance on any STK/Sun systems. CHE Consulting, Incorporate, has agreements in place with IBM to provide support to resolve difficult repair actions that may occasionally arise that are beyond the capability of a third party vendor. To date, performance has been satisfactory. In reference to STK/Sun systems support, there has been one task order recently awarded to CHE Consulting, Incorporated, for hardware maintenance, however, the action has been placed on hold pending resolution of a protest filed by a competing vendor. The vendor has alleged CHE Consulting, Incorporated, cannot meet the requirement of providing OEM support since they have no agreement in place with STK. The current STK support is being provided by a third party vendor, who in turn, subcontracts to STK. vi. Additional discussions were held with William Eldridge, DISA CSD Contracting Officer Representative (COR), to further clarify the DISA STK maintenance requirements. DISA does have a separate order for support of the Automated Cartridge System Library Software (ACSLS) with STK. This is the software that controls an automated cartridge system. However, he is uncertain if DISA can productively use the Library Storage Managers without the ACSLS software. Since the current requirement is being protested, DISA has extended the support through Northrop Grumman, who in turn subcontracts to STK.

12

AR 396

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 13 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

vii. It is important to note that although DISA has elected to issue separate contracts for hardware and software, their risk is mitigated by also requiring a relationship with the OEM and support of software that is "integral and essential" to the operation of the hardware supported. Since CHE has not had an opportunity to perform any STK maintenance for DISA, their ability to support the DISA operational requirements is yet to be determined.

viii. Russell Goebel, Sandia National Laboratories, Infrastructure Computing Systems, provided information concerning their requirements. Sandia has three 9310's and one SL8500 with two more being installed. Additionally, there is a large number of distributed 9740 class libraries. For the past 15 years STK has provided hardware and software maintenance for all STK systems. Factors considered in developing their maintenance approach included quality of service, timeliness of service and turnaround time. In response to the question "What degree did cost factor in?" Russell Goebel replied "If you consider the loss in productivity and/or information that can result when using a third party supplier that can quickly outweigh the cost. We paid a lot for these boxes and the information contained therein is very valuable. I wouldn't seek 3rd party support for the STK devices anymore

13

AR 397

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 14 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

than I'd look for shade tree mechanic to work on a BWM, just to save dollars". Sandia's requirement is 99% availability. There have been no problems or issues with the current maintenance strategy, combination hardware and software.

d.

GSA and Navy representatives contacted another potential user of similar systems. Larry Reynolds, Storage Management Branch Chief, National Information Technology Center (NITC), United States Department of Agriculture, stated "The NITC operates two 9310 model silos. One unit is dedicated to mainframe operation, which includes virtual tape real tape drive part of the VT subsystem operation. The other unit is dedicated to Open System NetBackup operation. STK model 9840 tape drives are attached to both units. NITC negotiated the maintenance agreement with STK as part of the procurement and STK is the sole maintenance provider". The maintenance agreement which includes hardware and software has been in place since 1996. Factors NITC considers in deciding the maintenance approach included OEM service engineer training, OEM technical support, ability for remote facility diagnostic service, reporting, and engineer dispatching, OEM parts availability and depot management, continuous operation days between service, subsystem available to produce compared to lost revenue and defamed reputation, security clearances, 15 minute response, 2 hour resolution and an escalation plan, cost and performance. NITC operates 7 days X 24 hours X 365 days and requires a 99.9% availability for their systems.

14

AR 398

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 15 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

e. Navy representative Pete Gruzinskas contacted Sun to obtain a point of contact to identify other Sun customers to survey.

i. In response,

, Sun Federal ­ Services, provided a partial listing of

recovered customers who transitioned to CHE for service and then returned to Sun for service. The list includes customers, but it is understood the list is a included on the

partial listing and there are more recovered customers than the

list. Records for recovered service contracts is not one of Sun's normal processes and most of the list transitioned from STK. The decision of a customer to return to the OEM for maintenance could be for any number of reasons, including technology refresh and performance. advised the customers should

not be contacted without authorization from the customer point of contact via the respective Sun sales/service account team. Pete Gruzinskas's attempts to obtain the necessary authorization were not successful.

ii.

commented many of the contracts were through third party type

arrangements which make it difficult to identify technical points of contact who could provide meaningful information. The merger of STK and Sun is still in transition and the details for each customer identified is not available at this time. Recovered customers on the list included .

15

AR 399

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 16 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

iii. According to

, the Each system comprises multiple

serial numbers for library, control units, tape transports so the raw number of serial numbers under a service contract is exponentially higher. CHE has stated they currently maintain 74 hardware maintenance service contracts for tape library systems across 201 locations within the United States and four contracts in foreign countries, or 12% of the entire market segment within the United States. Mr. York's affidavit states CHE supports a total of 201 each 9310 Sun/STK robotic tape libraries. Based on the numbers provided by STK/Sun, it appears CHE has approximately 2% of the market or 201 .

6. GSA did a review of the small business capability to meet the combined requirements. GSA's (GSA's) E-Library. E-Library was reviewed for other potential vendors who provide both hardware and software support of STK/Sun equipment. Under Schedule 70, General Purpose Commercial Information Technology Equipment, Software, and Services, Schedule Identification Number (SIN) 132-34, software maintenance, there are over 1200 contractors that have the ability to provide software maintenance. Schedules SIN 132-12, hardware maintenance, includes over 1,000 contractors that can provide hardware maintenance. A random sampling of service providers for both hardware and software maintenance were contacted to verify their ability to support STK/Sun hardware and software.

a. Paul Marshall, Marzik Incorporated (GSA Contract Number GS-35F-5857H) confirmed Marzik is an authorized reseller of hardware and software maintenance

16

AR 400

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 17 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

for STK/SUN products. This is supported by agreements they have in place with United States Department of Agriculture National Information Technology Center, ALTESS (Army), NCI, and the Smithsonian. Paul Marshall confirmed Marzik provides hardware and software maintenance based on client needs. Typically, Marzik does not quote hardware and software maintenance as it is not a core business line for them; however, they will submit quotes upon request from various clients. Marzik is capable of providing a quote for hardware and software maintenance regardless if the equipment is purchased through them. Paul Marshall stated that Marzik does not have the technical expertise in house and subcontracts their service out to the original equipment manufacturer (OEM) which in this case is STK/SUN. Marzik is a small business, 8(a).

b. Tammy Byrd and Mark Stafne, Commercial Data Systems (CDS) provided a list of references/customers to whom they provide hardware and software maintenance support for STK robotic tape libraries. CDS, a small disadvantaged 8(a) vendor, has established hardware and software contracts with Scott Air Force Base ($2.8 M), Global Star, Incorporated ($600K), Sandia National Laboratories ($1M) and Eclipse Aviation ($275). Mark Keifer, an engineer for Sandia National Laboratories, was contacted by GSA on 04/26/2007 to discuss their requirements for STK/SUN maintenance support. The information he was willing to provide was limited, but did confirm Sandia runs a "just in time" operation. Their maintenance contracts are established at the corporate level so he had limited knowledge regarding the selection of the service provider.

17

AR 401

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 18 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

c.

.

d. Published prices were obtained for a configuration similar to the NAVOCEANO MSRC requirement from vendors with FSS offerings. Prices were provided by Logical Choice Technologies (GS-35F-0519M), Dynamic Systems (GS-35F-0209R), and Marzik, Incorporated (GS-35F-5857H). All are small businesses and have confirmed their ability to provide hardware and software maintenance support for STK/Sun equipment.

7. As concerns FAR Subpart 7.107, Acquisitions Involving Bundling, the structure of this acquisition does not meet the definition of "Bundling" as defined in FAR Subpart 2. Historically, NAVOCEANO MSRC STK maintenance requirements have been solicited and awarded as total solution that included hardware and software maintenance. Therefore the

standards in FAR Subpart 7.107 for Acquisition Planning are inapplicable to this acquisition. FAR 7.107 is concerned with the potential impact of bundling (as defined in FAR Subpart 2.101) on small businesses. The bundling argument made by CHE as best I can tell is directed towards the allegation that the combining or bundling of the hardware and software requirements is unduly restricting competition and not consistent with Competition in Contracting Act. Based on the above market research, there are a number of small business vendors who can provide

18

AR 402

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED COPY Filed 05/25/2007 Page 19 of 34 MAY BE MADE PUBLIC

AR 403

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 20 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

COST BENEFIT ANALYSIS An analysis was performed in order to assess the potential costs and benefits to the Naval Oceanographic Office (NAVOCEANO) Major Shared Resource Center (MSRC) that would likely result from either a consolidated or split approach to satisfying the MSRC's requirements for hardware and software maintenance of Storage Technology/Sun (STK/Sun) systems. Assumptions a. The level, quality, and performance requirements for hardware and/or software maintenance would be the same for either the total solution or split maintenance approach. b. The cost of software maintenance would remain the same in either scenario. Therefore, the analysis does not include the cost of software maintenance. Analysis Costs: General Services Administration (GSA) Federal Supply Schedules (FSS) were researched to determine the available published price for the annual cost of hardware maintenance. The review consisted of a random sampling of equipment from the NAVOCEANO equipment list and obtaining published prices for monthly maintenance. The terms and conditions of each maintenance package were reviewed to ensure a valid comparison. Attachment A is a summary of the published prices for comparable service for key components of the system. In addition, attachment A includes the purchase price obtained from NAVOCEANO's Asset Manager inventory records. Historically, the MSRC has utilized % of the equipment cost per year as a fair and reasonable cost estimate for maintenance of systems. The FSS published annual unit prices outlined in Attachment A for hardware maintenance are approximately 1% to 19% of the equipment costs. Vendors A through D's prices appear to be in line with prices obtained from the OEM for the same support. Compared to the other prices, Vendor E's (CHE's) pricing appears to be substantially lower than expected. Vendors A through D's annual maintenance prices range from 5% to 19% of the equipment cost. Vendor E's prices range from 2% to 8% of the equipment cost. Based on the information available, it is difficult to determine if the price offered by Vendor E is realistic for the required services. There are many unknown factors that could have an impact on the pricing, i.e. use of original equipment manufacturer (OEM) parts to complete repairs, availability of trained service representatives in the local area, ability to engage the OEM to assist in resolving complex technical problems, the number of parts in inventory, the amount of risk a vendor is willing to assume, etc.

AR 404

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 21 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

For the purpose of this analysis, information obtained as a part of the government's supplemental market research was used to compare the potential costs of a consolidated approach to the potential costs of a split approach. Utilizing a random selection of model numbers and quantities from the NAVOCEANO requirement, published maintenance prices from federal schedules of STK (OEM) and CHE were compared. (See TABLE A below). Mr. Brad Comes, Shared Services Project Manager for the Department of Defense (DOD) High Performance Computing Modernization Office (HPCMO) and Tom Dunn, Director of NAVOCEANO MSRC, provided information for the analysis to assist in quantifying the costs to the MSRC and its customer agencies in the event of systems failures. Particularly, when systems are down, there are three main areas where measureable losses occur: 1) Loss of opportunity to use the investment, i.e., the investment is aging and system users are missing the opportunity to use systems during this aging process. MSRC high performance compute engines have a planned systems life of 4 years. 2) Loss of the operating costs. The experts running the systems are present but unable to effectively deliver a service due to system failure. This is the cost of the human capital to run the systems. The operating costs for the NAVOCEANO MSRC are $15 million a year. 3) Loss of the end users' ability to make progress on their projects. It is estimated at any point in time that approximately 40 scientists and engineers are disrupted for each unavailable HPC System. A single MSRC is composed of many HPC systems. Average loaded cost per year per scientist and engineer is estimated to be $150,000 per year. See TABLE B for estimated daily losses that would result from an outage of a system. Results: TABLE A Model 9840C 9940B 9330LMU LMS9310 MD04 D178 SNS 4524 QTY 10 33 5 9 2 2 -------------Annual Cost--------Consolidated Split $17,395 $9,000 $53,456 $28,116 $16,612 $ 9,900 $125,739 $69,228 $ 2,586 $ 480 $ 3,289 $ 1,320 $118,044

Total---------------------------$219,077

AR 405

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 22 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

Utilizing the potential price difference for a consolidated and split approach outline in TABLE A, it is estimated by splitting the requirement there would be a 50% cost savings. In the case of solicitations 4THG17078067 and 4THG17078068, the Government estimated annual cost for hardware maintenance is a total of , based on past experience and a review of the FSS price lists, TABLE A. It is projected that the cost for a split solution would be half (50%) or , reflecting a potential cost savings of annually or per day. Stated another way, based on a comparison of the annual unit prices in Attachment A, it is projected the cost for the selection of a consolidated approach, instead of a split approach, would double the cost of hardware maintenance. (Reference vendors A through D pricing compared to Vendor E.) TABLE B Loss 1 (Investment Opportunity Loss): MSRC invests $40 million every four years in replacing high performance compute engines. These systems have a planned systems life of 4 years. Based on a $40 million dollar investment, the capital loss for each day of downtime is approximately $27,397 per day. Loss 2 (Operating Costs Loss): The NAVOCEANO MSRC operating cost is $15,000,000 per year. This reflects an operational cost of $41,667 per day. Loss 3 (End User Operations Loss): An average of two systems per center is 80 scientists and engineers per center at $150K per year reflects a cost of $33K per day. $27,397 + $41,667 + $33,000 = $102,064 estimate total cost to the program (tax payer) for each day of system outage at the NAVO MSRC. In summary, the potential cost savings to the NAVO MSRC associated with splitting the maintenance requirements would be annually. The estimated losses that would result from an outage of a system for a 24 hour outage would be $102,064 (for 1 day's downtime). If the center is down more than in a year, the loss to the center would exceed any potential cost savings of a split approach. Systems are most vulnerable when something changes on the system, i.e., installation of an upgrade, change in configuration by addition of a system, or other planned changes. Once the action is completed there is risk the system will not come back up within the planned outage timeframe, which is usually two hours. It is estimated, within the NAVOCEANO MSRC, changes will be made to the STK systems on an average 6 to 8 times a year. Each time a change occurs there is opportunity for system failure or extended outages. This risk would increase if the hardware and software support were provided by separate vendors.

AR 406

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 23 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

AR 407

Case 1:07-cv-00055-TCW

Document 36

Filed 05/25/2007

AGREED-UPON REDACTED COPY Page 24 of 34 MAY BE MADE PUBLIC

Attachment A ANNUAL MAINTENANCE PRICE Vendor B Vendor D Vendor E PURCHASE VENDOR QTY PRICE A B C D E cost cost cost 1 $23,517.00 $1,908.00 $2,198.16 $1,788.00 $1,739.52 $900.00 9% 7% 4% 1 $28,655.00 $0.00 $2,198.16 $1,884.00 $1,619.88 $852.00 8% 6% 3% 1 $24,544.00 $0.00 $4,548.48 $5,604.00 $3,322.32 $1,980.00 19% 14% 8% 1 $85,212.00 $1,382.40 0 $1,368.00 $1,089.00 $1,398.96 0% 1% 2% 1 NSP $23,615.52 0 $23,352.00 $18,603.00 $0.00 #VALUE! #VALUE! #VALUE! 1 $5,416.00 $0.00 $467.76 $504.00 $316.68 $144.00 9% 6% 3% 1 $241,594.00 $0.00 $16,272.96 $16,848.00 $11,882.04 $6,996.00 7% 5% 3% 1 $19,360.00 $0.00 $2,865.84 $2,652.00 $2,089.20 $0.00 15% 11% 0% 1 $44,977.00 $1,108.20 $0.00 $1,092.00 $897.12 $480.00 0% 2% 1% $696.96 $703.08 $552.00 $521.40 $144.00 11% 8% 2% 1 $6,529.00 1 $27,422.00 $2,045.04 $1,830.12 $0.00 $0.00 $0.00 7% 0% 0% 1 $11,000.00 $319.92 $0.00 $0.00 $519.00 $0.00 0% 5% 0% 1 $14,406.00 $3,073.32 $0.00 $0.00 $2,018.76 $660.00 0% 14% 5%

MODEL 9840C 9940B 9330 LMU SFC512M 6140 4420-001 LMS9310_9311 9311 MD04 D178_280 D200-014 M03 D178 002 9170-014 SNS 4254

DESCRIPTION Drive Drive Library Mgmt Unit Network Switch Intrepid FC Director Pass Thru Port Silo, Control, Drive Cab Library Control Unit Storage Array Drive Module Storage Array Drive Module 24 Port Switch

AR 408

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 25 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

IN THE UNITED STATES COURT OF FEDERAL CLAIMS CHE CONSULTING, INC., Plaintiff, v. THE UNITED STATES, Defendant, and STORAGE TECHNOLOGY CORPORATION, Defendant-Intervenor. DECLARATION OF I, PETER P. GRUZINSKAS, pursuant to 28 U.S.C. § 1746, do hereby declare: 1. The Naval Oceanographic Office (NAVOCEANO) became a Navy supercomputing center in 1990 and a Major Shared Resource Center (MSRC) in 1994. During the time between 1993 and 2003 the NAVOCEANO MSRC was listed as the 10th most powerful high performance computing center in the world. The Storage Technology Corporation (STK) hardware/software utilized is a complex robotic tape silo known as a Powderhorn 9310. It consists of hardware, software, and firmware (software embedded on hardware) that controls the migration of data to and from NAVOCEANO's MSRC. The high performance computing environment which this subsystem supports consists of several IBM supercomputers which have a theoretical peak of over 60 trillion floating point operations per second. Far more critical than the sheer computational power fielded by this MSRC is the mission we are tasked to accomplish with these resources. These include but are not limited to: Support to ongoing operations -- Support for the NAVOCEANO Warfighting Support Center (WSC) which provides time-critical modeling and simulation which generates environmental support products provided directly to Navy warfighters deployed worldwide; Technology to solve the most computationally intensive problems faced by today's scientists and engineers in support of the Department of Defense; Storage/archival of a vast collection of accumulated static oceanographic and geophysical data which was determined (and published) to be "priceless" and a "National Treasure" by an elite consortium of international scientists; and, Disaster Recovery for the operational product generation of NAVOCEANO, as well as five other similar high performance computing centers. 2. The systems we support and the infrastructure required to maintain our operational posture (7 days a week, 24 hours a day, 365 days a year) are some of the most complex in

AR 409

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 26 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

existence and highly depended on for continuous operational readiness. These systems must run constantly. 3. Before, during and after Hurricane Katrina, one of the worst natural disasters to affect this nation, the NAVOCEANO MSRC ran uninterrupted, as the eye of the storm passed directly over us. I was here, at the center, at that time. 4. It has been this Center's policy to engage the Original Equipment Manufacturer (OEM) or an OEM authorized service provider to support both hardware and software during systems lifecycle. The OEM designed the systems, they own the proprietary rights to various software components and diagnostic routines, and they are intimately familiar with the interaction between hardware and software. This point cannot be overemphasized. Hardware, as a practical matter, is useless without the associated software component whether it is a hardware driver, an operating system, a file system, etc. Hardware and software cannot be separated effectively to achieve our desired results. NAVOCEANO MSRC requires the service provider, if not the OEM, to have an established formal working relationship with the OEM to ensure the service provider is cognizant of latest developments in hardware, software, and firmware relative to the overall system. This relationship is essential for the service provider to be best prepared to respond to our operational needs. 5. It is my understanding there is proprietary diagnostic routines used by STK that are resident on the system components and these routines are not available to 3rd party vendors without a formal OEM agreement. 6. Since NAVOCEANO MSRC has historically acquired hardware and software support as a package from the OEM or an OEM authorized service provider, we are unable cite specific instances where separate hardware and software maintenance contracts resulted in delays or unanticipated outages. The following "real world" example is provided to illustrate the complexity of our systems and the great challenge of maintaining their operation. This challenge is present even when one vendor is responsible for providing both hardware and software support. 7. NAVO MSRC maintains several large IBM supercomputers that perform the computational work for the center. These machines have thousands of processors, massive memory partitions, and hundreds of disk comprising multiple Terabytes of storage. They are the workhorse of the center and generate on an average about 20 to 30 Terabytes of new data each week which is stored on the STK mass storage systems. 8. In June of 2006, our operations staff noticed file corruption errors on the IBM console. In accordance with standard operating procedures, the operators called IBM and an IBM engineer was quickly dispatched (IBM's contract has a two hour response time). Diagnostics ensued and the initial thinking was that the culprit was the file system software, an IBM product called General Parallel File System. A patch was scheduled and system downtime (in addition to the diagnostic downtime) was suffered. These actions did not correct the problem.

2

AR 410

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 27 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

9. More diagnostics (and downtime) followed and the results were sent to IBM's main production facility in Poughkeepsie, NY, for analysis. Meanwhile the logs were searched for any external activity, such as hardware replacement, on the system. The system was constantly monitored and the errors in the file system continued. If the files being generated are corrupt, the system is useless and compute cycles are being wasted. The case was elevated within IBM to a Critical Situation within a couple of days and there were daily teleconferences with IBM experts from all over the country. IBM software engineers were literally working around the clock trying to solve the problem. 10. It was noted that a disk was replaced the day before the problem started but IBM routinely replaces hard drives on our systems without incident. The firmware level was verified as supported under our hardware configuration. Several diagnostic sessions later, the cumulative downtime was approaching a level that would dictate concessions from IBM to the government because IBM was not achieving the 97% system availability as required by their contract. Almost two weeks had passed and 16 or more hours had been lost to diagnostics and prospective fixes. Someone caught the fact that the disk replaced prior to the errors was manufactured by Hitachi. All the other disks in the system were made by Seagate, and had various levels of firmware on them dating back two or three versions, which was acceptable to the OEM. On-site IBM field engineers suggested updating the firmware on all the disks in the computer. 11. Upgrade of the firmware corrected the problem. The IBM system with the corrupted files was going offline periodically during this time, and the MSRC compensated by using other available systems to meet the need for critical model runs and product generation to support the operational Navy. To my knowledge, to this day IBM does not know and cannot explain why this happened. This problem was difficult for a single vendor, the OEM, and could have been further complicated had separate vendors been responsible for either the hardware or the software. It is anticipated separate contracts would have resulted in additional delays and obstacles while service providers attempted to interpret and diagnose the source of multiple symptoms. While the center maintains multiple high performance computing systems, they all rely on a single file system to store their data. The STK hardware is an integral part of this file system. 12. Complex service requirements such as the one described above are not unusual. Error messages can be very generic and diagnosing the source of the problem can be difficult. Similar situations have occurred on our SUN file servers which act as the front end to our STK storage silos. Again, I want to emphasize the complexity of these systems, as there is an approx 60TB Storage Area Network between the file servers and the Storage Area Network (SAN). All of these switches, controllers and disks have imbedded firmware and if these various components are not at the correct revision level then system problems will occur. While problems like this may not happen every week or month, when they do occur, the time required to diagnose and correct is critical. Calling multiple vendors every time an error message is received is not practicable.

3

AR 411

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED COPY Filed 05/25/2007 Page 28 of 34 MAY BE MADE PUBLIC

AR 412

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED COPY Filed 05/25/2007 Page 29 of 34 MAY BE MADE PUBLIC

AR 413

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED COPY Filed 05/25/2007 Page 30 of 34 MAY BE MADE PUBLIC

AR 414

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON REDACTED COPY Filed 05/25/2007 Page 31 of 34 MAY BE MADE PUBLIC

AR 415

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 32 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

AR 416

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 33 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

AR 417

Case 1:07-cv-00055-TCW

Document 36

AGREED-UPON 34 of 34 Filed 05/25/2007 PageREDACTED COPY MAY BE MADE PUBLIC

AR 418