Minutes from TCT Meeting:
Wednesday, December 8, 2004, 12:45P-1:45P

Members Present:
Chrystle Hall, Cathy Hart, Kurt Hill, Mary Lou Mosley, Jim Patterson, Rick Sheets, Jon Storslee, Casey Durandet

Guests:
Sam Fraulino, Darrel Huish, Loretta Mondragon, Jennifer Strickland

Issues/items discussed:

We had a special meeting today with Darrel Huish from District - Associate Vice Chancellor for Information Technologies.

Things we discussed with Darrel:

1) Darrel gave us a rundown on the circumstances involved with the crash of BlackBoard. I quote him from the bb status page (12/02/04):

"On Monday (last week), Maricopa Community Colleges experienced a severe disk storage
outage which impacts both our MIDAS and Blackboard course management
systems. In brief, a significant hardware failure occurred in the components
that manage data storage, which corrupted the data in the data bases. As soon
as the outage occurred, we began looking at options to recover the data and
restore operations. Unfortunately, the failing components have been very
difficult to identify so they can be replaced. Also in one instance an
incorrect part was shipped to us by the vendor. In short, nearly anything
that could go wrong has gone wrong. Information Technologies Services (ITS)
recognizes that the length of the outage is not acceptable, and we have been
employing all available resources to restore these services. We expect the
hardware to finally be fixed Thursday evening or Friday morning. To further
complicate the situation, once the hardware is fixed, the data in these systems
needs to be restored. We have reliable data backups from November 12th, but
regrettably we have doubts as to the complete availability of reliable data for
the period from Nov. 13th through Nov. 29th.

As of this writing, we are planning to have hardware ready Friday morning. We
have also engaged external data recovery specialists so we can rebuild as much
of the available data as possible by end of day on Friday. Our plan is to bring
up the Blackboard environment for viability testing over the weekend, and have
the system available to all on Monday. Please be advised that until we know
whether we have experienced a loss of data, it is hard to advise faculty and
students how to adapt. Good strategies for now would be to preserve any paper
documentation that may assist in manually rebuilding a record of the activities
from Nov 13th to 29th. In the event that the data cannot be recovered in an
automated fashion, ITS will make maximal efforts to create partial reports from
the damaged data to assist a reconstruction of the course activities.

As much effort as is going into restoring these services, even more effort and
resources will go into learning and fixing the combination of problems that have
led us to this catastrophic system outage. To say that we apologize is to
understate our sincere regret that this has occurred, and our resolve to do
better in the future. We will make more information available as soon as the
situation become clearer, which should be by 8pm Thursday evening."

2) Other discussions with Darrel centered on how we can optimize communication between the colleges and District. A suggestion was made to increase the communication links re:, e.g., the status of bb, etc., to IT personnel on campus so that they can disperse the information accordingly.

3) Rick mentioned that there is a growing need to have 24/7 staffing available for full operations. We are at that stage right now. We need to look at how the summer schedule could be changed to include Friday and Saturday coverage.

Darrel mentioned the shift culture - implementation of B and C shifts; better communication channels; more staffing - there needs to be an increase of the number of doers on the staff, not just one particular individual to cover too many areas.

4) New Student System - Chrystle is working with faculty on issues relevant to their needs, e.g., training (trainers need to be involved throughout the whole process, not at the end).

5) Bandwidth issues - We're at 100% utilization right now. What network impact will NSS cause to other systems? Cathy said that the bandwidth will increase by 3M (she said it should be more). Carol, do you know? The bigger schools will have 5M.

6) The main points we discussed at the meeting: there needs to be better communication between the campuses and district, reduce the isolation effect, more staff training, more campus exposure.

We're off to a great start in our dialog with Darrel. Many thanks, Darrel, for taking your time to visit with us!

Please visit Darrel's web page at:
http://www.public.asu.edu/~huish/

The next TCT meeting is scheduled for Wednesday, January 26, 2005, 12:45P-1:45P in the G-conference room (G126). Have a great holiday break!

This "Meeting Report" is my interpretation of what occurred during this meeting. If you would like to correct an error(s), make an addition(s), or have any questions, please call me @ 7-6651 or e-mail me.

Date: Wednesday, December 8, 2004
Recorder: Casey Durandet

--
Casey Durandet, Ph.D.
PVCC - Physics Department
602-787-6651
http://www2.pvc.maricopa.edu/ms/phy