Wholesale: Products & Services

Archived System CR SCR111203-01ES Detail

Within 24 hours (1 business day) of the event:

Qwest will send or post to a web location a preliminary OSS incident report. This report should contain the following information.

Incident ID:

Incident type: degradation OR outage ; degradation AND outage

Confirmed start date of the incident (incident is from the first alarm or report until the trouble is verified as cleared. Within an incident may be periods of degradation and periods of outage)

Confirmed start time of the incident

Confirmed identification of the functionality that was affected by the incident. For example, Order only, or Pre-Order AVQ and CSI and Loop Qual.

Confirmed end date of the incident

Confirmed end time of the incident

Total Minutes of Outage for this event: (outage as defined by performance measures)

Total Minutes of Degradation for this event:

Corrective Actions taken to clear the incident:

Initial assessment of root cause:

Within 7 business days of the event:

Qwest will send a final OSS incident report. This report should contain the following information.

Incident ID:

Incident type: degradation OR outage ; degradation AND outage

Confirmed start date of the incident (incident is from the first alarm or report until the trouble is verified as cleared. Within an incident may be periods of degradation and periods of outage)

Confirmed start time of the incident

Confirmed identification of the functionality that was affected by the incident. For example, Order only, or Pre-Order AVQ and CSI and Loop Qual.

Confirmed end date of the incident

Confirmed end time of the incident

Total Minutes of Outage for this event: (outage as defined by performance measures)

Total Minutes of Degradation for this event:

Corrective Actions taken to clear the incident:

Final assessment of root cause:

Actions taken to clear root cause: If the actions are future planned actions, like ‘new memory ordered and installation is planned for 10/29/03’ then a follow-up date must be indicated.

Next Follow-Up Due Date: Qwest to provide updated status until the actions required to correct the root cause have been tested and implemented.

Ideally, this information could be stored and sent or posted to a web location in a running Excel spreadsheet. If posted to a web location, the URL should be a consistent or ‘absolute’ URL and not a URL that changes. This will enable CLECs to easily develop scripts to download the file on a regular basis.

In managing the availability of the components of our OSS systems we have clearly identified the need to have specific incident data from all our OSS interfaces, both internal and external. We look forward to working with Qwest and the CLEC community to develop a standard process.

Expected Deliverables/Proposed Implementation Date (if applicable):

To be provided the requested pieces of information for root cause analysis on OSS Systems Outages

Status History

Project Meetings

Connie Winston/Qwest stated that this CR has been escalated by AT&T and will be worked outside of the CMP Meetings.

2/19/04 CMP Systems Meeting

Donna Osborne-Miller/AT&T stated that AT&T was requesting notification of root cause analysis and that Qwest’s response needs elaboration. Donna stated that when they asked for Incidence type Qwest responded that the request was outside of the negotiated CMP requirements and would require a revisitation of the agreement. Donna asked what part of the agreement in the CMP document did she need to review. Donna said that Qwest could respond to that question later. Donna said that if the language needs to be modified she needs to know. Donna stated that Qwest feels what they have accomplished on this issue with Documentation Global Action Item is sufficient and AT&T disagrees. Connie Winston/Qwest stated that the disagreement is around outages and that we will have an update to that later today. Connie said that this issue is specific to downstream outages. Donna Osborne-Miller/AT&T said that she agrees and that they want initial and final assessment on root cause. Donna stated that Qwest should have information about what went wrong and how to remedy. Donna also stated that Qwest believes the information is already provided. Donna stated that AT&T believes the information in insufficient and needs detail. Donna stated that Qwest needs to look at this issue from AT&T’s perspective and that their clients need detail. Connie Winston/Qwest said that this is specifically associated to downstream OSS issues. Connie stated that she did not understand how sharing root cause of a system outage helps AT&T better run their business. Connie said that, for example, there is a lot of technical detail on a network connectivity failure and that she did not understand the value in providing the CLEC with that detail. Connie stated that we could have an off-line meeting to discuss with AT&T’s partners and IT technical people. Donna Osborne-Miller/AT&T stated that would work for AT&T. Phyllis Burt/AT&T said that in the December meeting minutes AT&T was to provide an example (from SBC) of the detail they were looking for. Donna Osborne-Miller/AT&T stated that the example was not sent because the event notification from Qwest included more detail. Connie Winston/Qwest stated that we will work together to better understand how root cause will help AT&T run their business. Connie also said that in the future when we refer to document changes we need to be clearer as to what section we are referring to.

12/17/03 CMP Systems Meeting

Donna Osborne-Miller/AT&T stated that the clarification call was held and that Qwest is still evaluating this CR.

12/8/03 Additional Clarification Meeting Held

Attendees: Donna Osborne-Miller - AT&T, Terry Craig - AT&T, Leo Dimitriadis - AT&T, Kay Harris - AT&T, Rich Bergman - AT&T, Trent Weaver - Qwest, Bonnie Johnson - Eschelon, Kim Isaacs - Eschelon, Randy Owen - Qwest, Christy Turton - Qwest, Lynn Stecklein - Qwest

Review Description of Change Lynn Stecklein - Qwest recapped the description of the request for the participants that did not attend the November 20 Systems Meeting.

Discussion: Trent Weaver - Qwest asked how AT&T would use this report. Leo Dimitriadis - AT&T stated that this report would provide AT&T from Qwest's perspective information and root cause analysis on outages etc. Rick Bergman - AT&T stated that Verizon has provided AT&T with this type of information in the past and that they have made a lot of strides in partnering with them, for example, on response time. Trent Weaver - Qwest asked AT&T what type of information they were looking for and/or what type of dialogue would take place if for instance if machines have gone down. Rick Bergman - AT&T stated that SBC would notify them of issues and/or anomolies associated with the problem, Trent Weaver - Qwest asked if AT&T had an example of the agreement they have with Verizon/SBC. Rich Bergman - AT&T stated that there was no set agreement or forrum and that they have open dialogue when necessary. Randy Owen/Qwest stated that we have had a lot of discussion lately on event notifications and that we have added information to the event notification Kay Harris/AT&T asked if this would impact order and pre-order. Randy Owen/Qwest stated both. Trent Weaver/Qwest asked if AT&T could send an example on what type of information and root cause analysis Verizon/SBC was providing. Donna Osborne-Miller - AT&T stated that she would send an example.

Identify/Confirm CLECs Expectation AT&T would like to do root cause analysis with Qwest in those situations where OSS outages occur.

Establish Action Plan AT&T will provide an example of the root cause analysis Verizon is providing on outages.

11/20/03 CMP Systems Meeting

Donna Osborne-Miller/AT&T reviewed this CR as a walk on. AT&T is requesting the following: AT&T would like to do root cause analysis with Qwest in those situations where OSS outages occur. AT&T is proposing that Qwest should provide to the CLEC community, the following: Within 24 hours (1 business day) of the event: Qwest will send or post to a web location, a preliminary OSS incident report. This report should contain the following information. Incident ID: Incident type: degradation OR outage; degradation AND outage Confirmed start date of the incident (incident is from the first alarm or report until the trouble is verified as cleared. Within an incident may be periods of degradation and periods of outage) Confirmed start time of the incident Confirmed identification of the functionality that was affected by the incident. For example, Order only or Pre-Order AVQ and CSI and Loop Qual. Confirmed end date of the incident Confirmed end time of the incident Total Minutes of Outage for this event: (outage as defined by performance measures) Total Minutes of Degradation for this event: Corrective Actions taken to clear the incident: Initial assessment of root cause: Within 7 business days of the event: Qwest will send a final OSS incident report. This report should contain the following information. Incident ID: Incident type: degradation OR outage; degradation AND outage Confirmed start date of the incident (incident is from the first alarm or report until the trouble is verified as cleared. Within an incident may be periods of degradation and periods of outage) Confirmed start time of the incident Confirmed identification of the functionality that was affected by the incident. For example, Order only, or Pre-Order AVQ and CSI and Loop Qual. Confirmed end date of the incident Confirmed end time of the incident Total Minutes of Outage for this event: (outage as defined by performance measures) Total Minutes of Degradation for this event: Corrective Actions taken to clear the incident: Final assessment of root cause: Actions taken to clear root cause: If the actions are future planned actions, like ‘new memory ordered and installation is planned for 10/29/03’ then a follow-up date must be indicated. Next follow-up Due Date: Qwest to provide updated status until the actions required to correct the root cause have been tested and implemented. Ideally, this information could be stored and sent or posted to a web location in a running Excel spreadsheet. If posted to a web location, the URL should be a consistent or ‘absolute’ URL and not a URL that changes. This will enable CLECs to easily develop scripts to download the file on a regular basis. In managing the availability of the components of our OSS systems, we have clearly identified the need to have specific incident data from all our OSS interfaces, both internal and external. We look forward to working with Qwest and the CLEC community to develop a standard process. Donna Osborne-Miller/AT&T stated that the requested pieces of information are for root cause analysis on OSS Systems Outages and that is a national request. Liz Balvin/MCI said that this request sounds like the Event Notification root cause analysis that is parallel to the document project plan. Connie Winston/Qwest stated that we would not be doing any that we would not consider as events. Connie stated that we would look at a delay in the FOC but not look at anything such as our system thresholds for example. Liz Balvin/MCI said that Qwest needs to notify the CLECs of scheduled maintenance. Connie Winston/Qwest stated that we do notify of CLEC impacting maintenance that occurs during off-hours, she noted that we generally try to do maintenance during normal off-hours. Liz Balvin/MCI stated that you do not notify if maintenance is done off-hours and we would like to be more informed. Judy Schultz/Qwest asked wouldn’t the CLECs only want to be notified if it impacted them. Judy said that if the CLEC’s were impacted they would be notified. Liz Balvin/MCI said that the CMP Document says that you notify even when off-hours. Connie Winston/Qwest stated that this is not an outage. She explained that the purpose of off-hours is to perform maintenance to maintain the health of the system. She noted that if we patch anything that is CLEC impacting, regardless of if it’s during off-hours or scheduled maintenance, Qwest issues an event notification. Liz Balvin/MCI said that makes sense. The status of this CR will be changed to presented.

CenturyLink Response

March 22, 2004

Donna Osborne-Miller AT&T

Dear Ms. Osborne-Miller:

This letter is in response to your March 15, 2004 escalation regarding Qwest’s denial of SCR111203-01, Request for information regarding any unplanned outages”.

AT&T has requested that Qwest provide more detailed information and analysis of degradations and outages than what Qwest currently provides in the Production Support - Event Notifications. AT&T has also requested an OSS incident report or spreadsheet be created and provided via Qwest’s external Wholesale website. This report or spreadsheet would detail Qwest’s final assessment of root cause and the corrective actions taken to clear the incident.

In the original denial of this AT&T request, Qwest stated that a gateway availability Performance Indicator Definition (PID) exists which measures the quality of CLEC access to Qwest gateways and associated systems. Root-cause analysis, when appropriate, would be included with the information for this PID. Qwest’s performance results can be found at the Qwest Wholesale website at: http://www.qwest.com/wholesale/results/index.html.

Additionally, Qwest has improved the quality of unplanned outages information provided in the Event Notification process based on feedback provided by AT&T and other CLECs in the Qwest sponsored CLEC Documentation Forum.

AT&T’s escalation of Qwest’s denial states that; “While the collaborative work from the Documentation Forum has accomplished areas of improvement relative to the content of the event notification, AT&T believes that Qwest does not fulfill the requirement to provide to CLECs detailed root cause analysis of unplanned degradations/outages nor detailed final corrective actions taken as part of the root cause analysis.” AT&T also states it is not asking Qwest to divulge or compromise that which is proprietary to their (Qwest) systems.

Qwest understands and respects AT&T’s need to have reliable access to Qwest’s OSS interfaces for the purpose of transacting business. Qwest has numerous systems and business processes in place to ensure reliable access for AT&T and all CLECs. When an outage does occur, the event notification that is distributed provides the date and time of outage, and provides details of the known business impacts in addition to the resolution for the particular Production Support issue. Qwest does not believe that details of Root Cause Analysis (RCA) on most of these outages would provide the CLECs any additional business value since these details may be specific to Qwest’s back-end systems which the CLECs do not directly access. Providing a Qwest RCA in these instances would provide little opportunity for the CLECs to make changes and/or perform RCA on their side to mitigate any outages in the future. Qwest is unaware of any requirement to provide detailed root cause analysis of unplanned outages as stated in the escalation language. In the February Monthly Systems CMP meeting, AT&T expressed a desire to hold a meeting to discuss how RCA could provide business value to AT&T, and Qwest agreed to arrange that meeting. Subsequently, this meeting was declined by AT&T.

Sincerely,

Connie Winston Director-Qwest Information Technologies

2/13/04 Final Response

February 13, 2004

For Review by CLEC Community and Discussion at the February 2004 CMP Meeting

Donna Osborne-Miller AT&T

SUBJECT: Qwest’s Change Request Revised Response - SCR111203-01Request for information regarding any unplanned OSS Event Qwest experiences”

This request asks for an event notification to be sent or posted to a Web site that captures the items in the table below. Further, AT&T requests that Qwest send a follow-up report within seven (7) business days of the event that captures the items in the table presented in boldface.

Qwest is conducting its review and analysis of existing Web site functionality around event notification reporting.

Qwest maintains that: a.) all items indicated in the table are already captured in the event notification itself, or can be derived from the information currently provided in the event notification, b.) Qwest is providing all information required per the CMP process. Qwest maintains that the reporting is redundant to the information already provided in the event notifications themselves. In addition, the maintenance of the Web site with the information being requested above would require an additional resource commitment of 20 hrs/week. Qwest is therefore denying this change request based upon two factors. The information requested is duplicative to information already being provided and has no reasonable demonstrable business benefit.

Summary Comparison Table: AT&T Request against Qwest Event Notification Memo

AT&T Requests Qwest Response Incident ID: The Event Notification number and the Trouble Ticket number associated with the issue are identical. Already captured in the Event. Incident type: degradation OR outage ; degradation AND outage. This request is outside of the negotiated CMP requirements, and will require a revisitation of the agreement. Confirmed start date of the incident (incident is from the first alarm or report until the trouble is verified as cleared. Within an incident may be periods of degradation and periods of outage) The Event Notification captures the start date and time of the incident from the time when Qwest identifies the issue as multi-CLEC impacting. This is in compliance and consistent with the CMP requirements. Confirmed start time of the incident See above. Confirmed identification of the functionality that was affected by the incident. For example, Order only, or Pre-Order AVQ and CSI and Loop Qual. Qwest already captures this in the Event Notification in the Description of Trouble and the Business Impact. Confirmed end date of the incident The Event Notification captures the end date and time of the incident, after the resolution is implemented, tested, and the issue is confirmed resolved. When a targeted patch date changes, Qwest issues an update to its notifications informing the CLECs of the new date. Confirmed end time of the incident See above. Total Minutes of Outage for this event: (outage as defined by performance measures) This information can be derived from the information already provided on the Event Notification start date/time against end date/time. Total Minutes of Degradation for this event See above. Corrective Actions taken to clear the incident The Event Notification captures the resolution of the issue, and what steps Qwest or the CLEC took to resolve the issue. The language has been negotiated and agreed upon in the CMP forum. Initial assessment of root cause The Event Notification captures the root cause, when known, in terms that best summarize the root cause from the customer perspective in the Description of Trouble. In some cases, for additional clarification, the Business Impact is written to recap the root cause. Final assessment of root cause See above. Actions taken to clear root cause: If the actions are future planned actions, like ‘new memory ordered and installation is planned for 10/29/03’ then a follow-up date must be indicated. See above. Qwest notes that the Event Notifications remain open until the patch is implemented tested, and the issue is resolved. Only then does Qwest close the Event. Next Follow-Up Due Date: Qwest to provide updated status until the actions required to correct the root cause have been tested and implemented. Qwest maintains that this would require additional tracking and resources that do not constitute a reasonable demonstrable business benefit associated with the cost. Further, Qwest notes that the Event Notifications remain open until the patch is implemented tested, and the issue is resolved. Only then does Qwest close the Event.

When Qwest does not immediately have a resolution, it updates its notifications regularly based on severity (i.e., Severity 1 and Severity 2 tickets are updated every hour; Severity 3 and 4 tickets, every forty-eight hours).

Sincerely,

Connie Winston Director, Information Technology Qwest

DRAFT RESPONSE For Review by CLEC Community and Discussion at the February 2004 CMP Meeting

Donna Osborne-Miller AT&T

SUBJECT: Qwest’s Change Request Revised Response - SCR111203-01 “Request for information regarding any unplanned OSS Event Qwest experiences”

This request asks for an event notification to be sent or posted to a Web site that captures the items in the table below. Further, AT&T requests that Qwest send a follow-up report within seven (7) business days of the event that captures the items in the table presented in boldface.

Qwest is conducting its review and analysis of existing Web site functionality around event notification reporting.

Qwest maintains that: a.) all items indicated in the table are already captured in the event notification itself, or can be derived from the information currently provided in the event notification, b.) Qwest is providing all information required per the CMP process. Qwest maintains that the reporting is redundant to the information already provided in the event notifications themselves. In addition, the maintenance of the Web site with the information being requested above would require an additional resource commitment of 20 hrs/week. Qwest is therefore denying this change request based upon two factors. The information requested is duplicative to information already being provided and has no reasonable demonstrable business benefit.

Summary Comparison Table: AT&T Request against Qwest Event Notification Memo AT&T Requests… Qwest Response Incident ID: The Event Notification number and the Trouble Ticket number associated with the issue are identical. Already captured in the Event. Incident type: degradation OR outage ; degradation AND outage This request is outside of the negotiated CMP requirements, and will require a revisitation of the agreement. Confirmed start date of the incident (incident is from the first alarm or report until the trouble is verified as cleared. Within an incident may be periods of degradation and periods of outage) The Event Notification captures the start date and time of the incident from the time when Qwest identifies the issue as multi-CLEC impacting. This is in compliance and consistent with the CMP requirements. Confirmed start time of the incident See above. Confirmed identification of the functionality that was affected by the incident. For example, Order only, or Pre-Order AVQ and CSI and Loop Qual. Qwest already captures this in the Event Notification in the Description of Trouble and the Business Impact. Confirmed end date of the incident The Event Notification captures the end date and time of the incident, after the resolution is implemented, tested, and the issue is confirmed resolved. When a targeted patch date changes, Qwest issues an update to its notifications informing the CLECs of the new date. Confirmed end time of the incident See above. Total Minutes of Outage for this event: (outage as defined by performance measures) This information can be derived from the information already provided on the Event Notification start date/time against end date/time. Total Minutes of Degradation for this event See above. Corrective Actions taken to clear the incident The Event Notification captures the resolution of the issue, and what steps Qwest or the CLEC took to resolve the issue. The language has been negotiated and agreed upon in the CMP forum. Initial assessment of root cause The Event Notification captures the root cause, when known, in terms that best summarize the root cause from the customer perspective in the Description of Trouble. In some cases, for additional clarification, the Business Impact is written to recap the root cause. Final assessment of root cause See above. Actions taken to clear root cause: If the actions are future planned actions, like ‘new memory ordered and installation is planned for 10/29/03’ then a follow-up date must be indicated. See above. Qwest notes that the Event Notifications remain open until the patch is implemented tested, and the issue is resolved. Only then does Qwest close the Event. Next Follow-Up Due Date: Qwest to provide updated status until the actions required to correct the root cause have been tested and implemented. Qwest maintains that this would require additional tracking and resources that do not constitute a reasonable demonstrable business benefit associated with the cost. Further, Qwest notes that the Event Notifications remain open until the patch is implemented tested, and the issue is resolved. Only then does Qwest close the Event.

When Qwest does not immediately have a resolution, it updates its notifications regularly based on severity (i.e., Severity 1 and Severity 2 tickets are updated every hour; Severity 3 and 4 tickets, every forty-eight hours).

Sincerely,

Connie Winston Director, Information Technology Qwest

DRAFT RESPONSE December 11, 2003 RE: SCR111203-01

Qwest has reviewed the information submitted as part of AT&T's Change Request SCR111203-01. Based upon research that has been conducted following the Clarification meeting (held November 18, 2003 and December 8, 2003) Qwest is still examining the issue. Qwest will continue to research the problem and provide an updated response at the December Systems CMP Meeting.

At the December Systems CMP Meeting, CMP participants will be given the opportunity to comment on this Change Request and provide additional clarifications. Qwest is interested in the experiences of the CMP community as relates to this issue. Qwest will incorporate any feedback received into further evaluation of this Change Request.

Sincerely, Qwest