SIX Swiss Exchange suffered an unfortunate outage last week, the worst in over a decade. Unscheduled outages are every IT engineer’s worst nightmare and can slip through even the best-architected, robust, and resilient infrastructure. A good problem-management process is vital to reduce outages. 

Problem management is an overarching extension of incident management, which aims to restore service operations swiftly and minimise business impacts, focusing on the root cause investigations of Severity 1 and 2 incidents. It subsequently recommends preventative measures to avoid future recurrence.

All CJC's technical staff are ITIL accredited and CJC monitors IT environments for more than 400 clients 24/7, with an average of 500,000 monitoring points per client - half a million things that could go wrong every day.

Coordinating with all parties, from end-users to vendors, CJC performs root cause analysis to find and recommends remedial solutions to the problem. Instances are tracked until closure, followed by a post-mortem report as required.

The problem management process includes: 

1. Correlating and organising incidents to create a Problem Record.
2. Analysis of the Problem Record, including available history.
3. Characterising and prioritising the Problem Record and determining the appropriate actions.
4. Providing actionable recommendations to the client.
5. Working with the client’s Change Management team to deploy a permanent fix.
6. Providing root cause analysis for Severity 1 Incidents.
7. Close the Problem Record on resolving.

Formal and detailed incident reports are available to clients that experienced a Severity 1 incident where CJC is accountable for the root cause. The reports are designed to provide clients with complete transparency on significant incidents and inform clients of the proposed plan to reduce the risk of recurrence.

Each incident report provides a summary of the event, including:

• Details of the incident and impact.
• Description of the root cause.
• Details of the action plan to reduce recurrence.

CJC aims to deliver 3 updated reports in the following timeframes:

• Initial incident report published on the same business day, or within 24 hours after reaching an incident resolution.
• Updated (root cause) incident report within 3 business days of the incident.
• Final (root cause and action plan) incident report within 5 business days of the incident.