Regarding failures in Local Government Information Security Cloud managed by our company (final report)

＞Please check here for the first report.
＞Please check here for the second report.

On August 8, 2022, a problem occurred in Local Government Information Security Cloud provided by SB Technology Corp. (hereinafter referred to as the Company) that made the Internet access service and email service unavailable.
We would like to inform you that all planned measures to prevent recurrence of this incident have been completed.

We sincerely apologize for the great concern and inconvenience this has caused to our customers and many other stakeholders.

1. Fault overview

Impact time: August 8, 2022 (Monday) 4:27 a.m. to August 9, 2022 (Tuesday) approximately 1:30 a.m.
Scope of impact: Internet access services and email used by approximately 190,000 local government employees in nine prefectures: Aomori, Iwate, Miyagi, Akita, Fukushima, Niigata, Tochigi, Saga, and Nagasaki, as well as each municipality. service

2. Cause of failure

　

- Server aggregation switch failure
A failure (hereinafter referred to as this event) occurred in a multiplexed server aggregation switch, and communication with the virtualization infrastructure was completely cut off, resulting in the following failure.

1. Due to a communication breakdown between the local government's network and the proxy server on the virtualization platform, the local government was no longer able to browse the Internet.
2. The email relay system stopped due to a communication breakdown between the email relay server and the isolated server on the virtualization platform.
3. The operation management system on the virtualization platform became unable to communicate with the outside world, making monitoring and remote maintenance impossible.
4. When restarting and switching network equipment as an emergency response to a failure, the virtualization infrastructure detected the restart of the server aggregation switch as an abnormality and stopped it.

3. Causes of long-term disability

・Maintenance route
Local Government Information Security Cloud provided by our company is configured so that communication is not possible using any network or protocol other than those specified. As a result, communication to the operation management system was completely cut off, making it impossible to perform remote maintenance and grasp the situation. As a result, each piece of equipment had to be investigated and dealt with on-site, which resulted in a lengthy process.

4. Measures to prevent recurrence

- Server aggregation switch failure
Because the network equipment was restarted, there was no failure log on the server aggregation switch when this event occurred. Also, no known defects have been confirmed.
We conducted a reproduction test for approximately one month regarding the server aggregation switch where the failure occurred, but this event was not reproduced.
In response to this, we implemented the following measures.

1. Replaced server aggregation switch and all peripheral cables
2. Further multiplexing server aggregation switches and strengthening the failure detection system
3. Implemented version upgrades for various network devices

5. Regarding responses to prolonged disability

・Multiple maintenance routes and fault monitoring
We added more remote maintenance routes and installed a dedicated fault monitoring system.

・Inspection of each facility of Local Government Information Security Cloud
We conducted a complete inspection and correction of the multiplexed network in case a failure occurred.

We will continue to work to improve quality so that our customers can use our services with peace of mind.