Recover after total blackout

Description

This article explains how to restore StarWind Virtual SAN operations after a total blackout that affected all nodes in a High Availability (HA) cluster. It describes how to manually identify and mark the synchronization source if needed, and how to safely bring all nodes back online to resume normal operation. Following this procedure ensures data consistency after a full power loss or system-wide failure.

Steps

After all nodes of the HA cluster were down, StarWind VSAN by default is able to determine which node holds the most recent data and automatically starts the synchronization process if all nodes are online.

In case StarWind services cannot determine which node contains the most recent data, they block all incoming connections to keep data consistency until one of the HA partners is marked as Synchronized. This process would require manual involvement from an administrator or StarWind support and ensures that no outdated node can overwrite newer data that was possibly written while the device on the node remained active before the outage.

If you face the Not Synchronized device status on all StarWind nodes, you would need to determine which node has the most recent data. To do so, you have to investigate StarWind logs or Windows Application logs on all nodes and identify the node that last had the device(s) state as synchronized at the moment of the outage.

NOTE: StarWind generates events by device, thus in case you have several devices, you have to identify the state for each device.

• StarWind logs on CVM are located in /opt/starwind/starwind-virtualsan/drive_c/starwind/logs
• StarWind events on CVM are located in /opt/starwind/starwind-virtualsan/drive_c/starwind/events

To check StarWind events on Windows, open Windows Event Viewer

Application logs on each node.

The StarWind event codes are available here:

https://www.starwindsoftware.com/help/EventsofStarWindVSAN.html

If you know for sure which node has the most recent and actual data, select it as the synchronization source.

To do so, use StarWind Management UI and on the node that has the most recent data, right-click on the device and select “Mark as Synchronized”.
See this page for details: https://www.starwindsoftware.com/help/MarkasSynchronized.html

NOTE: The actual version of StarWind Management Console can be downloaded here.

If you are not sure which node has the most recent and actual data, in order to avoid accidental data rewrite, follow the further steps:

• Stop and disable the StarWind VSAN service on all nodes except one that you assume has the most recent data.

NOTE: The service stop will change the state for all active devices to “Not synchronized” – consider this if you have any active workloads running on the devices.

• On the node that you assume has the most recent data, mark the device as Synchronized (See this page for details: StarWind VSAN Help : Mark as Synchronized )
• Check and validate the data consistency and actuality.
• Once it’s confirmed that the data is consistent and valid on the node, enable and start the StarWind service on the remaining nodes and let the devices synchronize.
• If the data on the node appears not relevant, outdated, or inconsistent, stop and disable StarWind VSAN service on the active node and repeat the procedure with the next one.

Conclusion

Recovering from a total blackout requires careful validation of data integrity before re-enabling synchronization. StarWind Virtual SAN simplifies this process by automatically detecting the most current data source when possible. Still, in some cases, manual recovery is required. Following the outlined steps ensures safe cluster restoration, consistent data identification, and prevention of data loss/accident rewrite conditions after unexpected power loss.