Recover a lost controller

Your machines do not need the controller to keep running. If the controller's server is lost, every machine stays up. You lose the panel and the API until you restore it.

What is backed up

The controller copies its database and keys every night and keeps 14 copies. It also sends a sealed copy to every compute node, so a copy survives if the controller's server does not.

Before anything goes wrong

1. In the panel, open Settings, then Backups, and choose Make a recovery key.
2. Store the key somewhere safe that is not the controller or a compute node. It is shown once.
3. A day later, check the same page. It lists each node that holds the newest copy.

A compute node stores the sealed copy but cannot open it.

Restore on a new server

Install KLYRN VM on the new server, and do not start the service yet. Copy the newest file from /var/lib/klyrn-vm/controller-recovery/ on any compute node, then restore it with your recovery key.

klyrn-vm restore --recovery controller-<time>.recovery --key klyrn-recovery-1:...

Point the controller's hostname at the new server and start the service.

systemctl start klyrn-vm-controller

Your compute nodes reconnect on their own. You do not need to change anything on them.

What a restore cannot bring back

• Changes made after the last nightly copy.
• Machine backups that were stored on the lost server itself. Keep your backup server separate from the controller.