Troubleshooting

The whole server

klyrn doctor                    # twenty-two checks, failures first
klyrn doctor --verbose          # the evidence for every one, not only the failures
klyrn doctor --bundle           # also write a redacted support bundle

klyrn health asks the core daemon, so it cannot answer when core is down. The doctor reads the filesystem, systemd and the network directly. With core stopped on a test server, klyrn health printed one line about a missing socket. The doctor returned twelve checks passing and three failing.

What it examines, in the order a request travels:

• The binary the daemons run.
• Both units, and whether they are enabled at boot.
• The core socket, the panel's TLS listener, nginx and its configuration, the database server, every installed PHP-FPM pool.
• The panel database, including an integrity check when core is not holding it open.
• Permissions on every file that holds a secret.
• Disk space and inodes.
• Ports 80, 443 and the panel's, and which process holds each.
• The clock. Drift breaks certificate issuance and every two-factor code.
• Certificate expiry, cron file permissions and log growth. Cron ignores anything group-writable.
• When core answers: its own health checks, failed jobs, and whether the backup schedules are producing copies.

It exits 2 when something is failing and 0 when nothing is. With --strict it exits 1 on a warning, so monitoring can use the exit code.

The support bundle

A support bundle never reads these files:

• The master secret key, which decrypts everything sealed in the database.
• Any TLS or SSH private key.
• The panel database, which holds every account and every sealed secret.
• Any customer file, site content or database dump.

Everything that is read is scrubbed on the way in. That covers API tokens, backup keys, webhook secrets, private key blocks, access key ids, authorization headers, connection strings and two-factor seeds. It also covers anything assigned to a key named like a password. The bundle's README states how many values were removed and of what kind.

Process command lines are not captured, because they routinely carry credentials (mysqldump -p, a bearer token on a curl). The bundle records process names only.

The panel offers the same under Settings, then Diagnostics. Bundles are removed from the server after six hours.

Diagnose a site

klyrn site diagnose example.com
klyrn site diagnose example.com --repair php.pool.restart

Every probe runs, even after one has failed, because a site can be broken in two ways at once. The findings are sorted with the critical ones first.

What it checks, in the order a request travels:

• Whether the site is suspended.
• The nginx service, and this site's vhost.
• The PHP-FPM pool and its socket, or, for a Node site, the application service.
• The database the site uses, if it has one.
• Free space and inodes.
• The certificate and the names it covers.
• What the site returns over HTTP.
• Recent KLYRN jobs for this site.

There are three severities: critical, attention and healthy.

The six repairs

A finding offers a repair by name, or it offers advice. A repair has no field that could carry a command, a path or an argument.

Every repair re-runs the diagnosis afterwards and reports the worst remaining finding.

Common symptoms

502 Bad Gateway

Almost always the PHP-FPM pool or the application unit. klyrn site diagnose says which and offers the matching repair. The HTTP probe stays quiet when one of those already explains the 502.

403 Forbidden on every page

The site is suspended, and the diagnosis says by whom, or the file permissions are wrong. For those it offers site.permissions.repair.

The certificate never arrives

The name does not resolve to this server. klyrn dns check <domain> says what it resolves to now and what this server's address is. The scheduler retries every ten minutes.

The browser warns about the panel's certificate

The panel is still on its self-signed certificate. Give it a hostname: First setup.

An upload is refused

The account is over its disk limit. The message carries the limit, the usage and how old the measurement is. Delete something and try again a minute later. See Resource limits.

WordPress went blank after an update

An update started from the panel puts the site back from its pre-update backup by itself. A scheduled update does not, because the database may already have migrated. Restore the pre-update backup, which the failed job names.

The installer refuses to start

Preflight found another panel, Apache, or an nginx or MariaDB it did not configure. The message names what it found and how to stop it. See Requirements.

The health engine

klyrn health
klyrn health --events

Thirteen checks run on a staggered schedule, from every 30 seconds for the services to every six hours for certificates. A failed check gets a bounded repair: restart the unit, or restart application units that are in the failed state. It never deletes data, kills customer processes or changes firewall rules.

The circuit breaker allows three repair attempts per check per thirty minutes. After that it stops and records: "automatic repair paused: 3 attempts in 30 minutes did not hold. Needs a human." A check that returns to healthy closes the circuit and gets a fresh budget.

klyrn health --events records every attempt with its reason, action, result and duration. Read it first when a service has been flapping.

The checks cover nginx, PHP-FPM, MariaDB, the panel itself, disk space, inodes, memory, CPU load and clock synchronisation. They also cover the firewall, Node application units, certificates, KLYRN updates and per-account disk usage.

Where to look

Each site page in the panel has a logs centre. It lists the sources that site has (access, error, PHP) and follows them by cursor. An administrator also gets the server-wide sources there.

From a shell:

klyrn site logs example.com --kind error --lines 200
klyrn app logs app.example.com --lines 200
klyrn jobs
klyrn job 118

journalctl -u klyrn-core -n 200
journalctl -u klyrn-web -n 200
/var/log/klyrn/                     # structured logs
/home/<account>/logs/<domain>.error.log

A failed job keeps its steps and its log. The step that failed carries the error, and every step after it is recorded as cancelled.

When a job will not start

KLYRN refuses to run two jobs against the same site at once. A second attempt is rejected with a conflict and is not queued. A wedged job shows as running in klyrn jobs. Restarting klyrn-core marks anything that was running as failed with "interrupted: klyrn-core restarted", which clears the way.

Backups, migrations and system jobs run one at a time. Site jobs run three at a time, certificates two, everything else two. A queued job is waiting for that limit.