Troubleshooting

The whole server

klyrn doctor                    # twenty-two checks, failures first
klyrn doctor --verbose          # the evidence for every one, not only the failures
klyrn doctor --bundle           # also write a redacted support bundle

klyrn health asks the core daemon, which makes it useless at the one moment you most need it. The doctor reads the filesystem, systemd and the network directly, so it still answers when core is down, proven by stopping core on a test server, where klyrn health printed one line about a missing socket and the doctor returned twelve checks passing and three failing, every one of them read from disk.

What it examines, in the order a request travels through the system:

• The binary the daemons actually run: a new build copied somewhere they do not read changes nothing, silently.
• Both units, and whether they are enabled at boot.
• The core socket, the panel's TLS listener, nginx and its configuration, the database server, every installed PHP-FPM pool.
• The panel database, including an integrity check when core is not holding it open.
• Permissions on every file that holds a secret.
• Disk space and inodes: inodes run out first on a server full of small files, and the error when they do names no file at all.
• Ports 80, 443 and the panel's, and which process holds each.
• The clock, because drift breaks certificate issuance and every two-factor code at once.
• Certificate expiry, cron file permissions (cron silently ignores anything group-writable) and log growth.
• And when core answers: its own health checks, failed jobs, and whether the backup schedules are actually producing copies.

It exits 2 when something is failing, 0 when nothing is, and 1 on a warning with --strict, so monitoring can use the exit code instead of parsing the output.

The support bundle

A support bundle is what you send someone when you want help, which makes it the most likely way a server's secrets ever leave it. So the design starts from what must never be in one. These files are not read at all:

• The master secret key, which decrypts everything sealed in the database.
• Any TLS or SSH private key.
• The panel database, which holds every account and every sealed secret.
• Any customer file, site content or database dump.

Everything that is read is scrubbed on the way in (API tokens, backup keys, webhook secrets, private key blocks, access key ids, authorization headers, connection strings, two-factor seeds and anything assigned to a key named like a password), and the bundle's own README states how many values were removed and of what kind, so nobody has to take the redaction on trust.

Process command lines are not captured either. They routinely carry credentials (mysqldump -p…, a bearer token on a curl), and scrubbing arbitrary command lines with patterns is a bet not worth taking, so the bundle records process names only.

The same thing is in the panel under Settings → Diagnostics, and it says what the file does not contain before it offers the download. Bundles are removed from the server after six hours.

Diagnose a site

klyrn site diagnose example.com
klyrn site diagnose example.com --repair php.pool.restart

Every probe runs, even after one has already failed: a site can be broken in two ways at once, and stopping at the first is how somebody ends up fixing the same site twice. The findings are sorted with the critical ones first.

What it checks, in the order a request travels:

• Whether the site is suspended.
• The nginx service, and this site's vhost.
• The PHP-FPM pool and its socket, or, for a Node site, the application service.
• The database the site uses, if it has one.
• Free space and inodes.
• The certificate and the names it covers.
• What the site actually returns over HTTP.
• Recent KLYRN jobs for this site.

There are three severities (critical, attention, healthy) and no fourth. More would invite arguing about the boundary instead of fixing the site.

The six repairs

A finding offers a repair by name, or it offers advice. Nothing else is accepted: there is no field on a repair that could carry a command, a path or an argument.

Every repair re-runs the diagnosis afterwards and reports the worst remaining finding. A repair that does not say whether it worked is a button, not a fix.

Common symptoms

502 Bad Gateway

Almost always the PHP-FPM pool or the application unit. klyrn site diagnose distinguishes "the pool is not running", "the pool file is missing", "the socket is stale" and "the app is down", and offers the matching repair. The HTTP probe stays quiet when one of those already explained it: it confirms the symptom, it does not also claim to be surprised by it.

403 Forbidden on every page

Either the site is suspended, in which case the diagnosis says so and who suspended it, or the file permissions are wrong. The second offers site.permissions.repair.

The certificate never arrives

The name does not resolve to this server. klyrn dns check <domain> says what it resolves to now and what this server's address is. The scheduler retries every ten minutes.

The browser warns about the panel's certificate

The panel is still on its self-signed certificate. Give it a hostname: First setup.

An upload is refused

The account is over its disk limit. The message carries the limit, the usage and how old the measurement is. Delete something and try again a minute later; see Resource limits.

WordPress went blank after an update

KLYRN does not roll WordPress updates back: the database may already have migrated. Restore the pre-update backup, which the failed job names.

The installer refuses to start

Preflight found another panel, Apache, or an nginx or MariaDB it did not configure. The message names what it found and how to stop it. See Requirements.

The health engine

klyrn health
klyrn health --events

Thirteen checks on a staggered schedule, from every 30 seconds for the services to every six hours for certificates. A failed check gets a bounded repair (restart the unit, restart application units that are in the failed state) and nothing else. It never deletes data, kills customer processes or changes firewall rules.

The circuit breaker allows three repair attempts per check per thirty minutes. After that it stops and records: "automatic repair paused: 3 attempts in 30 minutes did not hold. Needs a human." A check that returns to healthy closes the circuit and gets a fresh budget.

klyrn health --events is the record of every attempt with its reason, action, result and duration. It is the first thing to read when a service has been flapping.

The checks: nginx, PHP-FPM, MariaDB and the panel itself; disk space and inodes; memory and CPU load; clock synchronisation; the firewall; Node application units; certificates; KLYRN updates; and per-account disk usage.

Where to look

The panel has a logs centre on each site page: it lists the sources that site has (access, error, PHP) and follows them incrementally by cursor, so a page that is following a log does not re-fetch what it already has. An administrator also gets the server-wide sources there, which the site view deliberately refuses: KLYRN's own journal carries every site's traffic and every decision the panel made.

From a shell, the same material and rather more:

klyrn site logs example.com --kind error --lines 200
klyrn app logs app.example.com --lines 200
klyrn jobs
klyrn job 118

journalctl -u klyrn-core -n 200
journalctl -u klyrn-web -n 200
/var/log/klyrn/                     # structured logs
/home/<account>/logs/<domain>.error.log

A failed job keeps its steps and its log. The step that failed carries the error, and every step after it is recorded as cancelled rather than left looking as though it never existed.

When a job will not start

KLYRN refuses to run two jobs against the same site at once, so a second attempt is rejected with a conflict rather than queued behind something that might be stuck. If a job is genuinely wedged, klyrn jobs shows it as running; restarting klyrn-core marks anything that was running as failed with "interrupted: klyrn-core restarted", which clears the way.

Backups, migrations and system jobs run one at a time; site jobs run three at a time, certificates two, everything else two. A queued job is waiting for that limit, not stuck.