Troubleshooting#

This page helps you find out why an agent’s call failed or Lockpin will not run, and what to do about it. It also says what Lockpin does when one of its own parts fails.

Start with lockpin doctor#

lockpin doctor checks the whole chain and prints a pass/fail list with a fix for each failure: the price catalog and its age, the port, whether a Lockpin answers there, the background service, the ledger, each provider’s key variable, which agents send their calls through Lockpin and which hooks they have, and where a test call to each provider would go. It makes no model calls and never prints a key.

lockpin doctor -config lockpin.json

Behind a company proxy, read its egress line first. Add -network to also send one HEAD request, with no credentials, to each provider’s host.

If Lockpin runs as a background service, check it and its log:

lockpin service status

System

Log

Windows

%LOCALAPPDATA%\Lockpin\lockpin.log

macOS

~/Library/Logs/Lockpin/lockpin.log

Linux

journalctl --user -u lockpin

What the agent sees#

Symptom

Meaning and fix

cannot listen on 127.0.0.1:8787

Another Lockpin (or program) holds the port. Stop it, or pass -listen 127.0.0.1:8788 and use that port in the agent.

Connection refused from the agent

Lockpin is not running (lockpin service status, then lockpin service start; the log says why it stopped), or the agent uses another port or localhost over IPv6; use 127.0.0.1.

401 invalid_api_key

Team mode: the agent is not sending a Lockpin key (lp_...), or it was disabled or rotated.

401 key_expired

Team mode: the key has passed its end date. The error names the command that renews it (lockpin keys rotate NAME).

402 budget_exceeded

The task or key reached its limit. The body says which, the numbers and, for key limits, when it resets. Raise it in the meter (tasks: Edit; beyond -allowance-pct only from the meter page, since the X-Lockpin-Budget-Usd header only lowers) or with lockpin keys set (keys).

402 “would exceed its daily limit”

The machine reached -day-limit (default $20 per UTC day, all tasks together). It resets at 00:00 UTC; raise it with -day-limit.

400 unknown_model

The model has no price, so there is no cap. See lockpin providers -models; add the model with -prices FILE.

400 task_required

A service key’s call did not name its run in X-Lockpin-Task.

403 not_allowed

Team mode refuses calls it cannot meter (embeddings, files, batches).

403 uncapped_call

Solo mode refuses a call it cannot meter (embeddings, images, audio, files, batches) until -allow-uncapped lists its path (* for all).

503 no_provider_key

Team mode: the server has no key for that provider. Set its variable and restart.

503 policy_unavailable

-policy-required is on and no signed machine policy is in force. Check it with lockpin policy show.

404 unknown_provider

The base URL is wrong. Meta is /v1; the others are /<provider>/v1, and Anthropic’s is /anthropic.

502 upstream_error with x509: certificate signed by unknown authority

The network inspects TLS with a private root the machine does not trust. Give it to Lockpin with -ca-file (or LOCKPIN_CA_FILE), or have IT install it in the system store. Never work around it by changing the provider URL to plain HTTP.

502 upstream_error with proxyconnect or Proxy Authentication Required

HTTPS_PROXY points at a proxy that is down, or wants credentials (http://user:password@proxy:3128). lockpin doctor shows which proxy each call uses.

503 ledger_unavailable, keys_unavailable or reserve_failed

One of Lockpin’s own parts failed. See If Lockpin itself fails.

Tip

A held call at the cap waits up to -ask-at-cap (default 3 minutes) for an answer in the meter or VS Code before it gets its 402. On a server with no screen, list and answer held calls with lockpin approve list, lockpin approve ID and lockpin deny ID (see Approvals at the cap).

If Lockpin itself fails#

Lockpin sits between your agent and the provider, so it is one more thing that can fail. It guards money, so when its own parts fail the default is to fail closed: a call that cannot be reserved and recorded is refused with a clear status and error code, never sent unseen. -on-fault allow is the choice to trade that guarantee for availability.

What fails

-on-fault block (default)

-on-fault allow

The ledger file cannot be written (disk full, file gone)

503 ledger_unavailable with Retry-After; calls resume when a write works again (probed every second)

Calls go on, still capped from memory, marked X-Lockpin-Degraded: ledger. Spend since the failure is lost if Lockpin restarts.

The ledger file cannot be read at start

Lockpin does not start

The same: starting empty would reset every cap

The keys file is unreadable or corrupt when reloaded (team mode)

503 keys_unavailable, until a reload works. The old keys are kept in memory, but they would keep a disabled key alive and ignore a lowered limit.

The old keys keep serving, marked X-Lockpin-Degraded: keys

Reserving a call fails for a reason other than the budget

503 reserve_failed

The call is forwarded uncapped, marked X-Lockpin-Uncapped: reserve_failed

A panic in Lockpin’s own code

The reservation is released (or charged in full if the call was already sent), the agent gets 500 handler_panic, the server keeps running

The same, and if the call had not been sent yet, it is forwarded uncapped, marked X-Lockpin-Uncapped: handler_panic. A call that was already sent is never sent twice.

Drift snapshots fail, or drift tracking panics

The call goes on; drift shows as degraded (after a panic it stays off until restart)

The same

Slack or the webhook cannot be reached

The call goes on; alerts show as degraded

The same

  • allow never turns a working check off. It waives the refusal for the part that is broken, and marks the call, so the spend can be found afterwards.

  • An uncapped call is still recorded: its real cost is added to the task and key when the reply reports usage (or its worst case, when it does not), and the meter’s JSON counts such calls as uncapped. A model with no price is refused even then.

  • Every refusal or pass-through because of a fault raises a fault alert (at most one per part every 10 minutes) and a WARN line in the log.

  • A panic after a reply has started cannot become an error, so the connection is cut: your agent sees a broken stream, not a clean end.

Health check#

GET /lockpin/healthz needs no key, even in team mode, and does no real work, so a load balancer or a supervisor can poll it:

{"status": "degraded", "on_fault": "block", "calls_refused": true,
 "components": {"ledger": {"status": "degraded", "reason": "the ledger file cannot be written", "since": "2026-10-20T09:14:02Z"},
                "keys": {"status": "ok"}, "handler": {"status": "ok"}}}

status is degraded when any part is. The HTTP status is 503 only when calls are being refused (block, with the ledger or keys down) and 200 otherwise, so a balancer pulls a server out of rotation only when it cannot serve; a monitor that wants warnings reads status. The error text, which can contain file paths, is only included for requests from the server itself.

For a team node in a container, lockpin node health exits 0 when the node answers /lockpin/healthz with 200, else 1.

When Lockpin is not running at all#

Nothing above helps if the process is gone or unreachable: your agent’s calls fail to connect. Your choices:

  1. Let the agent stop. This is the safe default, and the right one when the cap matters more than uptime. In team mode it is also the only option: developers hold only lp_... keys and cannot reach the provider without Lockpin.

  2. Fall back to the provider in the client. Point the client at the provider’s own URL when Lockpin cannot be reached. Those calls have no cap, no meter and no record, and the real key must be on the client. Fall back only when Lockpin was not reached (connection refused or a short connect timeout), never on a 402, 429 or 503 from Lockpin (those are decisions), and never after a read timeout (the call may be running, and would be sent twice):

    import httpx, openai
lockpin = openai.OpenAI(base_url="http://127.0.0.1:8787/v1", api_key=KEY,
                        timeout=httpx.Timeout(600, connect=2), max_retries=0)
direct = openai.OpenAI(base_url="https://api.meta.ai/v1", api_key=KEY)

def respond(**kw):
    try:
        return lockpin.responses.create(**kw)
    except openai.APITimeoutError:
        raise                      # it may be running; do not send it twice
    except openai.APIConnectionError:
        return direct.responses.create(**kw)   # Lockpin was not reached: uncapped!

  3. Keep it up. Run it under a supervisor that restarts it (lockpin service install, Restart=always in systemd, a Docker restart policy) and point its liveness check at /lockpin/healthz. A restart is quick, and the ledger is replayed from its file.

  4. Set timeouts. Give the client a short connect timeout (a second or two) so an outage is noticed at once, and a read timeout longer than your slowest call. Clients that retry on 503 (most SDKs do, twice) ride out a ledger_unavailable that clears within seconds, because the refusal carries Retry-After.

Warning

A call in flight when Lockpin dies was reserved in memory only, so the provider may bill it with nothing in the ledger. A power cut or kernel crash can lose the last events in the ledger (a process crash cannot). One Lockpin process holds the ledger; there is no standby that shares budgets with it.

Next steps#