Run a team node#

A team node is one Lockpin on a shared machine that holds the team’s provider keys sealed, runs as its own account, and hands out Lockpin keys (lp_...) or short-lived spend capabilities (lpc_...). This page is for the admin who sets one up on a server, in a container or in a Kubernetes pod.

When to use a node#

A plain team server keeps the provider keys in its environment. A node goes further:

  • The provider keys are sealed at rest by the operating system, readable only by the node’s account.

  • The program is copied where only an administrator can change it, so the people who use the machine cannot read the node’s keys through it.

  • Agents can hold a spend capability: a short-lived token that says which task, how many dollars, which models and tools, and until when.

Everything on Set up a team works on a node too: teams, member defaults, invites, keys and the member pages.

Before you begin#

  • Lockpin is installed on the machine. See Install.

  • You can run commands elevated: an administrator’s prompt on Windows, sudo on macOS and Linux.

  • You have the provider keys for the providers the team uses.

Install a node#

  1. Run the install elevated:

    From an administrator’s prompt:

    lockpin node install

    sudo lockpin node install

  2. Type the provider keys when asked, one per line as provider=key (meta for MODEL_API_KEY’s). An empty line ends them. They are not shown. To read them from a file instead, add -key-file FILE (provider=key per line, # comments).

  3. Copy the admin key it prints. It is shown only this once. It has no end of its own, and a maximum lifetime set later never ends it.

-dry-run prints every file and command and changes nothing.

What install does#

  1. Makes the node’s own account if it is missing: NT AUTHORITY\LocalService on Windows, _lockpin on macOS, lockpin on Linux.

  2. Makes the node’s folder, readable by that account and administrators only, and copies lockpin where the node’s account can run it but not replace it.

  3. Seals the provider keys into lockpin-node.sealed and writes lockpin-keys.json with one team key, admin.

  4. Registers the node to start at boot as that account and restart on failure, and starts it.

  5. Prints the node key’s public half, the admin key and the mint command.

System

Node folder

Registered as

Windows

C:\ProgramData\Lockpin\node

Scheduled Task \Lockpin Node

macOS

/Library/Application Support/Lockpin/node

LaunchDaemon dev.lockpin.node

Linux

/var/lib/lockpin

System unit lockpin-node.service

The node runs with -listen 127.0.0.1:8787 -keys ... -capabilities ... -ledger ... -admin-listen 127.0.0.1:8790, plus any -args "FLAGS" you give.

With the Windows installer#

Lockpin_<version>_x64-setup.exe /TEAMNODE installs Lockpin as usual, then runs lockpin node install -pause elevated (one UAC prompt) in a console window. The window asks for the keys and keeps the admin key on screen until you press Enter.

/S /TEAMNODE /KEYFILE=C:\path\keys.txt installs with no window. The admin key is then shown nowhere, so make one from an administrator’s prompt:

lockpin keys create NAME -role admin -file C:\ProgramData\Lockpin\node\lockpin-keys.json

Manage the node#

Command

What it does

lockpin node status

Installed or not, running or not, its pid and the log.

lockpin node start, stop, restart

Start, stop or restart it.

lockpin node install again

After upgrading lockpin, or to change -args. Give -args again: a reinstall without it adds none. The sealed store and keys file are kept.

lockpin node install -reseal

Ask for the provider keys again and replace them. The node key and root secrets stay, so tokens already minted keep working.

lockpin node uninstall

Stop it and remove the definition and the program’s copy. The folder and the account stay.

Provider keys given to a reinstall without -reseal are refused rather than dropped.

Run commands on the node#

Commands that read the sealed store run elevated, with -sealed set to the store in the node’s folder:

sudo lockpin capabilities mint -sealed /var/lib/lockpin/lockpin-node.sealed -issuer alice -usd 5

To change the keys file on macOS or Linux, run lockpin keys, teams and invites as the node’s account, so the file stays its own (sudo -u _lockpin on macOS):

sudo -u lockpin lockpin teams create ops -file /var/lib/lockpin/lockpin-keys.json
sudo -u lockpin lockpin keys create svc.ops.ci -service -team ops -file /var/lib/lockpin/lockpin-keys.json

On Windows, run them from an administrator’s prompt.

Reach the meter#

The meter and lockpin approve talk on -admin-listen, 127.0.0.1:8790, reachable from the node’s own machine only. From another computer, use an SSH tunnel:

ssh -L 18790:127.0.0.1:8790 node

On the node, the admin page asks for the admin key once per tab session. Admins on other computers can change keys, teams and limits over the remote admin channel.

Run a node in a container#

lockpin node run runs a node in the foreground as the current account. It needs no service manager and makes no account, so it fits a container, a Kubernetes pod or a service manager you set up yourself. SIGTERM stops it cleanly.

lockpin node run [-data DIR] [-key-file FILE] [-listen ADDR] [-admin-listen ADDR] [-- PROXY FLAGS]
lockpin node health [-listen ADDR]
lockpin node admin-key [-data DIR] [-keep]

Flag

Default

What it does

-data DIR

/var/lib/lockpin, or $LOCKPIN_NODE_DATA

The node’s folder. Mount a volume or a folder there, never single files.

-key-file FILE

$LOCKPIN_KEY_FILE

The provider keys, provider=key per line, read at the first start.

-listen ADDR

127.0.0.1:8787

In a container give 0.0.0.0:8787.

-admin-listen ADDR

127.0.0.1:8790

Where the meter and lockpin approve talk. It stays on loopback.

-- FLAGS

none

Everything after -- goes to the proxy, for example -- -ask-headless.

  • Keys at the first start. The node reads the provider keys from the file LOCKPIN_KEY_FILE names (a mounted secret), or else from LOCKPIN_PROVIDER_KEYS, and seals them into lockpin-node.sealed. Use the file: a value in the environment stays in the container’s definition and in /proc/1/environ. Later starts keep the store; keys given again replace the provider keys when they changed, keeping the node key and root secrets.

  • The admin key. The first start writes lockpin-keys.json with one team key, admin, and puts its secret in lockpin-admin-key.txt in the folder, never in the log. lockpin node admin-key prints it and deletes the file (-keep leaves it).

  • The store. A container has no DPAPI, Keychain or systemd-creds, so the sealed store is a file only the node’s user can read (mode 0600). The protection is the container’s user and the volume’s permissions.

  • Health. lockpin node health exits 0 when the node answers /lockpin/healthz with 200, else 1 (503 while calls are refused). Use it as the container’s health check.

  • Stopping. On SIGTERM the node refuses every held call at once, stops taking calls, gives the ones in flight 30 seconds, syncs the ledger and exits 0. Give it 45 seconds to stop.

  • Approvals. With no screen watching, add -- -ask-headless and answer held calls with lockpin approve. See Approvals at the cap.

Important

Run one node per ledger. The ledger is one file with one writer, so do not scale a node to several replicas: a second one is refused by the ledger’s lock or, on a volume that ignores locks, keeps a second ledger that caps nothing correctly. There is no high availability: while the node restarts, agents’ calls fail.

Port 8787 is plain HTTP. For callers on another network, put TLS in front.

Spend capabilities#

A spend capability (lpc_...) is a short-lived token that says which task, how many dollars, which models and tools, and until when. A client sends it where it sends a team key: Authorization: Bearer lpc_... or x-api-key. A request with a team key works exactly as before.

  1. On the node, mint a root capability:

    lockpin capabilities mint -issuer ci -usd 5 -expires 24h

    The token goes to stdout and what it allows to stderr. The node keeps no copy.

  2. Any holder can hand a narrower child to a sub-agent, with no node and no network. It reads the parent from LOCKPIN_CAPABILITY or stdin:

    lockpin delegate -usd 0.50

  3. Check what a token allows:

    lockpin capabilities show

Command

What it does

capabilities mint -issuer NAME -usd USD [-expires 24h] [-task TASK] [-models LIST] [-tools LIST]

Mint a root capability on this node.

delegate -usd USD [-expires DUR] [-task SEGMENT] [-models LIST] [-tools LIST] [-fresh SECONDS]

Print a child that can do no more than its parent. Its end defaults to the shorter of the parent’s and 2h.

capabilities show

Print what a capability allows.

capabilities rotate [-grace 10m]

Start a new root secret. Tokens under the old one work until the grace ends.

revoke -root ID or revoke -issuer NAME

Revoke a root, or every root of an issuer, in the next heartbeat. Run it on the issuer node.

Budgets#

A token’s usd bounds what it and every token made from it spend together. A child has its own budget and also spends from its parent’s, so children of one parent share what the parent has left. Before each call the node holds the call’s worst case against the token’s budget and every budget above it. Without -task, lockpin delegate adds a random six-character segment, so every child has a budget of its own.

A refused call gets 402 budget_exceeded with a lockpin.capability part: stopped_by is usd, models, tools, expires, fresh or revoked, and id is the budget that stopped it (a hash, never the token). A token this node did not mint, or one that was edited, gets 401 invalid_api_key.

Freshness and revocation#

The issuer signs a heartbeat with the time and the revoked roots and issuers. A node honours a capability only while its last heartbeat is younger than the token’s fresh (600 seconds unless a delegate narrowed it). A node that cannot reach its issuer refuses capabilities after at most that long, and keeps serving team keys.

lockpin revoke applies within a second on the issuer itself, within a minute on nodes that fetch its heartbeat, and within the token’s freshness window on any node. There is no command to undo a revocation.

Several nodes#

Each node mints and verifies its own capabilities. One node is the issuer of heartbeats and revocations. Keep its admin listener on loopback, give each other node an SSH tunnel to it, and add two flags to each other node, the second being the issuer’s node key as its seal printed it:

ssh -L 18790:127.0.0.1:8790 issuer

-capabilities-issuer http://127.0.0.1:18790 -capabilities-issuer-key HEX

Keep every node’s clock synchronised. A node more than 2 minutes behind its issuer refuses its heartbeats, and so soon refuses every capability.

The identity rule#

A node that runs as a user who signs in on its machine refuses to start: whoever holds a capability as that user could read the node’s keys. To run it that way anyway (one developer, one laptop), add -capabilities-same-user. Capabilities are then bookkeeping only, and the log and the meter say so.

A ledger an auditor can verify#

Every ledger line carries the SHA-256 of the line before it. Every 1,000 entries and when the node stops, it appends a checkpoint signed with its node key. Check a copy with the node key’s public half:

lockpin ledger verify -key HEX copy.jsonl

It prints verified N entries, chain broken at entry N or signature failed at checkpoint N. Keep the public key each seal prints somewhere safe.

Recovery#

  • The node’s task, LaunchDaemon or unit was removed, or the program is old. Run lockpin node install again. It keeps the sealed store and the keys file, and asks for no key.

  • The sealed store or node key is lost. A fresh seal makes a new root secret and a new node key. Every token minted before dies, other nodes need the new public key, and checkpoints signed before verify only with the old key.

Honest limits#

  • A capability is a bearer credential: anyone who holds it can spend what it allows until it expires or is revoked. Keep tokens out of prompts, tool arguments and shared files.

  • Sealing protects the keys at rest. While the node runs they are in its memory: the protection is the account the node runs as plus the sealed store.

  • On Windows the node runs as NT AUTHORITY\LocalService, which other Windows services share. Any other service running as LocalService can read the node’s keys too.

  • The revocation list beside the sealed store, lockpin-node.sealed.revoked.json, is protected only by the same account boundary. Deleting it un-revokes everything.

  • After a node restarts, a token revoked shortly before can be honoured again for up to its freshness window.

  • Stop the node before a reinstall (lockpin node stop).

Verify#

  1. Check the node is installed and running:

    lockpin node status

  2. Make one call with the admin key, or a key you made, against http://127.0.0.1:8787/v1.

Next steps#