Skip to content
Draft
78 changes: 78 additions & 0 deletions docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,84 @@ references and [kernel promise](#kernel-promise) states. The router handles
[delivery](#delivery) logic. See the
[KernelRouter](../packages/ocap-kernel/src/KernelRouter.ts) for routing logic.

## Networking

### netlayer

A pluggable network-transport backend for the [kernel](#kernel)'s remote communications. A
netlayer moves opaque message strings between kernels and authenticates each peer as the
[neutral peer id](#neutral-peer-id) derived from the kernel's key seed. The kernel core is
netlayer-agnostic: it consumes the `Netlayer` contract and handles the Ken protocol
(sequencing, acknowledgement, retransmission, deduplication) itself — see
[`docs/ken-protocol-assessment.md`](../docs/ken-protocol-assessment.md) — so a netlayer need
only provide best-effort ordered delivery per live connection. Netlayers ship as separate
packages — [`@metamask/netlayer`](../packages/netlayer/src/index.ts) (contracts + shared
machinery), `@metamask/netlayer-loopback`, and `@metamask/netlayer-libp2p` — and are supplied
to a runtime via a `NetlayerRegistry` and selected per kernel with a `NetlayerSpecifier`. A
WebSocket netlayer and an iroh netlayer are planned future implementations. See [writing a
netlayer](../docs/writing-a-netlayer.md).

### location hint

A netlayer-specific, opaque string that helps a [netlayer](#netlayer) re-locate and reconnect
to a peer — for example a libp2p relay multiaddr. The kernel treats location hints as opaque:
it persists a bounded pool of them, embeds a few in issued [ocap URLs](#ocap-url) to aid
redemption, and hands them back to the netlayer on reconnect, but never parses them. Formerly
called "relays" on kernel-facing surfaces. See the [location-hint store
methods](../packages/ocap-kernel/src/store/methods/location-hints.ts).

### neutral peer id

The netlayer-independent identifier for a [kernel](#kernel): the base58btc multibase encoding
of the raw Ed25519 public key derived from the kernel's key seed. The kernel uses the neutral
peer id for all persisted remote state and as the host of the [ocap URL](#ocap-url); each
[netlayer](#netlayer) converts it to and from its own native identity at its boundary (a
libp2p `PeerId`, an iroh `NodeId` — both wrap the same raw Ed25519 key). A netlayer **must**
authenticate every peer as the neutral peer id derived from that peer's key seed. See the
[identity helpers](../packages/netlayer/src/identity.ts) in `@metamask/netlayer`.

### loopback netlayer

An in-process [netlayer](#netlayer) that connects kernels running in the same JavaScript
realm through an in-memory hub keyed by [neutral peer id](#neutral-peer-id), with no real
transport. It is the reference netlayer implementation: used as a test fake in place of
hand-rolled `PlatformServices` mocks and for embedded multi-kernel setups. Because it
exercises the full `Netlayer` contract (including the [incarnation
handshake](#incarnation-handshake)) it is the recommended baseline when writing or testing a
new netlayer. See `@metamask/netlayer-loopback`.

### hub-and-spoke

A network topology in which kernels ("spokes") connect to a central endpoint ("hub") rather
than to one another. The hub's role differs by netlayer: a libp2p relay is a _forwarding_ hub
(spokes reach each other through it), and the [loopback netlayer](#loopback-netlayer)'s
in-memory hub is the degenerate single-process case. (The planned plain-WebSocket server
netlayer's hub is instead a _kernel endpoint_ — spokes would talk to the hub kernel itself,
spoke↔spoke unsupported — but that netlayer is future work.) Contrast with direct/peer-to-peer
connections (e.g. libp2p QUIC/TCP, or a WebRTC upgrade), where peers dial each other after an
initial rendezvous.

### incarnation handshake

The exchange a [netlayer](#netlayer) performs when a connection is established, in which each
side reports its `incarnationId` — a value that changes when a kernel restarts. Reporting it
on every successful handshake (not only on change) lets the receiving kernel detect a peer
restart even after its own in-memory peer state was lost, and suppress stale in-flight
messages. The handshake message is versioned (a `v` field). See the shared
[`handshake.ts`](../packages/netlayer/src/handshake.ts) in `@metamask/netlayer` and the
`onIncarnationChange` hook in
[`remotes/types.ts`](../packages/ocap-kernel/src/remotes/types.ts).

### ocap URL

A capability-bearing URL of the form `ocap:{oid}@{host}[,{hint}]*` that references an object
in a [kernel](#kernel). The `host` is the issuing kernel's [neutral peer id](#neutral-peer-id);
`oid` is the object reference, AES-GCM-encrypted so only the issuing kernel can decode it; and
each optional `hint` is a [location hint](#location-hint) that helps a redeemer's
[netlayer](#netlayer) reach the host. Redeeming a remote ocap URL adds its hints to the
redeemer's own hint pool. See the ocap-URL logic in
[`remote-comms.ts`](../packages/ocap-kernel/src/remotes/kernel/remote-comms.ts).

## Abbreviations

### clist
Expand Down
32 changes: 24 additions & 8 deletions docs/identity-backup-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,21 +79,27 @@ const seed = await mnemonicToSeed(mnemonic);

### Kernel Initialization with Mnemonic

The `mnemonic` parameter can be passed either to `Kernel.make` (recommended) or to `initRemoteComms`:
The `mnemonic` parameter can be passed either to `Kernel.make` (recommended) or to `initRemoteComms`. Identity derivation is netlayer-independent — the examples below select the libp2p netlayer, whose location hints (`config.knownRelays`) are relay multiaddrs, but the mnemonic/seed → peer id behavior is the same under any netlayer:

```typescript
// Option 1: Pass mnemonic to Kernel.make (recommended)
const kernel = await Kernel.make(platformServices, kernelDatabase, {
mnemonic: 'your twelve word mnemonic phrase here ...',
});
await kernel.initRemoteComms({
relays: ['/ip4/127.0.0.1/tcp/9001/ws/p2p/12D3KooW...'],
specifier: {
netlayer: 'libp2p',
config: { knownRelays: ['/ip4/127.0.0.1/tcp/9001/ws/p2p/12D3KooW...'] },
},
});

// Option 2: Pass mnemonic to initRemoteComms
const kernel = await Kernel.make(platformServices, kernelDatabase);
await kernel.initRemoteComms({
relays: ['/ip4/127.0.0.1/tcp/9001/ws/p2p/12D3KooW...'],
specifier: {
netlayer: 'libp2p',
config: { knownRelays: ['/ip4/127.0.0.1/tcp/9001/ws/p2p/12D3KooW...'] },
},
mnemonic: 'your twelve word mnemonic phrase here ...',
});
```
Expand All @@ -120,7 +126,9 @@ console.log(mnemonic);
const kernel = await Kernel.make(platformServices, kernelDatabase, {
mnemonic,
});
await kernel.initRemoteComms({ relays });
await kernel.initRemoteComms({
specifier: { netlayer: 'libp2p', config: { knownRelays: relays } },
});

// The peer ID is now derived from the mnemonic and can be recovered
const status = await kernel.getStatus();
Expand All @@ -136,7 +144,9 @@ If you don't provide a mnemonic, the kernel generates a random identity that can
const kernel = await Kernel.make(platformServices, kernelDatabase, options);

// Initialize remote comms without mnemonic
await kernel.initRemoteComms({ relays });
await kernel.initRemoteComms({
specifier: { netlayer: 'libp2p', config: { knownRelays: relays } },
});

// Get the peer ID - this identity cannot be backed up as a mnemonic
const status = await kernel.getStatus();
Expand Down Expand Up @@ -166,7 +176,9 @@ const kernel = await Kernel.make(platformServices, kernelDatabase, {
});

// Initialize remote comms
await kernel.initRemoteComms({ relays });
await kernel.initRemoteComms({
specifier: { netlayer: 'libp2p', config: { knownRelays: relays } },
});

// The kernel now has the same peer ID as before
const status = await kernel.getStatus();
Expand Down Expand Up @@ -222,7 +234,9 @@ const kernel = await Kernel.make(platformServices, kernelDatabase, {
mnemonic: recoveryMnemonic,
});

await kernel.initRemoteComms({ relays });
await kernel.initRemoteComms({
specifier: { netlayer: 'libp2p', config: { knownRelays: relays } },
});
```

If you attempt to provide a mnemonic when an identity already exists without `resetStorage: true`, you'll get:
Expand Down Expand Up @@ -282,7 +296,9 @@ try {
const kernel = await Kernel.make(platformServices, kernelDatabase, {
mnemonic,
});
await kernel.initRemoteComms({ relays });
await kernel.initRemoteComms({
specifier: { netlayer: 'libp2p', config: { knownRelays: relays } },
});
} catch (error) {
if (error.message === 'Invalid BIP39 mnemonic') {
// Handle invalid mnemonic
Expand Down
44 changes: 22 additions & 22 deletions docs/ken-protocol-assessment.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,23 +134,23 @@ This achieves atomicity: if a crash occurs before commit, both the run queue ent

Ken enforces per-sender FIFO ordering via `next_ready()` which only delivers the next expected sequence number.

**Our situation**: We use TCP-based transports (libp2p streams) which guarantee in-order delivery during normal operation. Out-of-order arrival only occurs after a crash when the sender retransmits. With duplicate detection, retransmitted messages for already-processed sequence numbers are dropped, maintaining FIFO semantics.
**Our situation**: The kernel depends on the **netlayer delivery contract** — best-effort, ordered per live connection — rather than on any specific transport. Whatever netlayer is in use (libp2p, loopback, or a future one) guarantees FIFO within a single live connection; out-of-order or duplicate arrival can occur across a reconnect (or after a crash when the sender retransmits). With duplicate detection, retransmitted messages for already-processed sequence numbers are dropped, maintaining FIFO semantics regardless of transport.

Therefore, explicit receive-side reordering is not required given our transport guarantees.
Therefore, explicit receive-side reordering is not required given the netlayer's per-connection ordering guarantee. See the delivery-semantics contract in [writing a netlayer](./writing-a-netlayer.md#5-delivery-semantics-contract) for the normative statement of what a transport must provide.

### Summary Table

| Ken Property | Our System | Notes |
| --------------------- | ---------- | --------------------------------------------------- |
| Transactional turns | **Yes** | Crank buffering provides turn boundaries |
| Output validity | **Yes** | Outputs escape only after originating crank commits |
| Deferred transmission | **Yes** | Run queue staging ensures this |
| Atomic checkpoint | **Yes** | Database savepoints for kernel state |
| Consistent frontier | **Yes** | Each kernel's checkpoint is independent |
| Local recovery | **Yes** | Crashes don't affect other processes |
| Sender-based logging | **Yes** | Messages persisted in remotePending until ACKed |
| Exactly-once delivery | **Yes** | Transactional receive with dedup check |
| FIFO ordering | **Yes** | TCP guarantees in-order; dedup handles retransmits |
| Ken Property | Our System | Notes |
| --------------------- | ---------- | ------------------------------------------------------------------- |
| Transactional turns | **Yes** | Crank buffering provides turn boundaries |
| Output validity | **Yes** | Outputs escape only after originating crank commits |
| Deferred transmission | **Yes** | Run queue staging ensures this |
| Atomic checkpoint | **Yes** | Database savepoints for kernel state |
| Consistent frontier | **Yes** | Each kernel's checkpoint is independent |
| Local recovery | **Yes** | Crashes don't affect other processes |
| Sender-based logging | **Yes** | Messages persisted in remotePending until ACKed |
| Exactly-once delivery | **Yes** | Transactional receive with dedup check |
| FIFO ordering | **Yes** | Netlayer guarantees per-connection order; dedup handles retransmits |

## Architectural Summary

Expand Down Expand Up @@ -184,15 +184,15 @@ If a crash occurs before commit, both the run queue entry and the sequence updat

## Progress Summary

| Area | Status |
| ---------------------------------------- | ---------------------------- |
| Kernel-internal output buffering | **Achieved** (Issue #786) |
| Rollback discards uncommitted outputs | **Achieved** (Issue #786) |
| Atomic kernel state + output queue | **Achieved** (Issue #786) |
| Output validity (send side) | **Achieved** (Issue #786) |
| Deferred transmission (send side) | **Achieved** (Issue #786) |
| FIFO ordering | **Achieved** (TCP transport) |
| Exactly-once receive (dedup + atomicity) | **Achieved** (Issue #808) |
| Area | Status |
| ---------------------------------------- | ----------------------------------------------- |
| Kernel-internal output buffering | **Achieved** (Issue #786) |
| Rollback discards uncommitted outputs | **Achieved** (Issue #786) |
| Atomic kernel state + output queue | **Achieved** (Issue #786) |
| Output validity (send side) | **Achieved** (Issue #786) |
| Deferred transmission (send side) | **Achieved** (Issue #786) |
| FIFO ordering | **Achieved** (netlayer per-connection ordering) |
| Exactly-once receive (dedup + atomicity) | **Achieved** (Issue #808) |

All Ken protocol properties are now implemented.

Expand Down
Loading
Loading