= Hub linking (federation) =

uhub can link several hub instances together so that they appear to clients as
one logical hub: users connected to any node see a single unified user list and
can chat, search, send private messages and set up peer-to-peer connections
across node boundaries.

This document describes the hub-to-hub link protocol and the SID-space
partitioning it depends on. It reflects the current implementation, which is
EXPERIMENTAL -- see "Status and limitations" at the end before deploying.

Links are a separate concern from the client-facing ADC protocol described in
architecture.txt. A linked peer is treated as a privileged node, not as a
client: it can inject users into the local roster, so links must only be made
between trusted hubs (see "Security").


== Topology ==

The intended model is IRC-style server linking over a full mesh: every node
holds a persistent authenticated link to every other node, and exchanges
presence and messages so the whole cluster shares one roster and one SID
namespace.

The current implementation is validated for two nodes (and a star of members
linking to one coordinator). Multi-hop relay across a 3+ node mesh is not yet
implemented -- a message is forwarded at most one hop.

The hard ceiling is the ADC SID: it is 20 bits, so the entire logical hub --
local users plus every linked node's users -- shares a single space of
1,048,576 session IDs (SID 0 is reserved for the hub). This bounds a cluster
to roughly 1M users total regardless of node count.


== Transport and detection ==

A link reuses the hub's normal listening port; there is no separate link port.
The connecting node opens an ordinary TCP connection and sends "LCHA" as its
first bytes. The probe in src/core/probe.c recognises this prefix -- exactly as
it recognises "HSUP" for an ADC client or a TLS ClientHello -- and hands the
connection to link_accept() instead of the client login path.

Messages are newline-terminated text. Every command is a four-character code
("FOURCC"), mirroring ADC's framing of one context character ('L' for link)
plus a three-letter command, optionally followed by a space and arguments.

Links can also run over a Unix domain socket instead of TCP. A hub listens on
one if link_socket is set, and a link_peer that begins with "/" connects to
such a socket. This is used to link worker processes of one logical hub on a
single machine (see "Multi-process hubs") without routing link traffic through
the shared client port. The link protocol is identical over either transport.


== Multi-process hubs (one logical hub on many cores) ==

uhub is single-threaded, so one process uses one core. To use a whole machine,
run the hub as several worker processes that link to each other and present one
logical hub -- the same federation machinery, but within one host. Set:

  workers = N        (or 0 for one worker per CPU)

The top-level process then becomes a master that forks N workers and supervises
them (restarting any that exit); it serves no clients itself. Each worker is a
normal hub, configured automatically:

  - all workers bind the same server_port with SO_REUSEPORT, so the kernel
    load-balances incoming client connections across them (across cores);
  - they link to each other in a full mesh over Unix sockets in a private
    runtime directory (worker_socket_dir, default /tmp), with an ephemeral
    shared secret generated by the master (never written to disk);
  - the SID space is partitioned per worker (node_id = worker index,
    node_count = N), so SIDs stay globally unique.

A client connecting to the single port lands on some worker and sees the whole
cluster's roster; the CPU-bound per-connection work (notably TLS) is spread
across cores. The trade-offs are the federation ones: each worker holds the
full roster (memory ~N x user state), and public chat crosses worker boundaries
(one relayed copy per worker, fanned out in parallel). The ~1M SID ceiling is
cluster-wide regardless of worker count.

SO_REUSEPORT is enabled automatically when workers > 1. The standalone
server_reuseport option exists for the advanced case of supervising the worker
processes yourself (e.g. a systemd template); leaving it off by default avoids
two unrelated hubs silently sharing a port.


== Authentication handshake ==

Both endpoints authenticate each other with a symmetric mutual
challenge-response keyed on a shared secret (the "link_secret" config option).
Neither side trusts the other until both directions have been verified.

  response = base32( tiger( secret || peer_nonce ) )

Each side generates a random nonce, challenges the peer with it, answers the
peer's challenge, and acknowledges a verified answer. The exchange (shown for
one direction; both sides do the same concurrently):

    ----------------------              ----------------------
    | connect (TCP)      |  ----------> | probe sees "LCHA",  |
    | send LCHA <nonceA> |              | calls link_accept() |
    ----------------------              ----------------------
              |                                   |
              |    <----  LCHA <nonceB>  ---------+   (peer challenges us)
              |                                   |
              +----  LRES base32(tiger(secret||nonceB))  ----->
              |                                   |
              |    <----  LRES base32(tiger(secret||nonceA))
              |          verify against nonceA    |
              |                                   |
              +----  LACK  (we verified peer) --------------->
              |    <----  LACK  (peer verified us)
              V                                   V
       ------------------------------------------------
       | ESTABLISHED: peer_verified && got_ok == true |
       ------------------------------------------------

A node is "established" only once it has BOTH verified the peer's LRES (proving
the peer knows the secret) AND received the peer's LACK (proving the peer
verified our answer). A failed verification sends LERR and closes the link.

The response is compared in constant time. The nonce is drawn from the TLS
library CSPRNG. nonce and response are base32, MAX_CID_LEN characters each.


== Command reference ==

  Command                    Dir.    Meaning
  -------------------------  ------  ---------------------------------------
  LCHA <nonce>               both    Challenge: authenticate me with this nonce.
  LRES <response>            both    Response to the peer's challenge.
  LACK                       both    Peer's response verified; ack.
  LERR                       both    Authentication failed / request refused.
  LELC <id>                  both    Election id (16 hex digits); lower wins.
  LWRQ                       member  Request a SID-window lease.
  LWIN <min> <max>           coord   Grant the SID window [min,max] (decimal).
  LINF <binf>                both    Roster entry / presence: a user's BINF.
  LQUI <sid>                 both    A user left (4-char base32 SID).
  LRTE <adc-message>         both    Forward an ADC message to be re-routed.
  LDSC <description>         both    Hub description (topic) changed.
  LBAN <cid> <nick>          both    Ban a user cluster-wide (and disconnect).
  LACQ <sid> <nick>          slave   Is this nick registered? (master auth)
  LACR <sid> <cred>          master  Reply: the nick's credential ("guest" if not).
  LVFY <sid> <chal> <resp> <nick>  slave   Verify a password challenge-response.
  LVRS <sid> <ok> <cred>     master  Reply: 1/0, and the credential on success.

All post-handshake commands (everything except LCHA/LRES/LACK/LERR) are only
accepted on an established link. Every command is a 4-character FOURCC ('L' plus
a 3-letter code); fields that may contain spaces (a nick, a description, a
wrapped ADC message) are always last and run to the end of the line.


== SID-space partitioning ==

Because the whole cluster shares one 1M SID space, each node must allocate
local SIDs from a disjoint slice ("window") so that two nodes never hand out
the same SID. The space is divided into "node_count" equal windows of
SID_MAX / node_count SIDs each; window 0 starts at SID 1 (SID 0 is reserved).
A node's lookup map still spans the whole space, so remote users from other
windows resolve through the same table.

There are three ways a node obtains its window, selected by the "node_id"
config option:

  node_id >= 0   Static. The node owns window node_id and allocates from it
                 immediately. Every node must agree on node_count and use a
                 unique node_id. node_id 0 is the cluster coordinator.

  node_id = -1   Dynamic. The node starts PENDING: it has an empty window and
                 accepts no local logins until it leases a window over a link
                 (see below). node_count still sets the window size.

A coordinator (the election winner, see below) tracks which windows are in use
and leases free ones to members:

    member (pending)                    coordinator
         |                                   |
         |  ------------  LWRQ  ----------->  |  pick lowest free window k
         |                                   |  mark k used
         |  <--------  LWIN <min> <max>  ---  |  [min,max] = window k
         |  apply window; accept logins      |
         |                                   |
         (on link drop, the coordinator returns window k to the free pool)

If no window is free the coordinator answers LERR and the member stays pending.

=== Coordinator election ===

The coordinator is not statically pinned: nodes elect one per link. Each
participating node has a 64-bit election id, exchanged with LELC right after
the handshake; the LOWER id wins and acts as coordinator for that link, the
loser leases from it.

  node_id = 0    election id 0 -- always wins (a deterministic, "forced"
                 coordinator, for clusters that want a fixed leader).
  node_id = -1   a random election id -- elects dynamically.
  node_id > 0    does not participate (it has a fixed static window).

A pending winner (node_id = -1 that wins) claims window 0 for itself at
election time, then grants the other windows. This lets an entire cluster come
up with NO node designated as coordinator: give every node node_id = -1,
node_count = N and a shared link_secret, and they elect among themselves.


== Roster synchronisation and remote users ==

On establishing a link, each node streams its local roster to the peer: one
"LINF <binf>" per locally-connected user, carrying that user's full BINF. The
receiver injects each as a REMOTE user -- it appears in the SID map, the nick
and CID maps, the user list and the user count, exactly like a local user, but
has no socket of its own; it exists only as presence state owned by the link it
arrived on.

Live changes propagate as deltas over the same commands:

  - a local user joining or updating its INF   -> LINF to every link
  - a local user leaving                        -> LQUI <sid> to every link

An LINF for a SID already known on the receiver is treated as an update; an
LQUI removes the remote user and tells local clients it quit.


== Message routing across links ==

Directed messages (ADC 'D'/'E' context: private messages, search results,
connect/RCM, NAT traversal) to a user that is remote are forwarded over the
link that owns the target, wrapped in LRTE:

  LRTE <full ADC message including its trailing newline>

The receiving node parses the wrapped message and re-routes it locally with the
normal routing path, delivering it to the target's socket.

Locally-originated broadcasts (ADC 'B'/'F' context: public chat and searches)
are relayed once over every link, also via LRTE. Presence (BINF) is excluded --
it uses the LINF/LQUI roster path above. To prevent loops, a node never
re-relays a broadcast whose source is a remote user: a broadcast that arrived
over a link is delivered to local clients only. This is loop-safe for two hops.


== State synchronisation (topic, bans) ==

Some hub state must look the same on every node. Live changes are propagated
over the links (each receiver applies it locally without re-propagating, which
is loop-free on a full mesh):

  - Topic: setting the hub description (the !topic command -> set_description)
    sends LDSC <description> to every link; each node updates its own IINF and
    tells its local clients.

  - Bans: the !ban command adds the user's CID and nick to the local ACL,
    disconnects a matching local user, and sends LBAN <cid> <nick> to every
    link. Each node adds the ban and disconnects its own matching session, so
    the user is dropped wherever connected and cannot reconnect on any node
    (login already consults the ACL). This makes a ban a cluster-wide kick too.

State that is file-backed -- chat history and the user database when stored in
SQLite -- is shared automatically among co-located workers because they read
the same config and therefore the same files. Cross-machine nodes do not share
a filesystem; authentication for them is handled by the master-slave scheme
below, and ban/topic state by the propagation above.

Note: only live changes propagate today. A node that links in *later* does not
yet receive the existing bans or the current topic (an initial snapshot on link
establish is a follow-up). The topic is also seeded consistently from each
node's hub_description config.


== Master-slave authentication ==

Cross-machine nodes each have their own config, so they cannot share an account
database by sharing a file. Rather than replicate accounts (uhub stores
passwords in plaintext, which should not be copied to every node), one node is
the auth master and the others are slaves that defer registered-user login to
it. A slave sets:

  auth_proxy = 1

and holds no accounts of its own. Its upstream link (link_peer) is the master.
A login on the slave is proxied in two steps; passwords never leave the master:

    slave (auth_proxy)                 master (has accounts)
        |                                   |
   BINF |  --------  LACQ <sid> <nick>  --> |  look up the nick
        |  <-------  LACR <sid> <cred>  --- |  reply credential ("guest" if none)
        |  (guest -> admit; registered -> challenge with IGPA)
        |                                   |
   HPAS |  -- LVFY <sid> <chal> <resp> <nick> -->  verify against stored password
        |  <-------  LVRS <sid> <ok> <cred> ----   reply pass/fail + credential
        |  (admit with that credential, or reject)

The slave pauses the login while waiting for each reply (a pending-auth registry
keyed by SID tracks the held login; the existing login timeout covers an
unreachable master). Guests are unaffected and log in locally. The master needs
no special flag -- it simply answers from its auth plugin -- so it is just a
normal hub that loads the account database (mod_auth_simple / mod_auth_sqlite).


== Netsplit ==

When a link drops, the node removes every remote user it learned over that link
and broadcasts a quit (IQUI) for each to its local clients -- the classic IRC
netsplit. Window leases granted over that link are returned to the free pool.
Rosters re-synchronise when the link is re-established.


== Configuration ==

Relevant uhub.conf options:

  link_peer      Peer(s) to connect to, comma-separated. Each is either a
                 host:port (TCP) or a path beginning with "/" (Unix socket).
                 When set, the node links out on start-up. Empty = accept only.
  link_secret    Shared secret for the mutual challenge-response. Must match on
                 both ends. Empty disables linking.
  link_socket    Unix socket path to also accept links on (for same-host
                 worker links). Must be short (sun_path is ~108 bytes).
  node_count     Number of windows the SID space is divided into (cluster size,
                 or a fixed over-provisioned ceiling). Must agree across nodes.
  node_id        This node's window index: >= 0 static, 0 = forced coordinator,
                 -1 = lease a window dynamically (and participate in election).
  auth_proxy     Slave auth: defer registered-user login to the upstream master.
  workers        Run one logical hub as N worker processes (0 = one per CPU).
  worker_socket_dir  Directory for the inter-worker Unix sockets (default /tmp).
  server_reuseport   Set SO_REUSEPORT on the listen socket (auto-on when
                     workers > 1; set by hand only if you supervise workers).

Recommended deployment: pick a generous fixed node_count ceiling (e.g. 64) on
every node so windows never need re-slicing as the cluster grows, and either
assign unique static node_ids or set node_id = -1 everywhere and let the nodes
elect and lease. For a single multi-core machine, set workers and let the master
configure the rest.

If linking is configured but node_count is 1 (and node_id >= 0), the hub warns
at start-up that SIDs are not partitioned and will collide.


== Security ==

A link is highly privileged: a peer can inject arbitrary users into the local
roster. Treat the link as a trust boundary.

The shared-secret handshake is a v1 baseline with the same weaknesses as a
shared password: one secret for the whole cluster, no per-hub identity, no
rotation or revocation, and it sits in the config file. Run links only over a
trusted/private network.

The intended production posture is mutual TLS with per-peer certificate pinning
(each hub has a keypair; peers pin each other's fingerprint), giving per-node
identity and revocation with nothing secret on the wire or at rest. Do not
treat the shared secret as the recommended long-term mechanism.


== Status and limitations ==

Implemented and tested (two nodes / star-to-a-coordinator, and multi-process):
  - probe-based link detection and the mutual LCHA/LRES/LACK handshake, over
    TCP or a Unix socket
  - static and dynamic SID partitioning; window leasing (LWRQ/LWIN)
  - coordinator election (LELC)
  - roster snapshot and live join/update/quit deltas (LINF/LQUI)
  - directed and broadcast routing across one hop (LRTE)
  - topic propagation (LDSC) and cluster-wide bans (LBAN)
  - master-slave authentication (LACQ/LACR/LVFY/LVRS, auth_proxy)
  - multi-process "one logical hub" via workers (SO_REUSEPORT + Unix-socket mesh)
  - netsplit cleanup on link drop

Not yet implemented:
  - Multi-hop relay and loop tags for a 3+ node mesh (only 2 hops are
    loop-safe today).
  - Globally consistent election across a mesh (election is per-link).
  - Coordinator failover: window leases are held in memory, so a coordinator
    restart forgets them; there is no lease persistence or re-lease protocol.
  - Initial-state snapshot on link establish: a node that links in later does
    not receive existing bans or the current topic (only live changes
    propagate). Runtime bans are in-memory (not written to file_acl).
  - The auth slave proxies to its first established upstream link (single-master
    topology); there is no auth-master election or failover.
  - TLS-wrapped links: detection is plaintext-only; a TLS link needs a
    post-handshake re-probe.
  - Reconnect/backoff: a failed outbound link is attempted once.
  - Cluster-wide nick/CID collision precedence (the propagation-window race),
    and aggregate hub counts across nodes.
  - mutual-TLS / certificate-pinned link authentication.

See the source in src/core/link.c and src/core/probe.c.
