docs(milestone): complete v0.4 — Edge + Messaging + Language-Derived Docs
---ci--- project: atelier phase: 0 milestone: v0.4 status: complete requirements: covered: [ATELIER-92, ATELIER-93, ATELIER-94, ATELIER-95, ATELIER-96, ATELIER-97, ATELIER-98, ATELIER-99, ATELIER-100, ATELIER-101, ATELIER-102, ATELIER-103, ATELIER-104, ATELIER-105, ATELIER-106, ATELIER-107, ATELIER-108, ATELIER-109, ATELIER-110, ATELIER-111, ATELIER-112, ATELIER-113, ATELIER-114, ATELIER-115, ATELIER-116, ATELIER-117] partial: [] ---/ci---
This commit is contained in:
Jon Chery
@@ -0,0 +1,307 @@
|
||||
# Messaging — First Principles
|
||||
|
||||
## 1. The Principles
|
||||
|
||||
### P1. Messages are Contracts
|
||||
A message has an explicit, versioned schema. Producer and consumer
|
||||
agree on shape before exchange; the schema is the boundary, not a
|
||||
guess. A schemaless message — a free-form JSON blob the consumer
|
||||
parses by hope — is a defect: the consumer breaks silently on the
|
||||
next shape change, and the producer has no contract to evolve
|
||||
against. This derives from `C1 Correctness` (the exchange must
|
||||
carry what the parties agreed to) and `C2 Clarity` (the schema
|
||||
makes the boundary obvious to both sides). This is the
|
||||
cross-process expression of the contract discipline that
|
||||
`domains/api/rest` owns for synchronous request/response: the
|
||||
message schema is to async exchange what the API contract is to
|
||||
sync exchange. It is distinct from
|
||||
`domains/concurrency/patterns` Pattern 1 (Message Passing), which
|
||||
owns the *in-process* channel primitive — here the contract spans
|
||||
separate systems and survives network failure (D-062). See
|
||||
`domains/messaging/queues.md` for the queue-flavored application
|
||||
and `domains/messaging/streams.md` for the durable-log-flavored
|
||||
application.
|
||||
|
||||
### P2. Ordering is a Property, Not an Assumption
|
||||
Ordering guarantees — per-partition strict, global, or none — are
|
||||
explicit and documented. "It's FIFO" is a claim that must be backed
|
||||
by the broker's partitioning contract, not an assumption the
|
||||
consumer makes and the broker may not honor. A standard queue
|
||||
delivers in arrival order per receive-node but offers no global
|
||||
ordering across shards; a FIFO queue delivers strict per-message-
|
||||
group order but at a latency cost; a partitioned stream delivers
|
||||
strict per-partition order but only within a partition. Each is a
|
||||
distinct, declared property. This derives from `C1 Correctness`
|
||||
(order is a correctness property — a consumer that assumes order
|
||||
the broker does not provide is wrong) and `C2 Clarity` (the
|
||||
ordering guarantee is documented, not discovered in production).
|
||||
This is distinct from in-process ordering, which
|
||||
`domains/concurrency/patterns` Pattern 1 owns for channels within
|
||||
one program: messaging ordering survives network failure, broker
|
||||
restart, and consumer crash-and-retry — a stronger failure model
|
||||
than thread-local channels (D-062). The `messaging-shared-
|
||||
subscription` chaos anti-pattern breaches this rule: two consumers
|
||||
sharing one subscription break per-consumer ordering because the
|
||||
broker dispatches each message to an arbitrary consumer. See
|
||||
`domains/messaging/streams.md` for the partition-order contract
|
||||
and `domains/messaging/delivery-semantics.md` for the interaction
|
||||
of ordering with the three delivery semantics.
|
||||
|
||||
### P3. Consumers are Idempotent
|
||||
Delivery is at-least-once by default across the network; a
|
||||
consumer deduplicates via idempotency keys or deterministic
|
||||
processing. "Exactly-once" is idempotency plus at-least-once, not a
|
||||
broker guarantee — Jepsen analyses of Kafka, RabbitMQ, and NATS
|
||||
establish that exactly-once claims require independent
|
||||
verification, and the durable engineering practice is to make
|
||||
consumers idempotent under redelivery. A non-idempotent consumer
|
||||
under at-least-once delivery doubles the effect on every retry; a
|
||||
non-idempotent consumer under a claimed exactly-once broker is a
|
||||
bug waiting for the broker's exactly-once invariant to break. This
|
||||
derives from `C1 Correctness`: correctness under redelivery is the
|
||||
contract, not a nice-to-have. This parallels
|
||||
`domains/edge/P5 Edge Operations are Idempotent` (the
|
||||
cross-partition device-and-cache-flavored analog) and is the
|
||||
cross-process instance of the retry-safety discipline that
|
||||
`domains/concurrency/patterns` Pattern 6 (Timeout on Every Block)
|
||||
implies for in-process retry. It is distinct from in-process
|
||||
retry because the redelivery comes from the broker across a
|
||||
network, not from an in-process loop (D-062). See
|
||||
`domains/messaging/delivery-semantics.md` for the idempotency-key
|
||||
dedup-store pattern.
|
||||
|
||||
### P4. Delivery Semantics are Explicit
|
||||
At-least-once / at-most-once / exactly-once is a declared choice
|
||||
per channel, not an emergent behavior. The tradeoff — latency cost,
|
||||
implementation complexity, operational cost — is made consciously
|
||||
and documented. At-most-once is fire-and-forget (low latency, lossy);
|
||||
at-least-once is acked with possible duplication (the default,
|
||||
requires idempotent consumers per P3); exactly-once is at-least-once
|
||||
plus idempotency or a transactional two-phase commit (highest cost,
|
||||
narrowest fit). An unstated semantic is a defect: the consumer
|
||||
guesses, and the guess is wrong under the first failure. This
|
||||
derives from `C1 Correctness` (the chosen semantic must hold) and
|
||||
`C2 Clarity` (the tradeoff is visible to the reader and the
|
||||
operator). This is the cross-process analog of the explicit-failure-
|
||||
mode discipline that `domains/errors/patterns` owns for synchronous
|
||||
code — messaging makes the delivery-mode choice as explicit as an
|
||||
error-handling choice. See `domains/messaging/queues.md` for the
|
||||
three-semantics comparison table and `domains/messaging/delivery-
|
||||
semantics.md` for the correctness properties of each.
|
||||
|
||||
### P5. Dead-Letter Handling is Defined
|
||||
Poison messages — unparseable, repeatedly failing, or exhausting
|
||||
the retry budget — are routed to a dead-letter queue, not retried
|
||||
forever or silently dropped. The DLQ is observable and drainable: an
|
||||
operator can inspect it, replay from it, or discard with audit. An
|
||||
unbounded retry loop is a livelock: the consumer never makes
|
||||
progress past the poison message. A silent drop is a correctness
|
||||
defect: the message vanished with no record. This derives from `C1
|
||||
Correctness` (poison messages must not livelock the consumer or
|
||||
silently disappear) and `C5 Reversibility` (the DLQ is the
|
||||
reversibility mechanism — a dead-lettered message can be reprocessed
|
||||
after the bug is fixed). This is the cross-process analog of the
|
||||
bounded-error discipline that `domains/errors/patterns` owns for
|
||||
synchronous code: a poison message is an error-as-data instance
|
||||
that must be observable and recoverable, not swallowed. It is
|
||||
distinct from in-process error handling because the failure spans a
|
||||
network and a consumer restart (D-062). See
|
||||
`domains/messaging/delivery-semantics.md` for the dead-letter
|
||||
strategy comparison table and the DLQ routing rule pattern.
|
||||
|
||||
### P6. Backpressure is Bounded
|
||||
A slow consumer cannot unbounded-buffer the broker or the
|
||||
producer. Backpressure is explicit: consumer lag is visible,
|
||||
max-unacked is bounded, the retry budget is capped. A consumer
|
||||
that falls behind without a visible signal is a silent backlog —
|
||||
the operator cannot fix what they cannot see, and the broker's
|
||||
memory grows without bound until it fails. This derives from `C1
|
||||
Correctness` (a backlog that grows until OOM is a correctness
|
||||
failure) and `C8 Economy` (the broker's memory is bounded by
|
||||
design, not by luck). This is distinct from
|
||||
`domains/concurrency/P9 Bounded Queues` and
|
||||
`domains/concurrency/patterns` Pattern 5 (Bounded Queue with
|
||||
Backpressure), which own the *in-process* analog: concurrency's
|
||||
bounded queue fails by OOM or thread crash; messaging's bounded
|
||||
backpressure fails by network partition, broker restart, or
|
||||
consumer crash-and-retry (D-062). The Reactive Streams
|
||||
specification (`request(n)`, `onNext` bounded) is the in-process
|
||||
instance; messaging's broker-backed backpressure is the
|
||||
cross-process instance above it. See `domains/messaging/queues.md`
|
||||
for prefetch and max-unacked and `domains/observability/metrics`
|
||||
for consumer-lag as an alert.
|
||||
|
||||
### P7. Partitioning is Intentional
|
||||
The partition key determines ordering, parallelism, and hotspots.
|
||||
Key choice is a design decision with documented rationale, not a
|
||||
default. A key that hashes unevenly creates a hot partition that
|
||||
limits throughput; a key that does not match the ordering need
|
||||
breaks per-key semantics; a key that is too coarse (one partition
|
||||
for the whole topic) serializes all the traffic. The partition
|
||||
count is a capacity bound: too few partitions cap parallelism, too
|
||||
many partition overhead the broker. This derives from `C4 Locality`
|
||||
(ordering and parallelism are co-located with the partition) and
|
||||
`C6 Composability` (the partition is the unit of parallelism and
|
||||
scaling — consumer groups compose from per-partition workers).
|
||||
This is the cross-process analog of the locality discipline that
|
||||
`domains/performance/` owns for generic data-near-compute
|
||||
optimization: performance's locality is algorithmic (data near
|
||||
compute); messaging's locality is partitional (order and
|
||||
parallelism near the partition). See `domains/messaging/streams.md`
|
||||
for the partitioned-log model and consumer-group rebalance
|
||||
strategies.
|
||||
|
||||
### P8. Replay and Retention are Configured
|
||||
Retention windows and replay-from-offset are explicit. A message
|
||||
is not ephemeral by default; the broker is a durable log, not a
|
||||
pipe. A topic with no retention is a fire-and-forget stream — a
|
||||
consumer that falls behind loses data permanently; a topic with
|
||||
infinite retention is an unbounded log — the broker grows until
|
||||
disk exhaustion. Both are defects: the retention window is a
|
||||
declared bound, and replay-from-offset is the mechanism that makes
|
||||
the log durable (re-consumable) rather than ephemeral. This derives
|
||||
from `C5 Reversibility` (a retained message is reversible — it can
|
||||
be re-consumed; an ephemeral message is not) and `C7 Observability`
|
||||
(the durable log is itself an observable record of what happened —
|
||||
the offset is the position from which to replay). This is the
|
||||
foundation for `domains/messaging/streams.md` and the rule that
|
||||
distinguishes a stream from a queue (a queue deletes on ack; a
|
||||
stream retains for replay). See `domains/messaging/pubsub.md` for
|
||||
the pub/sub-vs-stream durability boundary.
|
||||
|
||||
### P9. Schemas Evolve Compatibly
|
||||
Schema changes are backward- and forward-compatible by
|
||||
construction. Breaking changes are versioned migrations, not
|
||||
silent shape edits. A producer that ships a new field the old
|
||||
consumer ignores is backward-compatible; a consumer that handles a
|
||||
missing field the new producer omits is forward-compatible. A
|
||||
silent schema change — the producer renames a field and the
|
||||
consumer parses `undefined` — is a P1 violation (the contract was
|
||||
broken) compounded here as an evolution defect. This derives from
|
||||
`C5 Reversibility` (a schema change is reversible by versioning —
|
||||
the old shape is still readable) and `C6 Composability` (producers
|
||||
and consumers of different versions compose because the schema
|
||||
evolves compatibly). This parallels `domains/data/migrations`
|
||||
(schema migration for databases) and `domains/api/versioning`
|
||||
(API contract evolution): messaging's schema evolution is the
|
||||
async instance of the same compatibility discipline. See
|
||||
`domains/messaging/streams.md` for the stream-schema-evolution
|
||||
angle.
|
||||
|
||||
### P10. Messaging is Observable
|
||||
Consumer lag, DLQ depth, throughput, and consumer-group health are
|
||||
first-class signals. Silent backlog is a bug, not a feature: a
|
||||
consumer that falls behind with no lag metric is invisible until
|
||||
the downstream effect surfaces — by which time the backlog may be
|
||||
hours or days. A DLQ that grows without an alert is a silent
|
||||
correctness defect: poison messages are accumulating and no one
|
||||
knows. This derives from `C7 Observability` (the broker's behavior
|
||||
is visible to the operator) and `C1 Correctness` (backlog
|
||||
detection is a correctness bound — unbounded lag is a failure).
|
||||
This is distinct from `domains/observability/metrics`, which owns
|
||||
*generic* structured metrics; messaging owns the *broker-specific*
|
||||
signals — lag, DLQ depth, partition imbalance, consumer-group
|
||||
rebalance events. See `domains/observability/metrics` for the
|
||||
generic SLI/SLO discipline and `domains/observability/tracing` for
|
||||
cross-partition traces.
|
||||
|
||||
## 2. Core Principle Trace
|
||||
|
||||
Each messaging P-rule derives from one or more core C-rules
|
||||
(C1–C8). The matrix extension lands in P4 of the v0.4 plan; the
|
||||
traces below are authoritative. Messaging is a broad-derivation
|
||||
domain touching 7 of 8 core principles (C1, C2, C4, C5, C6, C7,
|
||||
C8); C3 (Simplicity) is not a primary derivation — messaging is
|
||||
inherently a tradeoff domain where simplicity yields to the
|
||||
correctness of delivery guarantees (a simpler-than-necessary
|
||||
delivery model does not handle the failure cases, per C3's
|
||||
"simpler than necessary is also a violation").
|
||||
|
||||
| P-rule | Core | Why |
|
||||
|--------|------|-----|
|
||||
| P1 Messages are Contracts | C1, C2 | Correctness of the exchange; clarity of the schema boundary |
|
||||
| P2 Ordering is a Property, Not an Assumption | C1, C2 | Correctness of order; clarity of the guarantee |
|
||||
| P3 Consumers are Idempotent | C1 | Correctness under redelivery |
|
||||
| P4 Delivery Semantics are Explicit | C1, C2 | Correctness of the chosen semantic; clarity of the tradeoff |
|
||||
| P5 Dead-Letter Handling is Defined | C1, C5 | Correctness of poison-message routing; reversibility of reprocessing |
|
||||
| P6 Backpressure is Bounded | C1, C8 | Correctness of bounded backlog; economy of broker memory |
|
||||
| P7 Partitioning is Intentional | C4, C6 | Locality of order; composability of parallelism |
|
||||
| P8 Replay and Retention are Configured | C5, C7 | Reversibility of replay; observability of the durable log |
|
||||
| P9 Schemas Evolve Compatibly | C5, C6 | Reversibility of schema changes; composability of versions |
|
||||
| P10 Messaging is Observable | C7, C1 | Observability of lag/DLQ; correctness of backlog detection |
|
||||
|
||||
## 3. What Violates These Principles
|
||||
|
||||
| Violation | Principle Breached |
|
||||
|-----------|-------------------|
|
||||
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts |
|
||||
| "It's FIFO" with no documented partition contract | P2 Ordering is a Property, Not an Assumption |
|
||||
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
|
||||
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
|
||||
| No dead-letter queue (poison message retried forever or silently dropped) | P5 Dead-Letter Handling is Defined |
|
||||
| Unbounded retry budget (no cap; slow consumer stalls the partition) | P6 Backpressure is Bounded |
|
||||
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
|
||||
| Ephemeral broker (no retention; no replay) | P8 Replay and Retention are Configured |
|
||||
| Silent schema change (producer breaks consumers with no version bump) | P9 Schemas Evolve Compatibly |
|
||||
| Silent backlog (no lag metric; consumer falls behind invisibly) | P10 Messaging is Observable |
|
||||
| Shared subscription (two consumers share one subscription; per-consumer ordering breaks) | P2 Ordering is a Property, Not an Assumption (P3 compounding) |
|
||||
| Blocking consumer (slow downstream call with no timeout; broker redelivers to the stuck consumer) | P6 Backpressure is Bounded |
|
||||
|
||||
## 4. Relationship to Other Domains
|
||||
|
||||
Messaging systems are the engineering discipline of
|
||||
**cross-process, cross-system asynchronous communication via
|
||||
brokers**. Producer and consumer are separate systems; the broker
|
||||
is the intermediary that brokers delivery, ordering, retention,
|
||||
and failure semantics. The distinguishing constraints are a
|
||||
cross-process failure model (network, not crash), explicit
|
||||
delivery semantics, decoupled producer/consumer lifecycle, and
|
||||
replay-and-retention as a durable-log property. Messaging overlaps
|
||||
`domains/concurrency/` by *subject* (messages, queues,
|
||||
backpressure) but not by *failure model*: per D-062, messaging
|
||||
owns the cross-process/network-failure-model angle; concurrency
|
||||
owns the in-process/crash-failure-model angle. The discriminator
|
||||
is the failure model: concurrency's queue fails by OOM or thread
|
||||
crash; messaging's queue fails by network partition, broker
|
||||
restart, or consumer crash-and-retry. Messaging extends
|
||||
concurrency's bounded-queue/backpressure model to the network-
|
||||
partition regime. Cross-links are one-directional outward (per
|
||||
D-026 extended); no back-link edits to v0.1/v0.2/v0.3 content.
|
||||
|
||||
- `domains/concurrency/patterns` ← P6 (the broker-backed bounded
|
||||
queue is the cross-process analog of the in-process bounded
|
||||
buffer — concurrency Pattern 5 owns in-process; messaging owns
|
||||
the network-failure-model instance above it, per D-062)
|
||||
- `domains/concurrency/patterns` ← P3 (idempotent retry is the
|
||||
cross-process analog of in-process retry-safety — the failure
|
||||
model differs: broker redelivery across a network vs in-process
|
||||
loop)
|
||||
- `domains/observability/metrics` ← P10 (consumer lag and DLQ
|
||||
depth as alerts; observability owns the generic SLI/SLO
|
||||
discipline, messaging owns the broker-specific signals)
|
||||
- `domains/observability/tracing` ← P10 (cross-partition traces
|
||||
for stream processing; observability owns the generic tracing
|
||||
discipline, messaging owns the cross-partition propagation)
|
||||
- `domains/data/schema-design` ← P1, P9 (message schema design
|
||||
and evolution; data owns the generic schema discipline,
|
||||
messaging owns the cross-process message-shape instance)
|
||||
- `domains/errors/patterns` ← P5 (errors as data for message
|
||||
failures; a poison message is an error-as-data instance that must
|
||||
be observable and recoverable, not swallowed)
|
||||
- `domains/edge/iot` ← P4 (the edge↔messaging cross-link
|
||||
resolves bidirectionally here: edge/iot.md links outward to
|
||||
messaging/queues for MQTT QoS parallels to delivery semantics;
|
||||
this first-principles doc acknowledges the back-link — the
|
||||
edge/iot.md → messaging/queues link from P1 now resolves because
|
||||
messaging/queues.md exists, completing the bidirectionality per
|
||||
IDEATE-40)
|
||||
|
||||
> Note: the edge/iot.md → messaging/queues cross-link (MQTT QoS
|
||||
> parallels for delivery semantics) was authored in P1 with a
|
||||
> dangling reference; this P2 authorship of messaging/queues.md
|
||||
> resolves it. The bidirectionality is verified in P5
|
||||
> (ATELIER-114 per IDEATE-40). The cross-link is one-directional
|
||||
> outward from edge/iot.md; this first-principles doc
|
||||
> acknowledges the resolution without editing edge/iot.md (per
|
||||
> D-026 extended — no back-link edits to v0.1/v0.2/v0.3 or to
|
||||
> P1-authored edge content).
|
||||
Reference in New Issue
Block a user