coreci/atelier
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 28 Wiki Activity

docs(milestone): complete v0.4 — Edge + Messaging + Language-Derived Docs

Browse Source 
---ci---
project: atelier
phase: 0
milestone: v0.4
status: complete
requirements:
  covered: [ATELIER-92, ATELIER-93, ATELIER-94, ATELIER-95, ATELIER-96, ATELIER-97, ATELIER-98, ATELIER-99, ATELIER-100, ATELIER-101, ATELIER-102, ATELIER-103, ATELIER-104, ATELIER-105, ATELIER-106, ATELIER-107, ATELIER-108, ATELIER-109, ATELIER-110, ATELIER-111, ATELIER-112, ATELIER-113, ATELIER-114, ATELIER-115, ATELIER-116, ATELIER-117]
  partial: []
---/ci---
This commit is contained in:
Jon Chery
2026-08-05 16:23:15 +00:00
parent 9ebc9c8868
commit 29ffb42898
53 changed files with 8404 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
domains/messaging/delivery-semantics.md
+379
View File
@@ -0,0 +1,379 @@
# Delivery Semantics — Derived Rules
> Derives from `domains/messaging/first-principles.md`. Applies P4
> (Delivery Semantics are Explicit), P3 (Consumers are Idempotent),
> P2 (Ordering is a Property, Not an Assumption), P5 (Dead-Letter
> Handling is Defined), and P6 (Backpressure is Bounded) primarily,
> with P10 (DLQ depth as an alert). For the dead-letter strategy
> decision, see the comparison table below. Includes a fenced
> idempotency-key dedup-store example (IDEATE-39). Cross-links
> `domains/concurrency/patterns` for the in-process retry/backoff
> analog, `domains/errors/patterns` for errors as data for message
> failures, and `domains/observability/metrics` for DLQ depth as an
> alert.
## The Three Delivery Semantics (P4 Delivery Semantics are Explicit)
- **At-most-once**: a message is delivered 0 or 1 times; loss is
possible, duplication is not. The producer fires-and-forgets; the
broker does not ack; the consumer does not dedup. Lowest latency,
lowest implementation cost, lossy. Fits telemetry where a dropped
sample is acceptable (MQTT QoS 0, fire-and-forget metrics).
- **At-least-once**: a message is delivered 1 or more times;
duplication is possible, loss is not. The producer sends and
waits for a broker ack; the consumer processes and acks; a crash
before the consumer's ack triggers redelivery. The consumer must
be idempotent (P3). The default for side-effecting operations
(orders, payments, commands). The vast majority of broker-backed
queues (SQS standard, RabbitMQ ack, MQTT QoS 1).
- **Exactly-once**: a message is delivered exactly 1 time; no
loss, no duplication. In practice, this is at-least-once plus
idempotency (P3), or a transactional consume-process-produce loop
(P4 — see `domains/messaging/streams.md`). Jepsen analyses verify
broker claims: "exactly-once" requires independent verification;
the durable engineering practice is at-least-once with idempotent
consumers (P3), which collapses to exactly-once under correct
dedup.
- The semantic is declared per channel (P4), not emergent. An
unstated semantic is a defect: the consumer guesses, and the
guess is wrong under the first failure. See
`domains/messaging/queues.md` for the three-semantics comparison
table (semantics, latency cost, implementation cost, when each
fits).
## Idempotency (P3 Consumers are Idempotent)
- Idempotency is the correctness property that makes at-least-once
safe. A consumer that processes the same message twice has the
same effect as processing it once. The mechanism is the
idempotency key: a per-message unique identifier the consumer
uses to dedup redeliveries.
- The idempotency key is per-message, not per-producer or
per-session. A consumer that dedups by producer alone drops
distinct messages issued in the same window. Use a UUID per
message, or a deterministic key derived from the message content
(e.g., `(entity, operation, version)`).
- The dedup store is bounded (P6 — Backpressure is Bounded): a
dedup store that grows without bound is a memory leak. Use a TTL
window longer than the broker's max-redelivery window, or a
bounded LRU. The TTL is the P6 bound: a key seen within the TTL
is a redelivery; a key older than the TTL is expired (the broker
has given up redelivering it).
## Idempotency-Key Dedup Store (IDEATE-39, P3 — fenced example)
The dedup store is the concrete mechanism that makes a consumer
idempotent under at-least-once delivery. This is the fenced
example required by IDEATE-39 (parallel to the v0.3 IDEATE-29
signed-attestation fenced example): it is NOT prose-only — the
consumer-with-dedup-store demonstrates P3 concretely.
```python
# Idempotent consumer with a dedup store (P3 Consumers are
# Idempotent, P6 Backpressure is Bounded — the dedup store is
# TTL-bounded). The consumer dedups by idempotency key before
# processing; a redelivered message is a no-op, not a double-apply.
import time
# P6: the dedup store is bounded by a TTL window. The TTL must
# exceed the broker's max-redelivery window; beyond the TTL, the
# key is expired (the broker has given up). A dedup store with no
# TTL is a memory leak (P6 violation).
DEDUP_TTL_SECONDS = 24 * 3600 # longer than max-redelivery window
class DedupStore:
"""A TTL-bounded idempotency-key dedup store (P3, P6).
seen(key): True if the key was processed within the TTL.
mark(key): Record the key as processed (with a timestamp).
"""
def __init__(self, backend):
# backend is a Redis, a DB, or an in-process LRU. The
# backend must be shared across consumer instances if the
# subscription is shared (P2/P3 — see domains/messaging/
# pubsub.md on shared vs independent subscriptions).
self.backend = backend
def seen(self, key: str) -> bool:
ts = self.backend.get(key)
if ts is None:
return False
if time.time() - ts > DEDUP_TTL_SECONDS:
# P6: expired. The broker has given up redelivering;
# this key is no longer a redelivery signal.
self.backend.delete(key)
return False
return True
def mark(self, key: str):
self.backend.set(key, time.time(), ttl=DEDUP_TTL_SECONDS)
# The idempotent consumer: dedup before process, mark after
# process, ack after mark. A crash before mark re-processes (the
# dedup store does not have the key); a crash before ack
# redelivers (the broker did not see the ack) and the dedup store
# makes the redelivery a no-op (P3).
def consume_idempotent(broker, dedup: DedupStore, process):
for message in broker.receive():
# P3: dedup BEFORE process. A redelivered message is a
# no-op, not a double-apply.
if dedup.seen(message["idempotencyKey"]):
broker.ack(message) # already processed; skip
continue
try:
process(message["payload"])
# P3: mark AFTER process succeeds. A crash between
# process and mark re-processes (acceptable: the
# process must be idempotent OR the mark must be
# transactional with the process — see below).
dedup.mark(message["idempotencyKey"])
broker.ack(message)
except TransientError as exc:
# P6: bounded retry with backoff. Nack for redelivery;
# the broker redelivers after exponential backoff.
broker.nack(message, delay=backoff(message["attempt"]))
except PoisonError as exc:
# P5: poison message. Route to DLQ, do NOT retry
# forever (see the DLQ routing rule below).
route_to_dlq(broker, message, exc)
broker.ack(message) # remove from the origin queue
```
- The order `process → mark → ack` gives at-least-once with
idempotent dedup: a crash before `mark` re-processes (the dedup
store does not have the key), and a crash before `ack`
redelivers (the dedup store makes the redelivery a no-op). If
`process` is not itself idempotent, the `process → mark` window
must be transactional (e.g., process and mark in one DB
transaction) — otherwise a crash in the window double-applies.
- For a non-idempotent `process` (e.g., a payment that must not
double-charge), use a transactional dedup: process and mark in
one DB transaction, so the mark commits iff the process
commits. This is the "exactly-once via idempotency" pattern
(P4): at-least-once delivery plus a transactional
process-and-mark collapses to exactly-once under correct
transactional semantics.
## Ordering (P2 Ordering is a Property, Not an Assumption)
- The delivery semantic interacts with ordering (P2). At-least-once
with per-partition ordering: a redelivery within a partition
preserves order (the redelivered message re-appears in its
original position relative to other messages the consumer has
not yet seen). At-least-once with no ordering: a redelivery may
appear out of order relative to messages delivered after it.
- The consumer must not assume an ordering property the broker
does not provide (P2). A standard queue delivers per-receive-node
arrival order but no global order and no order across redeliveries;
a FIFO queue delivers strict per-group order including across
redeliveries; a partitioned stream delivers strict per-partition
order across redeliveries. Document the property; do not assume
it. See `domains/messaging/queues.md` (FIFO vs standard) and
`domains/messaging/streams.md` (per-partition order).
## Dead-Letter Strategies (P5 Dead-Letter Handling is Defined)
- A poison message (unparseable, repeatedly failing, or exhausting
the retry budget) must be routed to a dead-letter queue, not
retried forever or silently dropped (P5). The DLQ is observable
(P10 — depth is an alert) and drainable (an operator can
inspect, replay, or discard with audit).
- The dead-letter strategy determines when a message is
dead-lettered and what the operator sees. The choice is the
decision matrix below (D-069).
## Dead-Letter Strategy Comparison (D-069)
| Strategy | When It Applies | Failure Visibility | Operational Cost |
|----------|-----------------|-------------------|------------------|
| **Retry-count-limit** | A fixed max-redeliveries count (e.g., 5). After N redeliveries, route to DLQ. Simple, predictable. | The redelivery count is visible in the DLQ entry; the operator sees how many times it was retried. | Low — a counter per message; no backoff tuning. Risk: retries fire as fast as the broker redelivers, hammering a downstream that is already failing (P6 — no backoff = no backpressure escape). |
| **TTL-with-backoff** | A max time-to-live for redelivery (e.g., 30 minutes) with exponential backoff between retries. After the TTL, route to DLQ. | The TTL and the backoff schedule are visible; the operator sees the retry timeline. | Medium — backoff tuning per message type. Benefit: backoff gives the downstream time to recover (P6 — the retry rate is bounded); fits transient failures (a downstream that is briefly unavailable). |
| **Poison-queue** | A separate queue for messages that fail a specific check (unparseable, schema-invalid, unknown type) before any processing retry. Routed immediately, not retried. | The poison queue is a separate signal from the DLQ; the operator sees parse-vs-process failures distinctly. | Low — a routing rule per check. Benefit: distinguishes "never going to succeed" (poison) from "might succeed on retry" (DLQ). Use for unparseable messages that no retry will fix. |
| **DLQ + alert** | Any of the above strategies, plus an alert on DLQ depth. The DLQ is observable (P10) — depth, age, and rate are alerted. | Highest — the operator is paged on DLQ growth; the DLQ is a first-class signal, not a graveyard. | Medium — alerting setup per DLQ. This is the P5/P10 floor: a DLQ without an alert is a silent correctness defect (poison messages accumulate invisibly). |
- The default for transient failures is **TTL-with-backoff + DLQ
+ alert**: backoff gives the downstream time to recover (P6),
the TTL bounds the retry budget (P5), the DLQ captures the
unprocessable, and the alert makes it visible (P10). The default
for unparseable messages is **poison-queue + alert**: route
immediately, do not retry a message no retry will fix.
- **Retry-count-limit alone** (no backoff, no alert) is the
`messaging-unbounded-retry` chaos anti-pattern's cousin: it
caps the count but hammers the downstream at full retry rate,
and the DLQ grows silently if no alert is wired. Always pair
a retry budget with backoff (P6) and an alert (P10).
- The failure-visibility column is the P10 check: every strategy
must surface the failure to the operator. A strategy with no
visibility is a P10 violation regardless of its retry semantics.
- The operational-cost column is the C8 tradeoff: more visibility
and more backoff cost more to set up but pay back in operational
stability. The P5/P10 floor is "DLQ + alert"; below that, the
strategy is a silent defect waiting to grow.
## DLQ Routing Rule (P5, P10)
```python
# DLQ routing rule (P5 dead-letter handling, P6 bounded retry with
# backoff, P10 DLQ depth alert). Combines TTL-with-backoff for
# transient failures and poison-queue for unparseable messages,
# with an alert on DLQ depth.
import json, time
MAX_RETRY_TTL_SECONDS = 30 * 60 # 30 min total retry window
DLQ = "orders-dlq"
POISON = "orders-poison" # unparseable; never retried
def route_to_dlq(broker, message, reason):
"""Route a message to the DLQ with audit metadata (P5)."""
broker.send(DLQ, body=json.dumps({
"original": message,
"reason": str(reason),
"deadLetteredAt": now_iso(),
"redeliveryCount": message.get("attempt", 0),
}))
# P10: emit a metric so DLQ depth alerts fire. A DLQ that
# grows with no alert is a silent correctness defect (P5/P10).
metrics.increment("dlq.depth", tags={"queue": "orders"})
# The ack removes the message from the origin queue; the DLQ
# is the durable record (P5 — observable and drainable).
def consume_with_dlq(broker, dedup, process):
for message in broker.receive():
# P1: parse first. An unparseable message is poison —
# route immediately, do NOT retry (no retry will fix it).
try:
payload = json.loads(message["body"])
except (ValueError, SchemaError) as exc:
broker.send(POISON, body=json.dumps({
"original": message["body"],
"reason": f"parse-failed: {exc}",
"poisonedAt": now_iso(),
}))
metrics.increment("poison.depth", tags={"queue": "orders"})
broker.ack(message) # remove from origin; poison queue holds it
continue
# P3: dedup before process.
if dedup.seen(payload["idempotencyKey"]):
broker.ack(message); continue
attempt = payload.get("attempt", 0)
first_attempt_ts = payload.get("firstAttemptTs", time.time())
try:
process(payload)
dedup.mark(payload["idempotencyKey"])
broker.ack(message)
except TransientError as exc:
# P6: TTL-with-backoff. If the retry window is
# exhausted, route to DLQ; otherwise redeliver with
# exponential backoff.
if time.time() - first_attempt_ts > MAX_RETRY_TTL_SECONDS:
route_to_dlq(broker, message, exc) # P5
broker.ack(message)
else:
broker.nack(message, delay=backoff(attempt))
except PermanentError as exc:
# A permanent error (e.g., a not-found dependency)
# does not benefit from retry — route to DLQ now.
route_to_dlq(broker, message, exc)
broker.ack(message)
```
- The routing rule distinguishes three failure modes: **poison**
(unparseable — route immediately, no retry), **transient**
(retry with backoff until the TTL, then DLQ), and **permanent**
(a retry will not fix it — DLQ now). This distinction is the P5
discipline: not every failure is a retry; some are immediate
DLQs.
- The DLQ entry carries `reason`, `deadLetteredAt`, and
`redeliveryCount` — it is auditable (the operator knows why each
message was dead-lettered and how many times it was retried).
This is the errors-as-data discipline — see
`domains/errors/patterns` for the general principle a DLQ entry
instantiates.
## Retry Budgets and Backoff (P6 Backpressure is Bounded)
- The retry budget is the cap on redelivery: a count, a TTL, or
both. After the budget, the message routes to the DLQ (P5). An
unbounded retry budget is the `messaging-unbounded-retry` chaos
anti-pattern (P5 breach): the consumer never makes progress past
the poison message.
- **Exponential backoff** spaces retries: 1s, 2s, 4s, 8s, ... with
a jitter to avoid thundering-herd synchrony. Backoff gives the
downstream time to recover (P6 — the retry rate is bounded,
giving the downstream a chance to catch up). A retry with no
backoff hammers the downstream at full rate, making the failure
worse.
- The retry budget × the backoff schedule is the P6 bound: the
consumer's retry load is bounded by design, not by luck. See
`domains/concurrency/patterns` Pattern 6 (Timeout on Every
Block) for the in-process retry/backoff analog; messaging owns
the broker-backed instance where the redelivery comes from the
broker across a network, not an in-process loop (D-062).
## Poison Messages (P5, P1)
- A poison message is one no retry will fix: unparseable (the
schema is wrong, P1), unknown type (the consumer does not handle
this version, P9), or a permanent failure (a not-found
dependency). Retrying a poison message wastes resources and
blocks the queue (P6 — the consumer never makes progress).
- Poison messages route to the poison queue immediately (no
retry), distinct from the DLQ (which holds messages that
exhausted their retry budget on transient failures). The
distinction is the P5 discipline: a poison queue is for
"never going to succeed"; a DLQ is for "might have succeeded
but didn't within the budget."
- A poison queue without an alert is the same defect as a DLQ
without an alert (P10): the operator cannot see the poison
accumulating. Wire both to `domains/observability/metrics`.
## Cross-Link to Concurrency (P6, cross-link concurrency/patterns)
- The retry/backoff discipline here is the cross-process analog
of `domains/concurrency/patterns` Pattern 6 (Timeout on Every
Block) for in-process retry. Concurrency owns the in-process
analog (a retry loop within one program, with a timeout per
attempt); messaging owns the broker-backed instance (the broker
redelivers across a network, the consumer applies backoff via
nack-with-delay). The failure model differs: in-process retry
fails by a thread crash or a timeout; broker-backed retry fails
by network partition, broker restart, or consumer crash-and-retry
(D-062).
- The cross-link is one-directional outward (messaging →
concurrency) per D-026 extended: messaging references concurrency
as the in-process foundation; concurrency does not back-link to
messaging.
## Cross-Link to Errors (P5, cross-link errors/patterns)
- A poison message is an errors-as-data instance: the failure is
captured as a DLQ entry (with `reason`, `redeliveryCount`,
`deadLetteredAt`), not swallowed. See `domains/errors/patterns`
for the general errors-as-data discipline a DLQ entry
instantiates. The DLQ is the async-messaging instance of an
error log — observable, auditable, drainable.
- The cross-link is one-directional outward (messaging → errors):
messaging references errors for the errors-as-data pattern;
errors does not back-link to messaging.
## What Violates Delivery-Semantics Discipline
| Violation | Principle |
|-----------|-----------|
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
| Dedup store with no TTL (memory leak; grows without bound) | P6 Backpressure is Bounded |
| Dedup key per-producer (distinct messages in the same window deduped) | P3 Consumers are Idempotent |
| Retry with no backoff (hammers the downstream at full rate) | P6 Backpressure is Bounded |
| Unbounded retry budget (consumer never progresses past the poison) | P5 Dead-Letter Handling is Defined |
| DLQ with no depth alert (poison messages accumulate invisibly) | P10, `domains/observability/metrics` |
| Poison message retried forever (no poison queue, no immediate DLQ) | P5 Dead-Letter Handling is Defined |
| Process-not-idempotent with non-transactional mark (crash in window double-applies) | P3, P4 |
| DLQ entry with no reason/audit metadata (uninspectable failure) | P5, `domains/errors/patterns` |
| Ordering assumption the broker does not provide (FIFO assumed on standard queue) | P2 Ordering is a Property, Not an Assumption |