iii-queue

iii-queue

The iii-queue worker provides asynchronous job processing with retries, configurable concurrency, FIFO ordering, and dead-letter (DLQ) support. It runs in two modes. Topic-based queues are durable pub/sub: register a consumer with a durable:subscriber trigger and produce with iii::durable::publish — every distinct function subscribed to a topic receives a copy of each message (fan-out). Named queues are defined in config (queue_configs) and targeted by enqueuing a function call with TriggerAction.Enqueue — no trigger registration needed, the target function is the consumer.

Three adapters back delivery: builtin (in-process; in_memory or file_based; single-instance; retries + DLQ + FIFO), rabbitmq (durable delivery, retries, and DLQ across instances), and redis (multi-instance topic pub/sub only — it publishes to named queues but does not implement named-queue consumption, retries, or DLQ). Choose builtin for local and single-instance, rabbitmq for multi-instance production.

When to Use

• A unit of work should run asynchronously with automatic retries and dead-letter capture on repeated failure.
• You need strict ordering within a key (FIFO) — payments, ledger entries, state machines — via a fifo queue with message_group_field.
• Durable fan-out where every subscribed consumer must process every event (unlike iii-pubsub, which is fire-and-forget).
• You need to inspect or recover stuck work: browse a DLQ and redrive or discard messages.

Boundaries

• Not fire-and-forget broadcast — for ephemeral notifications where missed events are fine, use iii-pubsub.
• The redis adapter is publish-only for named queues; named-queue consumption, retries, and DLQ require builtin or rabbitmq.
• FIFO queues force prefetch=1 (one job at a time) and require message_group_field present and non-null in every payload; they trade throughput for ordering and retry inline (blocking the group until success or DLQ).
• This worker carries work items, not key/value or stream state — use iii-state / iii-stream for those.

Functions

• iii::durable::publish — publish a message to a topic; fanned out to every distinct subscribed function. Empty topic returns topic_not_set.
• iii::queue::redrive — move all messages from a named queue's DLQ back to the main queue.
• iii::queue::redrive_message — move a single DLQ message back to the main queue by id.
• iii::queue::discard_message — purge a single DLQ message by id.
• engine::queue::list_topics, engine::queue::topic_stats, engine::queue::dlq_topics, engine::queue::dlq_messages — console/inspection helpers: enumerate topics, read per-topic stats, list DLQ topics with counts, and browse DLQ messages.

Reactive triggers

Bind a durable:subscriber trigger when a function should durably consume every message published to a topic, with retries and dead-letter handling. Distinct functions on the same topic each receive every message (fan-out); replicas of the same function compete for messages.

Reach for it when:

• A topic event must be processed reliably — a consumer that was offline should still get the message once it returns.
• You want retries and a DLQ around the consumer rather than the best-effort delivery of iii-pubsub.

For a single target function with retries (not fan-out), skip the trigger and enqueue the call directly with TriggerAction.Enqueue({ queue }) against a queue defined in queue_configs.

How to bind

1. Register a handler: iii.registerFunction('orders::process', handler).
2. Register the trigger:

iii.registerTrigger({
  type: 'durable:subscriber',
  function_id: 'orders::process',
  config: {
    topic: 'orders.created',  // topic to consume.
    // optional per-trigger queue_config (max_retries, concurrency, type, ...).
  },
})

Per-queue tuning (in queue_configs or queue_config): max_retries (default 3), concurrency (default 10; FIFO forces 1), type (standard | fifo), message_group_field (required for fifo), backoff_ms (default 1000, exponential), poll_interval_ms (default 100).

For the message payload shape, call iii get function info on the trigger type or handler function id.