OpenParable
Skip to content
← Back to skills

Writing Data

Development
writing-data

Writing Data is an Agent Skill for development published in the durable-streams/durable-streams repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Writing data to durable streams. DurableStream.create() with contentType, DurableStream.append() for simple writes, IdempotentProducer for high-throughput exactly-once delivery with autoClaim, fire-and-forget append(), flush(), close(), StaleEpochError handling, JSON mode vs byte stream mode, stream closure. Load when writing, producing, or appending data to a durable stream.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/durable-streams/durable-streams/blob/HEAD/packages/client/skills/writing-data/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/writing-data, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

This skill builds on durable-streams/getting-started. Read it first for setup and offset basics.

Durable Streams — Writing Data

Two write APIs: DurableStream.append() for simple writes, IdempotentProducer for sustained high-throughput writes with exactly-once delivery. Use the producer for anything beyond a few one-off appends.

Setup

import {
  DurableStream,
  IdempotentProducer,
  StaleEpochError,
} from "@durable-streams/client"

// Create a JSON-mode stream
const handle = await DurableStream.create({
  url: "https://your-server.com/v1/stream/my-stream",
  contentType: "application/json",
})

// Set up IdempotentProducer for reliable writes
const producer = new IdempotentProducer(handle, "my-service", {
  autoClaim: true,
  onError: (err) => {
    if (err instanceof StaleEpochError) {
      console.log("Another producer took over")
    } else {
      console.error("Write error:", err)
    }
  },
})

// Fire-and-forget writes — automatically batched and deduplicated
for (const event of events) {
  producer.append(JSON.stringify(event))
}

// Ensure all pending writes are delivered
await producer.flush()
await producer.close()

Core Patterns

Simple writes with append()

For writing a few items and waiting for completion, append() is straightforward:

import { DurableStream } from "@durable-streams/client"

const handle = await DurableStream.create({
  url: "https://your-server.com/v1/stream/events",
  contentType: "application/json",
})

// Each append waits for server confirmation
await handle.append({ type: "order.created", orderId: "123" })
await handle.append({ type: "order.paid", orderId: "123" })

High-throughput writes with IdempotentProducer

For sustained writes, the producer batches, pipelines, and deduplicates automatically:

import { DurableStream, IdempotentProducer } from "@durable-streams/client"

const handle = await DurableStream.create({
  url: "https://your-server.com/v1/stream/tokens",
  contentType: "text/plain",
})

const producer = new IdempotentProducer(handle, "llm-worker", {
  autoClaim: true,
  lingerMs: 10, // Batch window (default: 5ms)
  maxBatchBytes: 65536, // Max batch size (default: 1MB)
  maxInFlight: 4, // Concurrent HTTP requests (default: 5)
  onError: (err) => console.error(err),
})

for await (const token of llm.stream(prompt)) {
  producer.append(token) // Fire-and-forget — don't await
}

await producer.flush() // Wait for all batches to land
await producer.close() // Clean up

Closing a stream

Mark a stream as permanently closed (no more writes accepted):

// Close with optional final message
await handle.close({ body: JSON.stringify({ type: "stream.complete" }) })

// Or close without a final message
await handle.close()

Byte stream mode with custom framing

For non-JSON streams, use your own framing (e.g., newline-delimited JSON):

const handle = await DurableStream.create({
  url: "https://your-server.com/v1/stream/logs",
  contentType: "text/plain",
})

const producer = new IdempotentProducer(handle, "logger", { autoClaim: true })

producer.append(JSON.stringify({ level: "info", msg: "started" }) + "\n")
producer.append(JSON.stringify({ level: "error", msg: "failed" }) + "\n")

await producer.flush()

Common Mistakes

CRITICAL Using raw append() for sustained writes

Wrong:

const handle = await DurableStream.create({
  url,
  contentType: "application/json",
})
for (const event of events) {
  await handle.append(JSON.stringify(event)) // No dedup, no batching, sequential
}

Correct:

const handle = await DurableStream.create({
  url,
  contentType: "application/json",
})
const producer = new IdempotentProducer(handle, "my-service", {
  autoClaim: true,
  onError: (err) => console.error(err),
})
for (const event of events) {
  producer.append(JSON.stringify(event)) // Fire-and-forget, batched, deduplicated
}
await producer.flush()

Raw append() has no deduplication — on retry after network error, data may be duplicated. IdempotentProducer handles batching, pipelining, and exactly-once delivery.

Source: packages/client/src/idempotent-producer.ts

CRITICAL Not calling flush() before shutdown

Wrong:

for (const event of events) {
  producer.append(event)
}
// Process exits — pending batch lost!

Correct:

for (const event of events) {
  producer.append(event)
}
await producer.flush()
await producer.close()

IdempotentProducer batches writes. Without flush(), pending messages in the buffer are lost when the process exits.

Source: packages/client/src/idempotent-producer.ts

HIGH Awaiting each producer.append() call

Wrong:

for (const event of events) {
  await producer.append(JSON.stringify(event)) // Defeats pipelining!
}

Correct:

for (const event of events) {
  producer.append(JSON.stringify(event)) // Fire-and-forget
}
await producer.flush() // Wait for all to complete

append() is fire-and-forget by design. Awaiting it serializes every write and defeats batching and pipelining. Errors go to the onError callback.

Source: packages/client/src/idempotent-producer.ts

HIGH Passing objects to append instead of strings

Wrong:

producer.append({ event: "user.created" }) // throws!

Correct:

producer.append(JSON.stringify({ event: "user.created" }))

IdempotentProducer.append() accepts only string or Uint8Array — it does not auto-serialize objects, even for JSON-mode streams. Always call JSON.stringify() before appending.

Source: packages/client/src/idempotent-producer.ts

HIGH Not handling StaleEpochError for multi-worker scenarios

Wrong:

const producer = new IdempotentProducer(handle, "worker", {
  onError: (err) => console.error(err), // Logs but doesn't stop
})

Correct:

let fenced = false
const producer = new IdempotentProducer(handle, "worker", {
  onError: (err) => {
    if (err instanceof StaleEpochError) {
      fenced = true // Stop producing
    }
  },
})

for await (const token of source) {
  if (fenced) break
  producer.append(token)
}

When another producer claims a higher epoch, the current producer gets fenced. Without checking, your write loop continues appending to a dead producer.

Source: packages/client/src/idempotent-producer.ts

HIGH Manually managing epoch numbers

Wrong:

const producer = new IdempotentProducer(handle, "worker", { epoch: 2 })

Correct:

const producer = new IdempotentProducer(handle, "worker", {
  autoClaim: true, // Handles epoch transitions automatically
})

Epoch and sequence management is an internal protocol detail. Use autoClaim: true and let the producer handle transitions automatically.

Source: maintainer guidance

HIGH Tension: Simplicity vs. reliability in writes

This skill's patterns conflict with getting-started. Getting-started simplicity favors append() for quick demos, but production code should use IdempotentProducer for exactly-once guarantees. Agents optimizing for a quick setup may use append() and miss deduplication and batching.

See also: durable-streams/getting-started/SKILL.md § Common Mistakes

See also

Version

Targets @durable-streams/client v0.2.1.