Vortos
Messaging

Outbox Pattern

Guarantee at-least-once event delivery by writing events transactionally to the database before sending to Kafka.

Outbox Pattern

The transactional outbox pattern solves a fundamental problem in event-driven systems: how do you guarantee that a domain change and its corresponding event are always published together?

Without outbox, there is a race condition:

1. Write user to database    ← succeeds
2. Publish event to Kafka    ← Kafka is down → event is lost forever

Or in the opposite order:

1. Publish event to Kafka    ← succeeds
2. Write user to database    ← fails → event published for a change that didn't happen

The outbox pattern eliminates this by making the event write part of the same database transaction as the domain write:

BEGIN TRANSACTION
    1. Write user to database
    2. Write event to vortos_outbox table   ← same connection, same transaction
COMMIT (or ROLLBACK — both writes succeed or both fail)

[Separately, OutboxRelayWorker polls vortos_outbox]
    3. Read pending outbox rows
    4. Publish to Kafka
    5. Mark rows as published

Enabling Outbox

Outbox is enabled by default on all producers. Set outbox(true) explicitly or rely on the default:

#[RegisterProducer]
public function userProducer(): KafkaProducerDefinition
{
    return KafkaProducerDefinition::create('user.events')
        ->transport('user.events')
        ->publishes(UserRegistered::class)
        ->outbox(true);  // default — can be omitted
}

To bypass outbox and produce directly to Kafka (synchronously, no delivery guarantee):

->outbox(false)

Direct Production Risk

Disabling outbox means events can be lost if Kafka is unavailable when dispatch() is called. Only disable outbox for non-critical events where occasional loss is acceptable, or in development environments.

Database Table

You must create the outbox table in your database. Run this migration:

CREATE TABLE vortos_outbox (
    id                UUID          PRIMARY KEY,
    transport_name    VARCHAR(255)  NOT NULL,
    event_id          UUID          NOT NULL,
    aggregate_id      VARCHAR(255)  NOT NULL,
    aggregate_type    VARCHAR(512)  NOT NULL,
    aggregate_version INTEGER       NOT NULL,
    payload_type      VARCHAR(512)  NOT NULL,
    schema_version    INTEGER       NOT NULL DEFAULT 1,
    occurred_at       TIMESTAMP     NOT NULL,
    correlation_id    VARCHAR(255),
    causation_id      VARCHAR(255),
    trace_id          VARCHAR(255),
    metadata          JSONB,
    payload           TEXT          NOT NULL,
    status            VARCHAR(20)   NOT NULL DEFAULT 'pending',
    attempt_count     INTEGER       NOT NULL DEFAULT 0,
    created_at        TIMESTAMP     NOT NULL DEFAULT NOW(),
    published_at      TIMESTAMP,
    next_attempt_at   TIMESTAMP,
    failure_reason    TEXT
);

CREATE INDEX idx_vortos_outbox_status_next
    ON vortos_outbox (status, next_attempt_at)
    WHERE status = 'pending';

CREATE INDEX idx_vortos_outbox_aggregate_occurred
    ON vortos_outbox (aggregate_id, occurred_at);

CREATE INDEX idx_vortos_outbox_type_occurred
    ON vortos_outbox (payload_type, occurred_at);

Column reference:

ColumnDescription
event_idUUID of the specific event dispatch
aggregate_idID of the aggregate that raised the event
aggregate_typeFQCN of the aggregate class
aggregate_versionVersion of the aggregate at time of event
payload_typeFQCN of the POPO event class
schema_versionPayload schema version (default 1)
occurred_atWhen the event was recorded by the aggregate
correlation_idRequest correlation ID (from Metadata)
causation_idID of the command/event that caused this event
trace_idDistributed trace ID
metadataFull Metadata JSON (tenantId, userId, custom fields)
payloadJSON-serialized POPO

FOR UPDATE SKIP LOCKED

Both fetchPending (relay) and fetchFailed (replay) use SELECT ... FOR UPDATE SKIP LOCKED. Running multiple relay or replay workers in parallel is safe — each process locks the rows it is processing and other processes skip those rows entirely. No message is ever processed by two workers simultaneously.

How the Relay Works

The OutboxRelayWorker runs in a loop, polling the outbox table and producing pending messages to Kafka:

Poll for pending rows

SELECT * FROM vortos_outbox
WHERE status = 'pending'
  AND (next_attempt_at IS NULL OR next_attempt_at <= NOW())
ORDER BY created_at ASC
LIMIT 100
FOR UPDATE SKIP LOCKED

Produce to Kafka

For each pending row, deserialize the payload and call ProducerInterface::produce() to send to the Kafka topic specified in transport.

Mark as published

On success, update the row:

UPDATE vortos_outbox
SET status = 'published', published_at = NOW()
WHERE id = $id

Handle failure

On failure, a single atomic UPDATE increments attempt_count, computes the exponential backoff delay, and sets the status — all in one round-trip with no SELECT preceding it:

UPDATE vortos_outbox
SET
    attempt_count    = attempt_count + 1,
    failure_reason   = :reason,
    status           = CASE
        WHEN attempt_count + 1 >= :maxAttempts THEN 'failed'
        ELSE 'pending'
    END,
    next_attempt_at  = CASE
        WHEN attempt_count + 1 >= :maxAttempts THEN NULL
        ELSE NOW() + make_interval(secs => LEAST(:backoff_base * POWER(2, attempt_count + 1)::int, :backoff_cap))
    END
WHERE id = :id

Delay doubles on each attempt (backoffBase * 2^attempt), capped at backoffCap seconds. Both values are configurable — see Global Configuration below. When attempt_count reaches maxAttempts, the row is marked failed and next_attempt_at is cleared — it will not be picked up again by the relay and requires the vortos:outbox:replay command.

Running the Relay

The outbox relay runs as a separate long-lived process alongside your consumers. Add it to supervisord.conf:

docker/worker/supervisord.conf
[program:outbox-relay]
command=php /var/www/html/bin/console vortos:outbox:relay
autostart=true
autorestart=true
startsecs=3
stdout_logfile=/var/log/supervisor/outbox-relay.out.log
stderr_logfile=/var/log/supervisor/outbox-relay.err.log

Or run manually:

php bin/console vortos:outbox:relay
php bin/console vortos:outbox:relay --batch-size=50
php bin/console vortos:outbox:relay --sleep-ms=100

# dry run — lists pending rows without producing to Kafka
php bin/console vortos:outbox:relay --dry-run

Global Configuration

Outbox infrastructure settings are configured globally in your bootstrap via VortosMessagingConfig:

use Vortos\Messaging\DependencyInjection\VortosMessagingConfig;

$config = new VortosMessagingConfig();

$config->outbox()
    ->table('vortos_outbox')       // default — must match the table in your migration
    ->maxAttempts(5)               // default — rows failing beyond this are marked 'failed'
    ->backoffBase(30)              // default — initial backoff in seconds, doubles each attempt
    ->backoffCap(3600);            // default — backoff ceiling in seconds (1 hour)

These defaults are production-ready for most workloads. Tune maxAttempts and the backoff values to match your Kafka SLA and downstream service recovery times.

Inspecting Outbox State

Use vortos:outbox:list to inspect outbox rows without writing SQL:

# show all rows (pending + published + failed)
php bin/console vortos:outbox:list

# filter by status
php bin/console vortos:outbox:list --status=failed
php bin/console vortos:outbox:list --status=pending
php bin/console vortos:outbox:list --status=published

# filter by transport or event class
php bin/console vortos:outbox:list --transport=user.events
php bin/console vortos:outbox:list --event-class="App\\User\\Domain\\Event\\UserRegistered"

# show most recently created first, limit to 20
php bin/console vortos:outbox:list --latest --limit=20

Example output:

Found 2 outbox row(s) (failed).

  failed   UserRegistered  →  user.events    2min ago   agg: agg-001...  attempts: 5
           id: 019d4784-7b26-78a6-ba36-3b74ce5ad751
           reason: Connection refused

  pending  OrderPlaced     →  orders.placed  5s ago     agg: agg-002...  attempts: 0
           id: 019d4785-93cf-78a6-ba36-3b74ce5ad752

Tip: vortos:outbox:show <id> for full row details · vortos:outbox:replay to reset failed rows

Use vortos:outbox:show <id> for the full row — payload, all timestamps, metadata, and correlation IDs:

php bin/console vortos:outbox:show 019d4784-7b26-78a6-ba36-3b74ce5ad751

Replaying Failed Outbox Rows

When a relay attempt exhausts maxAttempts, the row is marked status = 'failed' and will not be picked up by the relay again. Use vortos:outbox:replay to reset these rows back to pending so the relay will retry them:

# inspect what will be replayed without making changes
php bin/console vortos:outbox:replay --dry-run

# replay all failed rows (prompts for confirmation)
php bin/console vortos:outbox:replay

# skip the confirmation prompt (safe for CI / cron)
php bin/console vortos:outbox:replay --force

# replay up to 50 rows (default: 50)
php bin/console vortos:outbox:replay --limit=50

# replay only rows for a specific transport
php bin/console vortos:outbox:replay --transport=user.events

# replay only a specific event class
php bin/console vortos:outbox:replay --event-class="App\\User\\Domain\\Event\\UserRegistered"

# replay a single specific row by ID
php bin/console vortos:outbox:replay --id=019d4784-7b26-78a6-ba36-3b74ce5ad751

# process the most recently failed rows first (default: oldest first)
php bin/console vortos:outbox:replay --latest

# replay rows created during a specific timestamp window
php bin/console vortos:outbox:replay --created-from="2026-05-11T10:00:00Z" --created-to="2026-05-11T10:30:00Z"

# combine flags freely
php bin/console vortos:outbox:replay --latest --limit=10 --transport=user.events --created-from="2026-05-11T10:00:00Z" --dry-run

--created-from and --created-to filter outbox rows by created_at and require full timestamps, not date-only values. The outbox table does not store a separate permanent-failure timestamp, so this selects rows by when the original outbox message was created.

Found 3 failed outbox row(s).

  • 019d4784-7b26-...  |  UserCreatedEvent  →  user.events  (attempts: 5)
  • 019d4784-93cf-...  |  OrderPlacedEvent  →  orders.placed  (attempts: 5)
  • 019d4785-1abc-...  |  UserCreatedEvent  →  user.events  (attempts: 5)

Reset: 3  Dry run: false

The command resets status back to pending, clears attempt_count to 0, and nulls failure_reason and next_attempt_at. The relay picks the rows up on the next poll cycle.

FOR UPDATE SKIP LOCKED on Replay

vortos:outbox:replay uses SELECT ... FOR UPDATE SKIP LOCKED when fetching failed rows. Running multiple replay workers in parallel is safe — each locks a non-overlapping batch.

Fix the Root Cause First

Resetting failed rows without fixing the underlying problem — Kafka unreachable, serialization error, misconfigured transport — will cause them to fail again and hit maxAttempts a second time. Investigate failure_reason before replaying.

Transaction Ownership

The outbox writer does not open its own transaction. It writes to the outbox table using the caller's existing database connection — which must already be inside a transaction.

This is correct behavior. The transaction boundary belongs to your application, not the framework:

// The TransactionalMiddleware wraps the entire handler in a transaction.
// Any outbox writes inside the handler are automatically included.

#[AsEventHandler(handlerId: 'order.placed.process', consumer: 'orders')]
final class ProcessOrderHandler
{
    public function __construct(
        private readonly OrderRepository $orders,
        private readonly EventBusInterface $eventBus,  // dispatches to outbox
    ) {}

    public function __invoke(OrderPlacedEvent $event): void
    {
        $order = $this->orders->find($event->orderId);
        $order->process();

        $this->orders->save($order);  // domain write

        // This outbox write happens in the SAME transaction as the save above.
        // If the transaction rolls back, neither write persists.
        $this->eventBus->dispatch(new OrderProcessedEvent($order->id()));
    }
}

Dispatch Outside a Handler

If you call $eventBus->dispatch() from a controller or service that is NOT wrapped in TransactionalMiddleware, you are responsible for the transaction boundary. Call $connection->beginTransaction() before your domain write and $connection->commit() after. If you dispatch without an active transaction, the outbox write will succeed independently of your domain write — defeating the purpose of the outbox.

Monitoring Outbox Health

Check for stuck or failed outbox rows:

-- Pending rows older than 5 minutes (relay may be down)
SELECT COUNT(*) FROM vortos_outbox
WHERE status = 'pending' AND created_at < NOW() - INTERVAL '5 minutes';

-- Failed rows that need attention
SELECT * FROM vortos_outbox
WHERE status = 'failed'
ORDER BY created_at DESC
LIMIT 20;

Set up an alert if pending rows older than your relay's expected processing time accumulate — this indicates the relay worker is down or unable to reach Kafka.

Idempotency

Prevent duplicate handler execution when Kafka redelivers messages using Redis-backed idempotency tracking.

Retry & Dead Letter

Automatic retry with exponential backoff and dead letter storage — Vortos handles failure recovery so your handlers don't have to.

On this page

Outbox PatternEnabling OutboxDatabase TableHow the Relay WorksPoll for pending rowsProduce to KafkaMark as publishedHandle failureRunning the RelayGlobal ConfigurationInspecting Outbox StateReplaying Failed Outbox RowsTransaction OwnershipMonitoring Outbox Health