How to Handle Retries and Sagas in Temporal

Getting a Temporal workflow to run is the easy part. Keeping it correct when a third-party API dies halfway through a five-step process is where most teams stumble. This covers retry policies, the saga pattern for rollback, and how to change a running workflow without breaking history.

It assumes you already have a worker and activities running.

How do Temporal retry policies work?

Every activity has a RetryPolicy. By default Temporal retries with exponential backoff indefinitely, which is rarely what you want for a payment call.

ao := workflow.ActivityOptions{
  StartToCloseTimeout: time.Minute,
  RetryPolicy: &temporal.RetryPolicy{
    InitialInterval:    time.Second,
    BackoffCoefficient: 2.0,
    MaximumAttempts:    5,
  },
}

Set MaximumAttempts and a NonRetryableErrorTypes list for errors that should fail fast (bad input, auth failure). Retrying a validation error just burns attempts.

How do I implement a saga (compensation) pattern?

A saga runs compensating activities in reverse when a later step fails. In Temporal you do this with a defer-style cleanup, using a disconnected context so the compensation runs even if the workflow is cancelled.

1. Book the flight (activity A).
2. Reserve the hotel (activity B).
3. Charge the card (activity C).
4. If C fails, run CancelHotel then CancelFlight in reverse order.

Wrap the compensations in workflow.NewDisconnectedContext so a cancellation still triggers rollback. Without it, a cancelled workflow skips cleanup and leaves the booking dangling.

How do I change a workflow that is already running?

Determinism means you cannot just edit workflow code — old histories will fail replay. Use workflow.GetVersion to branch by version.

v := workflow.GetVersion(ctx, "add-email-step", workflow.DefaultVersion, 1)
if v >= 1 { /* new step */ }

Old workflows take the default path, new ones take v1. Temporal records which branch each run used, so replay stays consistent. For bigger rewrites, run a new workflow type and migrate in-flight work rather than forcing a change.

Temporal retries vs a plain queue retry

Feature	Temporal	Basic queue
Per-step retry policy	Yes, configurable	Manual
State across restarts	Persisted automatically	You build it
Compensation support	Pattern + context helpers	Ad hoc
Replay debugging	Event history	Logs only

The gap shows up most in compensation and replay. A queue makes you write the rollback logic and the state store yourself; Temporal gives you both, at the cost of the determinism rules.

What are the gotchas?

Three bite teams. Activity payloads have size limits — keep large blobs in S3 and pass a reference. Timers over a few decades are capped, so extremely long waits need a different design. And compensation activities must themselves be idempotent, because they can also be retried. If CancelFlight runs twice, the second call should no-op, not error.

Quick Summary

• Set explicit retry policies; add NonRetryableErrorTypes so bad input fails fast.
• Implement sagas with reverse-order compensations in a disconnected context.
• Use workflow.GetVersion to evolve running workflows without breaking replay.
• Keep activity payloads small and compensations idempotent.