How to Retry External API Calls Without Duplicate Side Effects

A retry is safe only when every attempt refers to the same logical operation and the system performing the side effect can recognize that identity. Generate the idempotency key once, reuse it for every attempt, retry only failures you know happened before the operation completed, and reconcile any request whose outcome is unknown. If the remote API cannot enforce idempotency or show you a result by key, a timeout after a mutating request is not an ordinary retry. It is an operational decision.

This guide builds that pattern from first principles, then implements it in WOML against a deliberately unreliable local API. You will see when an automatic retry is correct, when a workflow should stop, and what evidence an operator needs before continuing. The pattern works without WOML; the second half is simply a runnable version of it.

The failure window that makes retries dangerous

Picture a workflow step that creates a fulfillment job after a payment:

1. The worker sends POST /jobs.
2. The provider commits the job.
3. The network drops before the response reaches the worker.
4. The worker sees no success result.
5. Sending the request again may create a second job.

he uncertainty is about what the caller knows, not necessarily about what the provider did. A timeout means the caller stopped waiting. It does not prove the provider rolled anything back. Amazon’s EC2 documentation describes this directly: mutating requests can time out or hit server issues after the operation has already happened, and blind retries can then create more resources than you intended (AWS EC2 idempotency guide). The IETF’s Idempotency-Key draft opens with the same scenario: a POST times out and the client cannot tell whether the resource was created or whether a retry is safe.

So “just add retries” is incomplete advice. Treat the attempt as failed and you may duplicate the effect. Treat it as succeeded and you may lose work. You need a third state: unknown.

Separate the identities engineers commonly mix together

Most duplicate-effect bugs come from using one ID where two or three different ones are needed.

Decide whether an operation can be retried

Don’t reduce this to “5xx means retry.” AWS does recommend retrying 500-series responses with backoff for its idempotent EC2 requests, but that advice is attached to requests that carry a client token. The provider’s contract, and the point at which it returned the status, decide whether a retry is safe.

The retry-safe pattern

1. Define one logical effect. “Create the fulfillment job for order 42” is one effect, however many attempts it takes.
2. Create or obtain a stable operation identity before the attempt boundary. Never inside the retried code path.
3. Keep the payload semantically identical when reusing the key. The IETF draft says a key must not be reused with a different payload, and that a provider should reject that case (it suggests HTTP 422). EC2 returns IdempotentParameterMismatch.
4. Pass the key using the provider’s documented mechanism. A header, a request field, or a client token. The draft also notes that a general client cannot assume a server honors the header without prior knowledge.
5. Configure a bounded retry budget with backoff.
6. Record attempts and results durably, so a crash does not erase what you knew.
7. Retry only outcomes your runtime and the provider contract classify as safe.
8. For an unknown outcome, ask the provider by key (or compare against target state) before doing anything else.
9. Escalate when the provider cannot prove the outcome.
10. Compensate only with a real business inverse (a refund, a cancellation). It is a new side effect with its own failure modes, not a database rollback across systems.

Three decisions fall out of this: confirmed success, definitive failure (retry), and unknown outcome (reconcile, don’t retry).

Implementing the pattern in WOML

What the runtime gives you The WOML retry and idempotency docs define the rules this implementation relies on:

• retry is a step attribute holding the total attempt count, from 1 to 10. retry="3" means at most three attempts, not three retries after the first.
• Backoff is fixed or exponential. Exponential doubles from retry-delay up to retry-max-delay and currently adds no jitter.
• Only a definitive script-thrown failure retries automatically. Timeouts, invalid JSON, oversized results, crashes, interruption, and cancellation fail closed.
• attempt.idempotencyKey stays stable across all attempts of one logical step. attempt.number is there for diagnostics.
• Managed HTTP requests (see HTTP and Fetch) accept an idempotency option that sends the key in a header you choose.

The durability and recovery docs add the crash behavior. WOML records run truth as an append-only event history. A recorded step start proves an attempt began, not that the remote effect happened. If there is a start with no definitive result, WOML fails that work closed unless a reviewed capability can prove safe reattachment. The execution model page states the principle bluntly: event sourcing gives durable truth, but it does not make arbitrary side effects exactly-once.

One boundary matters more than any other: WOML supplies a stable key and runtime evidence. The external API must still document and enforce what a repeated key means. The docs say plainly that sending a key to an API that ignores it provides no protection.

The step

This is the core of the pattern, adapted from the complete payment webhook ledger example:

<step
  id="createFulfillment"
  name="Create fulfillment job"
  retry="3"
  retry-backoff="exponential"
  retry-delay="1s"
  retry-max-delay="10s"
>
  <script>
    const response = await services.http.request({
      method: "POST",
      url: "http://127.0.0.1:4040/jobs",
      json: { orderId: context.payload.orderId },
      timeout: "10s",
      idempotency: {
        header: "Idempotency-Key",
        value: attempt.idempotencyKey
      }
    }, { name: "create-fulfillment" });

    return response.data;
  </script>
</step>

Notice what is not here: no key generation inside the script, and no per-attempt identifier. The key belongs to the logical step.

A provider that misbehaves on purpose

To test the pattern you need an API that fails in the ways real ones do. This small Node or Bun service stores each key with a fingerprint of its payload, replays the original result for a repeated key, rejects a changed payload, and can be scripted to fail in three ways:

• fail-before-commit: return 503 without storing anything.
• commit-then-error: store the job, then return 503.
• commit-then-drop: store the job, then destroy the connection with no response.

// provider.mjs  (run: node provider.mjs, or bun provider.mjs)
import http from "node:http";
import { createHash, randomUUID } from "node:crypto";

const jobs = new Map(); // idempotency key -> { fingerprint, job }
let faults = [];        // scripted faults, consumed one per NEW operation

const fingerprint = (body) =>
  createHash("sha256").update(JSON.stringify(body)).digest("hex");

const readJson = async (req) => {
  const chunks = [];
  for await (const chunk of req) chunks.push(chunk);
  return chunks.length ? JSON.parse(Buffer.concat(chunks).toString()) : {};
};

const server = http.createServer(async (req, res) => {
  const url = new URL(req.url, "http://127.0.0.1:4040");
  const send = (status, body) => {
    res.writeHead(status, { "content-type": "application/json" });
    res.end(JSON.stringify(body));
  };

  // Test controls
  if (req.method === "POST" && url.pathname === "/_test/faults") {
    faults = (await readJson(req)).faults ?? [];
    return send(200, { faults });
  }
  if (req.method === "GET" && url.pathname === "/_test/jobs") {
    const all = [...jobs.entries()].map(([key, v]) => ({ key, ...v.job }));
    return send(200, { count: all.length, jobs: all });
  }

  // Reconciliation: what happened to this operation?
  const prefix = "/jobs/by-idempotency-key/";
  if (req.method === "GET" && url.pathname.startsWith(prefix)) {
    const record = jobs.get(decodeURIComponent(url.pathname.slice(prefix.length)));
    return record ? send(200, record.job) : send(404, { error: "not_found" });
  }

  // The mutating endpoint
  if (req.method === "POST" && url.pathname === "/jobs") {
    const key = req.headers["idempotency-key"];
    if (!key) return send(400, { error: "idempotency_key_required" });

    const body = await readJson(req);
    const fp = fingerprint(body);
    const existing = jobs.get(key);

    if (existing) {
      if (existing.fingerprint !== fp) {
        console.log(`[provider] key=${key} REJECTED (payload changed)`);
        return send(422, { error: "idempotency_key_reused_with_different_payload" });
      }
      console.log(`[provider] key=${key} REPLAY job=${existing.job.id}`);
      return send(200, existing.job);
    }

    const fault = faults.shift() ?? "none";
    console.log(`[provider] key=${key} NEW fault=${fault}`);

    if (fault === "fail-before-commit") {
      return send(503, { error: "temporarily_unavailable" }); // nothing stored
    }

    const job = { id: `job_${randomUUID().slice(0, 8)}`, orderId: body.orderId };
    jobs.set(key, { fingerprint: fp, job }); // the commit

    if (fault === "commit-then-error") {
      return send(503, { error: "temporarily_unavailable" }); // committed, caller told 503
    }
    if (fault === "commit-then-drop") {
      req.socket.destroy(); // committed, caller never hears back
      return;
    }
    return send(201, job);
  }

  send(404, { error: "not_found" });
});

server.listen(4040, "127.0.0.1", () =>
  console.log("[provider] listening on http://127.0.0.1:4040"));

Before involving any workflow engine, prove the fixture itself behaves. This script asserts the key properties: a retry after a pre-commit failure creates one job, a retry after a post-commit 503 replays rather than duplicates, a changed payload is rejected, a new key per attempt creates a duplicate (the bug you are avoiding), and a dropped connection is still recoverable by lookup.

// check-provider.mjs  (run while provider.mjs is up: node check-provider.mjs)
import assert from "node:assert/strict";

const base = "http://127.0.0.1:4040";
const post = (key, body) =>
  fetch(`${base}/jobs`, {
    method: "POST",
    headers: { "content-type": "application/json", "idempotency-key": key },
    body: JSON.stringify(body),
  });
const setFaults = (faults) =>
  fetch(`${base}/_test/faults`, { method: "POST", body: JSON.stringify({ faults }) });
const jobCount = async () => (await (await fetch(`${base}/_test/jobs`)).json()).count;

await setFaults(["fail-before-commit"]);
assert.equal((await post("key-A", { orderId: "order-1" })).status, 503);
assert.equal(await jobCount(), 0);
assert.equal((await post("key-A", { orderId: "order-1" })).status, 201);
assert.equal(await jobCount(), 1);

await setFaults(["commit-then-error"]);
assert.equal((await post("key-B", { orderId: "order-2" })).status, 503);
assert.equal(await jobCount(), 2);                 // committed despite the 503
assert.equal((await post("key-B", { orderId: "order-2" })).status, 200); // replay
assert.equal(await jobCount(), 2);                 // no duplicate

assert.equal((await post("key-B", { orderId: "order-999" })).status, 422);

await setFaults(["commit-then-error"]);            // the anti-pattern: new key per attempt
await post("key-D-attempt-1", { orderId: "order-3" });
await post("key-D-attempt-2", { orderId: "order-3" });
assert.equal(await jobCount(), 4);                 // duplicate for order-3

await setFaults(["commit-then-drop"]);
await assert.rejects(post("key-E", { orderId: "order-4" }));
assert.equal((await fetch(`${base}/jobs/by-idempotency-key/key-E`)).status, 200);

console.log("all provider assertions passed");

The workflow under test

To exercise the step in isolation, wrap it in a small workflow with a webhook trigger. (In the full payment example, the ledger workflow starts a separate fulfill-paid-order workflow; this demo collapses that hop so you can aim a single curl at the step.)

<woml>
  <workflow
    id="fulfill-retry-demo"
    name="Fulfillment retry demo"
    description="Create one fulfillment job per order, safely."
    version="1.0.0"
  >
    <config concurrency="1" timeout="2m" />

    <triggers>
      <webhook id="fulfill" path="/webhooks/fulfill" method="POST" auth="none">
        <schema>
          {
            "type": "object",
            "required": ["orderId", "paymentId", "amount"],
            "properties": {
              "orderId": { "type": "string" },
              "paymentId": { "type": "string" },
              "amount": { "type": "number", "minimum": 0 }
            },
            "additionalProperties": false
          }
        </schema>
      </webhook>
    </triggers>

    <steps>
      <step
        id="createFulfillment"
        name="Create fulfillment job"
        retry="3"
        retry-backoff="exponential"
        retry-delay="1s"
        retry-max-delay="10s"
      >
        <script>
          const response = await services.http.request({
            method: "POST",
            url: "http://127.0.0.1:4040/jobs",
            json: {
              orderId: context.payload.orderId,
              paymentId: context.payload.paymentId,
              amount: context.payload.amount
            },
            timeout: "10s",
            idempotency: {
              header: "Idempotency-Key",
              value: attempt.idempotencyKey
            }
          }, { name: "create-fulfillment" });

          return {
            jobId: response.data.id,
            idempotencyKey: attempt.idempotencyKey,
            attempt: attempt.number
          };
        </script>
      </step>
    </steps>
  </workflow>
</woml>

auth="none" is acceptable here only because the demo listens on localhost. The webhook docs are explicit that it should never face untrusted traffic without an authenticating proxy.

Run three failure scenarios

# Terminal 1: the provider
node provider.mjs

# Terminal 2: validate, then run (it prints the webhook URL)
woml check fulfill-retry-demo.woml
woml run fulfill-retry-demo.woml

# Terminal 3: script a fault, trigger a run, inspect
curl -s -X POST http://127.0.0.1:4040/_test/faults -d '{"faults":["fail-before-commit"]}'
curl -s -X POST http://127.0.0.1:3000/webhooks/fulfill \
  -H 'content-type: application/json' \
  -d '{"orderId":"order-42","paymentId":"pay-42","amount":120}'
curl -s http://127.0.0.1:4040/_test/jobs
woml list
woml get <run-id>

Use the URL that woml run prints if it differs from port 3000. Repeat with each fault script:

Scenarios A and B are where bounded retry earns its keep, and B is the instructive one: the first attempt really did create the job, and the retry is safe only because the second attempt carries the same key. Scenario C is the reason the pattern includes a reconcile step.

• What to expect in scenario A. Attempt 1 reaches the provider, which answers 503 before storing anything, so /_test/jobs is still empty. Under the default accepted-status policy that response surfaces as a thrown failure, the kind WOML retries. After the backoff delay, attempt 2 sends the same Idempotency-Key and gets 201. woml get <run-id> should show one createFulfillment step with two attempts, and the run should finish successfully. The provider log shows two NEW lines for the same key (fault=fail-before-commit, then fault=none), and /_test/jobs reports a count of 1.
• What to expect in scenario B. Attempt 1 creates the job but the provider answers 503, so from the workflow’s side it looks like scenario A. WOML retries with the same key, and this time the provider recognizes it and replays the stored job instead of creating a new one. The provider log shows NEW fault=commit-then-error followed by REPLAY job=... with the same job ID, the run finishes successfully after two attempts, and /_test/jobs still reports a count of 1. This is the case that proves the point: the first attempt really did create the job, and the retry is safe only because the key stayed the same.
• What to expect in scenario C. The provider stores the job and then destroys the connection, so the workflow never receives a response and cannot know whether the job exists. According to the WOML docs, ambiguous outcomes fail closed, so the run should stop in a failed state instead of sending the request again. The provider is the source of truth here: a single NEW fault=commit-then-drop line and no REPLAY line means the request was not repeated. /_test/jobs reports a count of 1, because the job was created even though the workflow never heard about it.

Reconcile the unknown outcome

In scenario C the workflow has done the responsible thing by stopping. Now a human or a follow-up process needs facts, not a guess:

woml list --status failed
woml get <run-id>
# the fixture logs every key it receives; match it to the run, then ask the provider:
curl -s http://127.0.0.1:4040/jobs/by-idempotency-key/<key>

Two outcomes are possible. If the lookup returns a job, the effect happened: record the verified job ID and continue with downstream work. Do not create it again. If it returns 404, the provider has no record under that key, so a new attempt with the same key is the safest resubmission, provided the key has not expired at the provider. Either way, the decision is made from evidence.

Note that woml get is deliberately an operations view. According to the manage runs docs, it reports status, steps, attempts, and waits, but omits payloads and result values. If the business result matters, store it yourself in a database or object store, and see the logs and inspection docs for what you can follow live. For multi-step flows with a real business inverse, WOML’s saga support runs author-defined compensations, and for an effect that cannot be proven it stops for explicit operator resolution.

• What to expect from the reconciliation lookup. Asking the provider for the key from scenario C returns the job with HTTP 200, confirming the effect happened even though the workflow never received a response. The operator’s decision is then simple: record that job ID against the order, continue with any downstream work, and do not create the job again.

Inbound webhook deduplication is a separate problem

There are two boundaries in the payment flow, and they use different mechanisms.

• Inbound: the payment provider delivers payment.succeeded, possibly more than once. The identity here is the provider’s event ID. The defense is a uniqueness constraint on that ID.
• Outbound: your workflow asks the fulfillment API to create a job. The identity here is the logical operation key. The defense is provider-enforced idempotency.

The payment ledger example enforces the inbound side in the database, then only the first delivery proceeds:

const result = await db.execute({
  text: `
    INSERT INTO payment_events
      (provider_event_id, type, payment_id, order_id, amount, received_at)
    VALUES ($1, $2, $3, $4, $5, $6)
    ON CONFLICT (provider_event_id) DO NOTHING
  `,
  values: [/* eventId, type, paymentId, orderId, amount, receivedAt */]
}, { name: "insert-payment-event" });

return { firstDelivery: result.rowsAffected === 1 };

Three points are worth stating. First, verifying a webhook’s signature or bearer token proves who sent it, not that you haven’t seen it before; an authenticated redelivery is still a redelivery (see the OWASP Webhook Security Cheat Sheet for signature and replay guidance). Second, a check-then-set in application code or a cache can race when duplicates arrive concurrently; an atomic insert against a unique constraint picks one winner. Third, deduplicating the inbound event does not make the outbound call safe. The two ids are different on purpose.

WOML’s webhook trigger takes on schema validation, authentication, deduplication, and durable admission, and the example layers a database uniqueness proof on top. Read the complete example for the full flow, including the failed and duplicate branches.

Failure cases your tests should cover

The manual-rerun row deserves a deliberate answer. A rerun is a new run, so don’t assume it inherits the earlier run’s key. If a rerun must mean “the same fulfillment for order 42,” derive the key from the business identity you intend (for example fulfillment:order-42) instead of leaning on per-run identity, and confirm your provider’s retention window.

Limitations and tradeoffs

• Idempotency needs storage and a retention policy at the system enforcing the key. Providers may expire keys, and a retry outside that window can behave like a brand-new request.
• Retaining full responses for replay can expose sensitive data or consume storage.
• A key cannot correct an operation that was wrong but successfully executed.
• Deduplication prevents repeats. It does not guarantee ordering.
• Backoff without jitter can synchronize retries during an outage. WOML’s exponential retry currently adds none, so keep retry counts small.
• Failing closed reduces duplication risk but creates operational work and can delay completion.
• Compensation is a new side effect with its own failure modes, not a distributed rollback.
• Use “exactly-once” only for a boundary controlled and proven by the enforcing system. This pattern does not make an external effect exactly-once.
• WOML provides a stable identity and durable runtime evidence. The external API stays responsible for its own idempotency semantics. See current WOML limitations; notably, the supported production profile is a single runtime and state authority, and cancellation cannot undo an external effect.

Conclusion

Retry the logical operation, not an untracked HTTP request. Reuse one identity, depend on an explicit external idempotency contract, and stop for reconciliation whenever the outcome is unknown.

Run the failure case yourself. Open the Payment Webhook Ledger example, point the fulfillment step at the local fixture above, and inspect how the logical operation and its attempts appear in the run history. For the exact runtime rules, read WOML’s retry and idempotency semantics. For the broader concept, see why durable execution is more than retrying a function.