What Is Idempotency? Idempotency Keys Explained in Python

An operation is idempotent when doing it twice has the same effect as doing it once. Pressing a lift's call button five times still brings one lift. An idempotency key makes a payment work the same way: the app sends the same key with every retry, and the server charges once and repeats its first answer.

In this post we pick up the shop app from the message queue post, where a slow worker meant Asha got two order emails. First we see the two ways a request gets repeated: a reply lost on the network, and a queue that delivers a message again. Then we read what the HTTP standard says about idempotent methods, build idempotency keys in plain Python with SQLite, let two retries race each other, and make the queue worker safe to run twice. We finish with how long to keep keys, when you don't need them, and the beginner mistake that charges a card twice even with keys.

A few words first. A client is the program that asks, such as a shop app, and an API is the set of requests a service accepts from clients. A request is a message asking a server to do something, and the response (or reply) is its answer. A retry is sending the same request again because the first one seemed to fail. A side effect is a change a request makes in the world: money taken, an email sent, a row added to a table. A transaction is a group of database changes that are saved all together or not at all. SQLite is a small database that comes with Python, in its sqlite3 module. A connection is one program's open line to a database, and a lock is the database's hold that stops other connections writing until it's released. And from the last two posts: an event is a record of something that happened, such as "order 101 was placed"; the producer is the program that sends (publishes) it to a queue; a worker takes it off the queue, does the job, then acks it, telling the broker (the service that runs the queue) "done, you can delete it". If no ack comes within the visibility timeout, the broker hands the message out again.

What is idempotency in simple words?

Think of the call button for a lift. You press it, and nothing seems to happen, so you press it again, and again. One lift still comes. Pressing more times doesn't bring more lifts. The button is idempotent: once or five times, the result is the same. Now think of the tally counter a guard clicks at a gate. Every click adds one, so two clicks for one visitor count two people. That click isn't idempotent.

Back at the dosa restaurant from the last two posts. The manager clips a ticket back on the rail when a cook takes too long, in case the cook has walked off. But sometimes the cook was only slow, and two dosas come out for one order. So the owner adds a rule. Every ticket gets a number, and there's a board by the stove. When a cook takes a ticket, he goes to pencil its number on the board. If the number is already inked, the dosa has gone out, so he bins the ticket. If it's pencilled, another cook is on it, so he waits: once it's inked he bins the ticket, and if it's rubbed out he cooks it himself. If the number isn't there, he pencils it in, cooks, and inks it over when the dosa goes out. If a cook walks off halfway, the manager rubs out his pencil mark. Now a ticket can come round twice, even while the first cook is still at the stove, and the table still gets one dosa.

That number is an idempotency key: a label that says "this is the same job as before". The rest of this post puts it into code.

What problem does idempotency solve?

Requests get repeated in two common ways.

1. The reply gets lost. Your shop app asks the payment service to take Rs 450 for order 101. The payment service takes the money, but the phone's signal drops before the reply arrives. From the app's side, the request just timed out. Did the money go? It can't tell. Stripe, a payment company, puts it plainly in its docs: "When a connection fails, you might not know whether Stripe received or executed the request."

The natural fix is to try again. The Amazon Builders' Library, where Amazon's engineers write about how they build their services, says retrying works well: "a surprisingly large number of transient or random faults can be overcome by simply retrying the call." (A transient fault is one that goes away by itself.) But it adds a warning: "However, retrying a service call as mitigation for a transient fault is based on a simplifying assumption that an operation can be retried without any side effects." If the first try did take the money, the retry takes it again.

2. A queue delivers a message again. In the message queue post, a worker took order 101's message and was slow. The broker thought the worker had died and handed the message to a second worker, and Asha got two emails. Brokers that work this way promise at-least-once delivery: every message arrives, but some arrive more than once. The Amazon SQS docs give the rule: "Design your applications to be idempotent (they should not be affected adversely when processing the same message more than once)."

People repeat requests too: a customer who taps "Pay" twice because the button seemed stuck. Whatever the cause, the fix isn't to stop repeats. They can't be stopped. The fix is to make a repeat harmless.

Which HTTP methods are idempotent?

The idea has a formal home in HTTP, the language browsers and apps use to talk to servers. Each HTTP request has a method, a word that says what kind of request it is. RFC 9110, the standard that defines HTTP's methods (an RFC is an official internet standards document), says: "A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request." It then lists them: "Of the request methods defined by this specification, PUT, DELETE, and safe request methods are idempotent." The safe methods are the ones that only read: "the GET, HEAD, OPTIONS, and TRACE methods are defined to be safe."

The standard also says why this matters, in the very situation from the last section: "Idempotent methods are distinguished because the request can be repeated automatically if a communication failure occurs before the client is able to read the server's response."

Look at the difference between "set" and "add". Setting a status to "shipped" twice leaves it "shipped". Adding 2 to a basket twice leaves 4. MDN, Mozilla's web docs, sums up the last two rows: "The POST and PATCH methods are not guaranteed to be idempotent."

Idempotent is about the effect, not the reply. MDN's example is DELETE: "the first call of a DELETE will likely return a 200, while successive ones will likely return a 404." (200 means "OK", 404 means "not found".) The coupon is gone after the first call, and still gone after the second; only the answer changed.

Taking money is a POST: each call means "charge again". So a payment API can't rely on the method. It needs another way to tell a retry from a new payment, and that's the idempotency key.

How do idempotency keys work?

Stripe's API reference describes it in one sentence: "A client generates an idempotency key, which is a unique key that the server uses to recognize subsequent retries of the same request." Step by step:

1. The client makes a key once, before the first try. It must be unique to this one payment. Stripe suggests "using V4 UUIDs, or another random string with enough entropy to avoid collisions." A version 4 UUID is a 36-character random id such as 16fd2706-8baf-433b-82eb-8c7fada847da; enough entropy means enough randomness that two payments practically never end up with the same key by chance. Python makes one with uuid.uuid4(), which the docs say will "Generate a random UUID in a cryptographically-secure method" (so it's hard for anyone to guess).
2. The client sends the key with the request, and with every retry of it. Stripe reads it from a header (a labelled line sent along with an HTTP request): "Idempotency keys are sent in the Idempotency-Key header."
3. The server looks the key up. New key: it does the work, and saves the key together with its answer. Stripe, for example, saves the status code and body of its first answer for each key.
4. A key it has seen before means a retry. The server does nothing new and sends back the saved answer. The client gets the same reply it would have got the first time, so it doesn't need to know which try did the work.

Two more rules make it safe. First, a key belongs to one request only. Stripe: "The idempotency layer compares incoming parameters to those of the original request and errors if they're not the same to prevent accidental misuse." (Parameters are the details sent with the request, such as the order and the amount.)

Second, the key and the work are saved together. The Amazon Builders' Library says the step that records the key and makes the changes "must meet the properties for an atomic, consistent, isolated, and durable (ACID) operation." In plain words: all of it is saved, or none of it is; the database's rules still hold afterwards; two at once can't get in each other's way; and once saved, it stays saved. Amazon's article names the two half-done cases to avoid. A saved key with no charge would answer every retry with a payment that never happened. A charge with no saved key would let the retry charge again.

How do you build idempotency keys in Python?

Let's play the payment company. Our payment service keeps its records in SQLite: a charges table with one row per charge, and an idempotency_keys table with one row per key. First the service, with two ways to charge:

# payments.py
import json
import sqlite3


def open_db(path):
    db = sqlite3.connect(path, timeout=5)  # wait up to 5 s if another connection is writing (the default)
    db.executescript("""
        CREATE TABLE IF NOT EXISTS charges (
            id INTEGER PRIMARY KEY, order_id INTEGER, amount INTEGER);
        CREATE TABLE IF NOT EXISTS idempotency_keys (
            key TEXT PRIMARY KEY, request TEXT, response TEXT);
    """)
    return db


def charge(db, order_id, amount):
    """Take the money. Call it twice and it takes the money twice."""
    with db:
        charge_id = db.execute("INSERT INTO charges (order_id, amount) VALUES (?, ?)",
                               (order_id, amount)).lastrowid
    return {"charge": charge_id, "order": order_id, "amount": amount}


def charge_once(db, key, order_id, amount):
    """Take the money once per key. A retry with the same key gets the first answer back."""
    request = json.dumps({"order": order_id, "amount": amount})
    try:
        with db:  # one transaction: the key, the charge and the answer are saved together, or not at all
            db.execute("INSERT INTO idempotency_keys (key, request) VALUES (?, ?)", (key, request))
            charge_id = db.execute("INSERT INTO charges (order_id, amount) VALUES (?, ?)",
                                   (order_id, amount)).lastrowid
            response = {"charge": charge_id, "order": order_id, "amount": amount}
            db.execute("UPDATE idempotency_keys SET response = ? WHERE key = ?", (json.dumps(response), key))
        return response
    except sqlite3.IntegrityError:  # the key is already saved, so this is a retry
        saved_request, saved_response = db.execute(
            "SELECT request, response FROM idempotency_keys WHERE key = ?", (key,)).fetchone()
        if saved_request != request:
            raise ValueError(f"key {key} was already used for a different payment")
        return json.loads(saved_response)  # the first answer again; no new charge

• PRIMARY KEY on the key column means no two rows can have the same key. The database itself enforces it, and that's what the whole idea rests on.
• charge is the plain version: one new row in charges on every call. .lastrowid is the id SQLite gave the new row, which we use as the charge number.
• charge_once first tries to insert the key. If the key is new, it inserts the charge and saves the answer in the key's row. with db: makes all three changes one transaction; the Python docs say the connection "automatically commits or rolls back open transactions when leaving the body of the context manager." (The context manager here is the with block.) (Commit means save for good; roll back means undo.)
• If the key is already there, the insert fails with sqlite3.IntegrityError, the error for a broken database rule. That's a retry: we read the saved answer and send it back. If the saved request doesn't match this one, the key is being reused for a different payment, so we refuse.
• json.dumps turns the request and the answer into text so they fit in one column; json.loads turns the text back. The ? marks are placeholders: sqlite3 fills them in safely from the values that follow.

Next, a network that loses a reply. The service does the work, but the answer never reaches the shop:

# network.py
class FlakyNetwork:
    """Carries each request to the payment service, but loses the first reply on its way back."""

    def __init__(self, replies_to_lose=1):
        self.replies_to_lose = replies_to_lose

    def call(self, service, *args):
        reply = service(*args)              # the payment service does the work...
        if self.replies_to_lose > 0:
            self.replies_to_lose -= 1
            raise TimeoutError("no reply")  # ...but the answer never reaches the shop
        return reply

And the shop, which pays for order 101 and retries up to three times:

# shop.py
import sys
import uuid

from network import FlakyNetwork
from payments import charge, charge_once, open_db


def pay(net, db, order_id, amount, key=None):
    """Try up to 3 times. Every try sends the same key, the one made with the order."""
    for attempt in range(1, 4):
        try:
            if key:
                return net.call(charge_once, db, key, order_id, amount)
            return net.call(charge, db, order_id, amount)
        except TimeoutError:
            print(f"Try {attempt}: no reply. Did it go through? Trying again.")
    raise RuntimeError("the payment service isn't answering")


db = open_db(":memory:")           # a database that lives only while the program runs
order_key = str(uuid.uuid4())      # one key for this payment (a real shop saves it with order 101)
use_key = sys.argv[1:] == ["key"]  # python shop.py key, or python shop.py nokey
reply = pay(FlakyNetwork(), db, 101, 450, order_key if use_key else None)
print("Reply:", reply)
times, total = db.execute("SELECT COUNT(*), SUM(amount) FROM charges WHERE order_id = 101").fetchone()
print(f"Order 101 was charged {times} time(s), {total} rupees in all")

The key is made once for this payment, outside pay, so all three tries carry the same one. A real shop saves it with order 101, so a second tap on "Pay", or a retry after the app restarts, sends that same key too. (sys.argv[1:] is the list of words typed after python shop.py.) Put the three files in one folder and run python shop.py nokey first, without a key:

Try 1: no reply. Did it go through? Trying again.
Reply: {'charge': 2, 'order': 101, 'amount': 450}
Order 101 was charged 2 time(s), 900 rupees in all

The first try took Rs 450 and lost the reply. The retry took another Rs 450. Asha paid Rs 900 for a Rs 450 order, and the shop only ever saw one reply, so it doesn't even know. Now python shop.py key:

Try 1: no reply. Did it go through? Trying again.
Reply: {'charge': 1, 'order': 101, 'amount': 450}
Order 101 was charged 1 time(s), 450 rupees in all

Same lost reply, same retry. But this time the retry's key was already saved, so charge_once sent back the first answer, charge 1, and charged nothing new. The shop gets the reply it missed, and Asha pays once.

What if someone sends a saved key with a different payment? This runs in the same folder. (The key pay-101-a is a readable one for the demo; a real one is a UUID.)

from payments import charge_once, open_db

db = open_db(":memory:")
print(charge_once(db, "pay-101-a", 101, 450))
print(charge_once(db, "pay-101-a", 101, 450))   # a retry: same key, same payment
try:
    charge_once(db, "pay-101-a", 102, 990)      # the same key, sent with a different payment
except ValueError as error:
    print("Refused:", error)

{'charge': 1, 'order': 101, 'amount': 450}
{'charge': 1, 'order': 101, 'amount': 450}
Refused: key pay-101-a was already used for a different payment

The retry gets charge 1 again. The different payment is refused, instead of quietly getting order 101's answer and leaving order 102 unpaid.

What if two retries arrive at the same time?

A customer taps "Pay" twice, fast. Both taps send order 101's saved key, and both requests reach the payment service at the same instant, on two different servers. You might be tempted to write the check as two steps: first look the key up with a SELECT, and if it isn't there, insert it. But there's a gap between the two steps. Both requests can look, both find nothing, and both charge.

charge_once has no gap, because it never looks first. It tries to insert the key, and the PRIMARY KEY makes the database refuse the second one. SQLite's docs: "If an INSERT or UPDATE statement attempts to modify the table content so that two or more rows have identical primary key values, that is a constraint violation." (A constraint is a rule the database enforces on its data.) Let's test it with two threads (two lines of work running at once inside one program), each with its own connection to one database file, like two servers sharing one database. A Barrier holds both threads until both are ready, then lets them go together:

import threading
from pathlib import Path

from payments import charge_once, open_db

Path("shop.db").unlink(missing_ok=True)  # start with an empty database file
open_db("shop.db").close()               # make the tables
both_ready = threading.Barrier(2)
answers = []


def tap_pay():
    db = open_db("shop.db")  # each tap gets its own connection, as two servers would
    both_ready.wait()        # ...and both arrive at the same instant
    answers.append(charge_once(db, "pay-101-a", 101, 450)["charge"])


taps = [threading.Thread(target=tap_pay) for _ in range(2)]
for t in taps:
    t.start()
for t in taps:
    t.join()
db = open_db("shop.db")
print("Answers:", answers)
print("Charges for order 101:", db.execute("SELECT COUNT(*) FROM charges WHERE order_id = 101").fetchone()[0])

Answers: [1, 1]
Charges for order 101: 1

One tap's insert got in first. SQLite lets one connection write at a time, so the other tap's insert waited until the first transaction was saved (Python's sqlite3 waits up to 5 seconds for a lock by default; the timeout=5 in open_db just makes that visible), then found the key already there, got IntegrityError, and read back the saved answer. Both taps got charge 1, and there's one charge.

A real service would often use a bigger database such as PostgreSQL, but the tool is the same. PostgreSQL's docs: "Unique constraints ensure that the data contained in a column, or a group of columns, is unique among all the rows in the table." Its INSERT can even say what to do instead of failing: "ON CONFLICT can be used to specify an alternative action to raising a unique constraint or exclusion constraint violation error." (An exclusion constraint is another kind of rule; here only the unique one matters.) Either way, let the database's unique rule decide who was first, not a check in your own code.

One honest gap: in our toy, the charge is just a row in the same database, so one transaction covers it. A real payment service has to call a bank, and no database transaction can include that, so it can't simply hold one transaction open the way ours does. Stripe, for example, answers a second request that arrives while the first is still running with a 409 Conflict error (409 is HTTP's "conflict" code), which means "still working, try again". Its docs say: "If a 409 indicates that another request with the same key is still executing, wait and retry with that key and the same parameters."

How do you make a queue worker idempotent?

A worker taking messages off a queue needs the same thing, with one change: there's no key from a client, so it uses the event's id. The order service gives each event an id when it saves the order, such as order-101-placed, and every send of that event carries the same id. Don't use the broker's own message number instead: our toy broker from the last post numbers every send(), so if the order service sent the same event twice, the two copies would get two different numbers.

Here is the stock listener from the event-driven architecture post, rebuilt on SQLite. Back then, the "Try it" answer kept a set of done order ids in memory, which is forgotten when the program restarts. Now the done list is a table, saved in the same transaction as the stock change:

# stock.py
import sqlite3


def open_stock(path):
    db = sqlite3.connect(path)
    db.executescript("""
        CREATE TABLE IF NOT EXISTS stock (item TEXT PRIMARY KEY, count INTEGER);
        CREATE TABLE IF NOT EXISTS processed (event_id TEXT PRIMARY KEY);
        INSERT OR IGNORE INTO stock VALUES ('dosa-mix', 10);
    """)
    return db


def reserve(db, event):
    """Take an order's items out of stock once, however many times its event arrives."""
    try:
        with db:  # the "done" note and the stock change are saved together, or neither is
            db.execute("INSERT INTO processed (event_id) VALUES (?)", (event["id"],))
            db.execute("UPDATE stock SET count = count - ? WHERE item = ?", (event["qty"], event["item"]))
    except sqlite3.IntegrityError:  # its id is already in processed
        print(f"Event {event['id']}: seen before, skipped")
        return
    left = db.execute("SELECT count FROM stock WHERE item = ?", (event["item"],)).fetchone()[0]
    print(f"Event {event['id']}: stock now {left} {event['item']}")

The shop starts with 10 dosa-mix (INSERT OR IGNORE adds that row only if it isn't there yet). Now deliver one event three times, with a restart in the middle:

from pathlib import Path

from stock import open_stock, reserve

Path("stock.db").unlink(missing_ok=True)  # start with 10 dosa-mix
event = {"id": "order-101-placed", "item": "dosa-mix", "qty": 2}

db = open_stock("stock.db")
reserve(db, event)
reserve(db, event)          # the broker hands the same message out again
db.close()                  # the worker stops...
db = open_stock("stock.db") # ...and starts again
reserve(db, event)          # one more repeat, after the restart

Event order-101-placed: stock now 8 dosa-mix
Event order-101-placed: seen before, skipped
Event order-101-placed: seen before, skipped

Three deliveries, one stock change: 10 − 2 = 8. The third delivery came after a restart and was still skipped, because the done note is in the database file, not in the program's memory. And because the note and the stock change are one transaction, a crash can't leave one without the other. The worker from the last post still acks after reserve returns, skipped or not, so the broker can delete the message.

What about the email? Here we hit a real limit. The mail server isn't in our database, so sending an email and saving a "sent" note can't be one transaction. You have to pick an order, and each has a failure:

For an order confirmation, a rare second email is better than none, so send, then save. Checking the note first still skips most repeats. Two cases get through: a crash between sending and saving, and a send so slow that the broker hands the message to another worker before the first one has saved its note. That second case is exactly how Asha got her two emails, so set the visibility timeout well above your slowest send.

The stock worker doesn't have this gap. Its open transaction is the pencil mark on the dosa board: the insert into processed holds the database's lock until the transaction is saved, so a second worker waits, then skips. You could keep the transaction open while the email sends, so a second worker waits. But that holds a database lock for a whole slow network call (in SQLite, a lock on the whole database), and a crash after the send but before the save still rolls the note back, so the email goes again. A sent email can't be undone the way a stock change can. The real fix is on the other side: when the outside service accepts an idempotency key itself (Stripe's API does, for payments), pass the event's id as the key and retry within its key window (24 hours for Stripe). Then the outside service does the remembering.

How long should you keep idempotency keys?

Not forever: every key is a row to store. But long enough to cover every retry. Stripe keeps its keys for at least a day: "Stripe retains keys for at least 24 hours and can remove them after that." It warns what happens after that: "Reusing a key after it's removed can execute a new request." So its advice for clients is to "Retry with the same idempotency key and parameters within the first 24 hours after the original request."

Some brokers remove repeats for you, for a much shorter time. Amazon SQS FIFO queues (first in, first out) take a MessageDeduplicationId, and the docs say: "It ensures that within a 5-minute deduplication window, only one instance of a message with the same deduplication ID is processed and delivered." That catches a producer that sends the same event twice in quick succession. It doesn't help a worker that crashes before its ack, so the worker still needs its own done table.

The Amazon Builders' Library points out the awkward case, a retry that arrives very late, after things have changed. Amazon's EC2 service (for renting computers) still gives the late retry the first answer, and the article says what that costs: "To support this behavior we need to retain knowledge of the initial idempotent request in our service." A simple rule: keep keys longer than the longest time a retry could arrive, and clean out older ones on a schedule.

When don't you need an idempotency key?

• Reads. A GET changes nothing, so repeating it is harmless. Stripe tells clients: "Don't send idempotency keys in GET and DELETE requests because it has no effect. These requests are idempotent by definition."
• Operations that are idempotent already. "Set order 101's status to shipped" or "delete coupon SAVE10" give the same result however often they run. One cheap fix is to turn an "add" into a "set": instead of "take 2 off the stock", save "order 101 reserves 2" in a table where each order can appear once.
• Repeats that don't matter. RFC 9110 itself notes that even idempotent requests can have small side effects such as logging: a server "is free to log each request separately". A log line or a page-view count that's off by one now and then isn't worth a keys table.

Everything else that moves money, sends a message to a person, or creates something should be safe to run twice. A keys table is a small price next to a customer charged twice.

What mistake do beginners make with idempotency keys?

Making a new key on every try. It's an easy slip: the key line ends up inside the retry loop, next to the request it goes with. Here's that version of pay, with the same service and the same flaky network:

import uuid

from network import FlakyNetwork
from payments import charge_once, open_db


def pay(net, db, order_id, amount):
    for attempt in range(1, 4):
        key = str(uuid.uuid4())  # a brand-new key on every try...
        try:
            return net.call(charge_once, db, key, order_id, amount)
        except TimeoutError:
            print(f"Try {attempt}: no reply. Trying again.")


db = open_db(":memory:")
pay(FlakyNetwork(), db, 101, 450)
print("Charges for order 101:", db.execute("SELECT COUNT(*) FROM charges WHERE order_id = 101").fetchone()[0])

Try 1: no reply. Trying again.
Charges for order 101: 2

Two charges, with idempotency keys switched on. The retry carried a key the service had never seen, so it looked like a new payment. A key means "this payment attempt", not "this try". Make it once, before the first try, keep it with the order, and send that same key with every retry of that attempt. A saved answer can be a failure too: Stripe "caches the result after the API method begins execution, including errors." So once an attempt has really ended, say the card was declined, any new attempt, on another card or the same card after a top-up, is a new payment and gets a new key. Stripe's error-handling guide says the same: "When retrying the same operation, preserve the idempotency key and parameters within the retry window."

A second slip: building the key from personal details, such as the customer's email address. Stripe warns: "Avoid using sensitive data (for example, email addresses or personal identifiers) as idempotency keys." A random UUID tells nobody anything.

Event order-101-placed: stock now 8 dosa-mix
Event order-101-placed: seen before, skipped
Event order-101-placed: seen before, skipped

Keep going

• Previous in this System Design series: Message queues and workers, where acking after the work made repeats possible in the first place.
• Next: "Webhooks", how one service tells another that something happened by calling its URL, with signing and retries.
• Where the order events come from: Event-driven architecture.
• A real payment, step by step: How does UPI work?

Checked on 4 November 2026 against the 13 sources below.

Sources

• IETF: RFC 9110, HTTP Semantics (section 9.2.1, safe methods; 9.2.2, idempotent methods)
• MDN Web Docs: Idempotent (glossary)
• Stripe API reference: Idempotent requests
• Stripe documentation: Advanced error handling (network errors, key retention)
• Amazon Builders' Library: Making retries safe with idempotent APIs
• AWS documentation: Amazon SQS at-least-once delivery
• AWS documentation: Using the message deduplication ID in Amazon SQS
• Google Cloud documentation: Exactly-once delivery (Pub/Sub)
• Python documentation: sqlite3 (the connection as a context manager)
• Python documentation: uuid (uuid4)
• SQLite documentation: CREATE TABLE (primary keys)
• PostgreSQL documentation: Constraints (unique constraints)
• PostgreSQL documentation: INSERT (ON CONFLICT)

Every Python sample in this post was run in Python 3.13 before publishing.