Request a demo

Integrations and Technology

Avoiding Duplicate Records When Integrating Business Systems

Duplicate customers and double invoices come from retries and missing IDs, not careless people. How idempotency keys and external references keep integrated systems clean, with a worked example.

By Flux Editorial TeamPublished 4 min read

Illustrative example with fictional data.

The most expensive integration bug isn't a crash. It's a customer billed twice, or the same company created three times with slightly different names, discovered months later when a statement makes no sense. These duplicates rarely come from careless people. They come from networks that fail at the wrong moment, and from systems that can't tell they've already seen something. Both have well-understood fixes.

How duplicates happen

The retry that wasn't needed

Your system asks the finance system to raise an invoice. The finance system raises it, but the reply is lost: a timeout, a dropped mobile connection, a server restart. Your code sees an error and, sensibly, tries again. The finance system receives what looks like a new request and raises a second invoice.

Nobody did anything wrong, and the customer now has two invoices for the same month.

The record nobody could match

Your booking website knows a customer as booking ws_42. The finance system knows them as customer 812. If nothing links the two, the next time ws_42 books, the website has no way to ask "do you already have this customer?" so it creates another.

Over a year, "Garki Training Institute", "Garki Training Inst." and "GTI Abuja" all appear in your customer list, each with part of the history.

Fix 1: an external reference on every record

Give every record your own stable identifier when you create it in the other system, and store it there. Flux calls this externalRef.

Record Good external reference Why
Customer ws_42 (your account id) Never changes, even if the name does
Invoice ws_42-2026-11 (account + billing period) One per customer per month, by construction
Invoice for a one-off order order_88213 One per order

The invoice pattern is the important one: build the reference from what you're billing, not from a random number. Then "have I already billed ws_42 for November?" has a definite answer.

In Flux, externalRef behaves differently for invoices and customers, and both behaviours are worth knowing:

  • Invoices are idempotent on externalRef permanently. If an invoice with that reference already exists, Flux returns the existing one with a 200 and raises nothing new. That holds even for two requests that arrive at the same moment, and even a year later.
  • Customers are unique on externalRef, but a repeat isn't replayed. Creating a customer with a reference that's already used returns 409 conflict. Look the customer up first with GET /customers?externalRef=ws_42, or use an idempotency key.

Fix 2: an idempotency key on every create

An idempotency key is a unique label you attach to a request so the receiving system can recognise a retry. Send the same request with the same key, and you get the first response back instead of a second record.

In Flux, you send it as an Idempotency-Key header on creates (customers, invoices and webhook endpoints). Flux keeps each key for 24 hours, per API key:

You send a key that was used before… Flux
with the same request Replays the first response, marked Idempotent-Replayed: true. Nothing runs again
with a different request Refuses it: 422 idempotency_key_reused
while the first request is still running Refuses it: 409 idempotency_in_progress. Wait and retry
after the first failed with a server error Runs it, because the key wasn't kept

Two rules make keys work:

  1. Derive the key from your own ids, such as create-customer-ws_42, or generate one and save it before sending, so a retry after a crash uses the same key.
  2. Use a new key for each distinct operation. Reusing one for a different body is refused, by design.

A worked example

The academy's website bills ws_42, a learner from Trans-Amadi, ₦450,000 for November:

curl -X POST "https://api.flux-erp.app/api/v1/invoices" \
  -H "Authorization: ApiKey $FLUX_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: invoice-ws_42-2026-11" \
  -d '{"externalRef":"ws_42-2026-11","customerId":812,"currency":"NGN",
       "dueDate":"2026-11-30","lines":[{"serviceId":9,"quantity":1,"unitPrice":450000.00}]}'

The request times out. The website retries with the same key and the same body: Flux replays the original result. Suppose the website's server had restarted and lost the key: the retry carries a new key, but externalRef is still ws_42-2026-11, so Flux returns the invoice it already raised. Either way, one invoice.

Duplicates arriving in your direction

Webhooks can be delivered more than once too, because they're retried. The fix is the same idea in reverse: record each event's id and skip any you've already processed. Flux sends it as the Flux-Event-Id header. Our webhooks guide covers this with the other things a receiver should handle.

Cleaning up what's already there

If duplicates already exist, prevention won't remove them. Before you integrate:

  • Export your customer list and sort by tax ID, then by email. Matching TINs are the strongest sign of a duplicate.
  • Agree which record survives, and move history to it, before switching the integration on.
  • Add your system's id as the externalRef on each surviving customer, so the integration matches them from day one.
  • Check open invoices for pairs with the same customer, amount and period.

How Flux helps

The Flux integration API supports Idempotency-Key on creates and permanent externalRef matching on invoices, so retries don't double-bill, and it refuses a second customer with a reference that's already taken. Invoices raised through the API still pass through approval, which gives a person one more chance to spot something odd. See how it fits together on the Developers page, and the full idempotency guide at developers.flux-erp.app.

From the blog