Will Refresh in Power Query re-add old months?

It re-reads every file in the folder each time, so old months are included again, but the dedupe step runs again too and the result stays correct.

Can I dedupe in Google Sheets?

Yes. Data > Data clean-up > Remove duplicates works on selected columns, or use UNIQUE on a stacked range with curly braces.

Does Excel's Remove Duplicates keep the first or last row?

The first occurrence from the top of the range. Sort so the version you want to keep comes first.

Should I dedupe on all columns?

Usually not. A record that changed between exports differs in some column and would survive. Dedupe on the record's key.

How do I know which export a row came from?

Keep Power Query's Source.Name column, or add a file name or export date column when you stack. See our guide to adding a source column.

What if there's no unique ID?

Build a composite key from several columns, accept that identical genuine rows may merge, and review those cases.

How to append monthly files without duplicates

By the Analistable team · Updated 2026-10-10 · 5 min read

When monthly exports overlap (for example each covers the last 35 days), stacking them repeats some rows. Pick a natural key that identifies a record, such as order_id or order_id plus line number, append the new file, then remove duplicates on that key, keeping the most recent version. Power Query does this with Append then Remove Duplicates; Excel 365 can do it with VSTACK and UNIQUE.

Part of our guide: How to consolidate data in Excel

Why overlaps happen

Many systems export "last 30 days" or "this month to date" rather than exact calendar months. Others re-export a record when it changes, such as an order that was later refunded. Either way, appending the files row for row double-counts. Automating the stack itself is covered in combining multiple Excel files automatically; this page adds the dedupe step.

Three overlapping exports
File	Date range	Rows
export_aug.xlsx	28 Jul – 31 Aug	1240
export_sep.xlsx	27 Aug – 30 Sep	1185
export_oct.xlsx	26 Sep – 31 Oct	1310
Stacked		3735
After dedupe on order_id		3512

223 rows were in two exports because of the overlapping days (3,735 − 3,512 = 223). Without the dedupe step, revenue for the boundary days would be counted twice.

Choose the natural key

Order-level data: order_id.
Line-level data: order_id plus line_number (or sku). Order ID alone would wrongly remove genuine extra lines.
Bank or card transactions: the provider's transaction ID. If there isn't one, date + amount + reference is a fallback, but it can merge two real identical payments, so review those.
Snapshots (stock levels, balances): item + snapshot date. Here you usually don't want to dedupe across dates at all.

Removing duplicates across all columns only catches perfectly identical rows. If a status changed between exports, the rows differ and both survive, so dedupe on the key.

Power Query: append, sort, remove duplicates

Put the monthly files in one folder and use Data > Get Data > From File > From Folder > Combine & Transform (see combine files from a folder).
Keep the Source.Name column, or add an export date column, so you know which file each row came from.
Sort by export date descending, so the newest copy of each record comes first.
Select the key column(s), right-click > Remove Duplicates.
Close & Load. Next month, drop the new file in the folder and click Refresh.

Power Query's Remove Duplicates keeps the first occurrence, but sorting before it isn't always respected because the engine can reorder rows. Wrap the sorted step in Table.Buffer to make it reliable:

let
    Source   = Folder.Files("C:\Exports"),
    // ... combine steps that produce Combined with an ExportDate column ...
    Sorted   = Table.Buffer(Table.Sort(Combined, {{"ExportDate", Order.Descending}})),
    Deduped  = Table.Distinct(Sorted, {"order_id"})
in
    Deduped

Excel 365 formulas

For a few files on separate sheets, a single formula can stack them and keep one row per key. This keeps the first match it meets, so stack the newest sheet first:

=LET(
  data, VSTACK(Oct!A2:F2000, Sep!A2:F2000, Aug!A2:F2000),
  data2, FILTER(data, CHOOSECOLS(data, 1) <> ""),
  keys, CHOOSECOLS(data2, 1),
  firstPos, XMATCH(UNIQUE(keys), keys),
  CHOOSEROWS(data2, firstPos)
)

XMATCH returns the position of the first occurrence of each unique key, and CHOOSEROWS pulls those rows. VSTACK, CHOOSECOLS and CHOOSEROWS need Microsoft 365 or Excel 2024. For a deeper look see VSTACK and dynamic arrays.

Python for a running master file

import pandas as pd

master = pd.read_csv("master.csv", dtype={"order_id": str})
new = pd.read_excel("export_oct.xlsx", dtype={"order_id": str})
new["export_date"] = "2026-10-31"

combined = pd.concat([master, new], ignore_index=True)
combined = (combined.sort_values("export_date")
                    .drop_duplicates(subset=["order_id"], keep="last"))
combined.to_csv("master.csv", index=False)

keep="last" after sorting by export date keeps the newest version of each order, so a refund that appears in the October export replaces the August copy.

Checks each month

Rows added this month = new file rows − rows already in master. Log that number.
Totals for overlap days should match the source system's own report.
Keys that appear more than twice in the stacked data deserve a look; they often signal a key that isn't really unique.

If the goal is just to answer questions across the months, Analistable can load each export and query them together in your browser, and you can ask it to count each order once. It doesn't write a master file back for you. The consolidation guide covers summary-style alternatives.

Overlap or genuine repeat?

Not every repeated key is an overlap. Some systems reuse IDs, for example order numbers that restart each year or in each store. Before trusting a dedupe, look at a sample of the removed rows: if the dates are far apart or the amounts differ a lot, the key isn't unique on its own and you need to add year or store to it.

It also helps to check what the export actually covers. If the system lets you export by creation date instead of a rolling window, exact calendar months remove most overlaps at the source, leaving dedupe as a safety net rather than the main mechanism.

Keeping history instead of overwriting

Sometimes you want every version, for example to see when an order's status changed. Then don't drop older copies: keep the export date column and query the latest version per key when you need current figures. Storage is cheap; a lost history usually isn't recoverable.

Frequently asked questions

Will Refresh in Power Query re-add old months?: It re-reads every file in the folder each time, so old months are included again, but the dedupe step runs again too and the result stays correct.
Can I dedupe in Google Sheets?: Yes. Data > Data clean-up > Remove duplicates works on selected columns, or use UNIQUE on a stacked range with curly braces.
Does Excel's Remove Duplicates keep the first or last row?: The first occurrence from the top of the range. Sort so the version you want to keep comes first.
Should I dedupe on all columns?: Usually not. A record that changed between exports differs in some column and would survive. Dedupe on the record's key.
How do I know which export a row came from?: Keep Power Query's Source.Name column, or add a file name or export date column when you stack. See our guide to adding a source column.
What if there's no unique ID?: Build a composite key from several columns, accept that identical genuine rows may merge, and review those cases.