Merged CSV has duplicate header rows: why and how to remove them

Why it happens

Command-line concatenation has no idea a CSV has a header; it appends text. Three files with a header each give one real header and two that land in the data. They turn numeric columns into text, add bogus categories to pivot tables, and count as extra rows.

Seven lines, but only four orders. A COUNTA on order_id says six, and a SUM on amount may ignore the header text silently, so the error is easy to miss.

Skip the headers while merging

On a Mac or Linux, take the header once and then every file without its first line:

head -n 1 jan.csv > ../merged.csv
tail -q -n +2 *.csv >> ../merged.csv   # -q stops tail printing file names (GNU tail)
# macOS tail has no -q; use: for f in *.csv; do tail -n +2 "$f"; done >> ../merged.csv

On Windows, PowerShell parses each file, so the header is used once:

Get-ChildItem -Filter *.csv | ForEach-Object { Import-Csv $_.FullName } |
  Export-Csv ..\merged.csv -NoTypeInformation -Encoding UTF8

Write the output outside the folder (or with another extension), otherwise a rerun picks up the merged file as one of its inputs. Export-Csv takes its columns from the first file, so all files must share the same headers. Full walk-throughs are in merging CSV files on Windows and combining CSV files on a Mac.

Watch for the missing final newline

If a file doesn't end with a line break, copy or cat glues the next file's first line onto its last row, producing something like A-102,2026-01-19,75.50order_id,date,amount. That row is neither a header nor valid data, so a header filter won't catch it. Check the line after each file boundary, or merge with a parser (PowerShell, Python, Power Query) that treats each file separately.

Remove repeated headers after merging

1. In Excel, turn on a filter (Data > Filter) on the merged sheet.
2. On the first column, filter to show only the header text, for example order_id.
3. Select the visible rows below the real header, right-click > Delete Row, then clear the filter.

In Power Query, Get Data > From Folder combines files and promotes each file's header for you, so repeats never appear. If you already have a merged file, load it, use Use First Row as Headers, then filter the first column with does not equal order_id. In pandas, df = df[df['order_id'] != 'order_id'] does the same.

If the header row was exported with a byte order mark, the repeated copies may read as order_id and slip through an exact filter. Filter with contains instead, or see CSV encoding problems.

Confirm the merge

• Rows in the output should equal the sum of each file's rows minus one header per file.
• Numeric columns should convert to numbers with no text values left.
• A distinct count of the first column should not include the header name.

Analistable avoids the problem by loading each CSV as its own table in the browser and stacking them in SQL, so each file's header is read as column names. See the merge CSV files guide for other methods.

Doing it in Python

If you already merge with a script, read each file with a CSV parser rather than concatenating text. Each file's header becomes the column names once, so nothing to strip, and you can record the source:

import glob
import pandas as pd

frames = []
for f in sorted(glob.glob("monthly/*.csv")):
    df = pd.read_csv(f, dtype=str, encoding="utf-8-sig")
    df["source_file"] = f
    frames.append(df)

merged = pd.concat(frames, ignore_index=True)
merged = merged[merged["order_id"] != "order_id"]  # in case a file already contained repeats
merged.to_csv("merged.csv", index=False)

Reading with utf-8-sig removes a byte order mark if one is present, and dtype=str stops IDs or codes being converted on the way through. The filter line handles a source file that was itself produced by an earlier bad merge, which happens more often than you would expect with shared folders.

pandas aligns columns by name, so if one month's export adds or reorders a column, the merge still lines up and fills the gaps with empty values. A plain text merge would put values in the wrong columns instead. That is a second reason to prefer a parser when files come from a system that changes its export format from time to time.

Frequently asked questions

Why does copy *.csv repeat my headers?

copy concatenates text files byte for byte. It doesn't know the first line is a header, so each file's header is copied too.

How do I merge CSVs and keep only one header in Windows?

Use PowerShell's Import-Csv and Export-Csv, or Power Query's From Folder, both of which treat the first line of each file as headers.

Why didn't my filter catch every repeated header?

Usually a byte order mark or trailing space makes some copies slightly different, or a missing final newline glued a header onto a data row.

Why is my merged row count higher than expected?

Each repeated header counts as a row. Subtract one line per source file after the first, or remove the repeats and count again; any remaining difference points to blank lines or rows glued together at file boundaries.

Can Power Query's From Folder produce duplicate headers?

Not normally, because it promotes headers in each file before appending. Repeats appear only if a source file already contains them, or if the header step was removed from the sample file transformation.

Related guides

• How to merge CSV files on Windows
• How to combine CSV files into one on a Mac
• How to split a large file into chunks and rejoin it safely
• The best free ways to merge CSV files
• How to combine CSV files in Excel