Duplicate Line Remover

How do you remove duplicate lines and check the result?

Paste one item per line into Input. Result updates automatically, keeping the first copy of each exact line by default. For a worked example, enter the following five lines with no blank line at the end. Leave case comparison, trimming, sorting, counts, and consecutive-only mode off.

apple
pear
apple
plum
pear

Results for the same five-line list with different standard options

Selection	Result, one item per line	Output lines	Removed
All unique lines, Keep first occurrence	apple / pear / plum	3	2
All unique lines, Keep last occurrence	apple / plum / pear	3	2
Only duplicated lines, Keep first occurrence	apple / pear	2	3
Only lines that occur once	plum	1	4

The slashes above represent line breaks. All unique lines means one representative from every matching group, including repeated groups. Only lines that occur once excludes every group with repeats. Keep last occurrence orders retained copies by their last positions; it does not simply reverse the first-copy result.

How does the matching algorithm decide which lines are duplicates?

Standard processing splits text at newline characters, optionally trims each line, optionally removes whitespace-only entries, then groups matching text using a JavaScript Map. Each group retains its occurrences and original line numbers. Case-insensitive comparison uses lowercase comparison keys, preserving the selected copy's casing in Result. Without that option, Error and error remain separate.

After grouping, the tool selects the requested groups and their first or last entries. It orders retained entries by input position, then applies alphabetical sorting if selected. Show occurrence counts prefixes each retained line with its group's size and a tab. Counts become part of the copied or downloaded text, so turn them off when you need a plain list.

What changes when you remove only consecutive duplicates?

Only remove consecutive duplicates groups adjacent matching entries instead of matching across the whole list. For red, red, blue, red on separate lines, it keeps red, blue, red. Global deduplication keeps just red and blue. Consecutive mode suits repeated log messages when a later recurrence after another message should remain visible.

Output filters and counts operate on these runs too: Only duplicated lines retains runs longer than one, while Only lines that occur once retains single-entry runs. Remove empty lines runs before grouping in standard mode, so deleting a blank separator can make two previously separated matching lines consecutive. Sorting happens afterward and does not change which runs were identified.

How do spaces, empty lines, and sorting affect the output?

Trim leading/trailing spaces changes both comparison text and output text. It removes edge whitespace, including tabs, but leaves internal spacing untouched. Without trimming, apple and apple followed by a space are different entries. Remove empty lines also removes whitespace-only rows even when trimming is off. With both options off, identical empty lines form a group just like other text.

A final newline creates a trailing empty entry in the line count. An entirely empty Input instead shows no text entered. Sort lines alphabetically uses JavaScript localeCompare on the retained text; it is not a numeric sort or a configurable locale selector. Leave sorting off when input order carries meaning, such as a sequence of events or an ordered checklist.

What does the ordered cleanup recipe do differently?

Select Use cleanup recipe instead of standard options to replace the standard controls with seven fixed steps: Normalize Unicode (NFC), Trim, Collapse spaces, Case-fold for comparison, Remove duplicates, Remove empty lines, and Sort. Each step is optional, but enabled steps always run in that order. Initially, normalization, trimming, and duplicate removal are selected.

NFC normalization makes composed and decomposed forms of an accented character comparable. Collapse spaces replaces runs of internal whitespace, including tabs, with one ordinary space. Case-fold for comparison changes keys rather than visible casing; its handling includes matching the German sharp s with ss. Duplicate removal keeps the first matching entry, and recipe sorting uses comparison keys with the en-US locale.

The recipe replaces keep-last, output filters, consecutive matching, and occurrence prefixes; those standard settings do not carry over. Step counts report lines changed, removed, or moved, rather than a single duplicate total. A line may change in several steps, so adding those counts does not tell you how many distinct original lines changed.

Can you clean spreadsheet columns, CSV rows, and configuration lists?

Use a single-column list for addresses, keywords, identifiers, or reference entries. For log analysis, deduplication compares the entire message line: different timestamps remain different. For bibliography cleanup, it removes identical entries rather than recognizing two citations to the same work. Review the matching rule before discarding information.

Open file offers .txt, .csv, .tsv, and .log files and normalizes carriage-return line endings to newlines. CSV content remains plain text: no column selection, header protection, or multiline-field parsing occurs. Trimming only removes whitespace at the edges of a complete row. Configuration entries with the same name but different values remain different lines; this is not a key-based conflict resolver.

How does this compare with spreadsheet and command-line cleanup?

Choose column-aware processing when duplicates are defined by a particular field rather than a complete line. This tool fits pasted lists where the whole line is the comparison unit. On the command line, sort -u sorts and deduplicates, while uniq collapses adjacent duplicates. Use this tool's consecutive option for that run-based behavior, or leave it off to match across the entire input without sorting first.

The uniq -d, uniq -u, and uniq -c patterns correspond to repeated runs, single-entry runs, and counts. Here, the selected consecutive setting determines whether similar output choices apply to runs or global groups.

How can you inspect removals and reuse the cleaned list?

Expand Removed lines to see dropped entries with their original one-based line numbers and original text. The preview shows the first 500 removals. The total removed count includes all discarded lines, including blank rows and groups excluded by an output filter; it does not always mean repeated copies alone. Compare total lines and output lines before exporting.

Copy takes the Result text, and Download saves deduplicated.txt as plain text. For a final check, use Character Counter to inspect text length or Word Counter to review the cleaned content. The broader Text Tools collection covers related text transformations.

What are the processing limits and privacy boundaries?

The tool reads files locally with FileReader and processes complete strings in browser memory, without an explicit size or line cap. There is no streaming mode or guaranteed capacity for a particular file size. Standard results refresh after a 200-millisecond delay following input or option changes; large inputs can take longer to compute.

The Content Security Policy permits connections only to this site. Duplicate removal itself makes no network requests, and the download is created from the local result. Matching is exact after your chosen transformations: the tool does not detect similar meanings, spelling mistakes, or near-duplicate sentences. Keep a source copy when you need to revisit the comparison rules.