Remove Duplicate Lines

How to use the remove duplicate lines

1. Paste your list, one entry per line.
2. Choose whether to trim whitespace, ignore case, sort or drop blank lines.
3. Copy or download the deduplicated list.

About this tool

Lists collected from several places almost always contain repeats: exported email addresses, keyword lists, URLs pulled from a crawl, IDs copied out of two spreadsheets. This tool keeps the first occurrence of each line and drops the rest, then reports how many it removed so you can sanity-check the result.

The options matter more than they look. Trimming whitespace first means 'apple' and 'apple ' are recognised as the same entry, which is usually what you want from copied data. Ignoring case merges 'Apple' and 'apple'. Neither is applied unless you ask, because for some data — passwords, case-sensitive IDs — that merge would be wrong.

Common uses

• Cleaning a mailing list before an import.
• Merging two keyword exports into one unique set.
• Removing repeated URLs from a crawl report.

Frequently asked questions

Which duplicate is kept?

The first one, in the order you pasted. If you enable sorting, the deduplicated result is then sorted alphabetically.

Is there a size limit?

There is no fixed limit, but very large lists (hundreds of thousands of lines) depend on your device's memory. The tool processes the text in the page, so nothing is uploaded regardless of size.