Remove Duplicate Lines, Including Ones That Only Look Identical
Two lines that both read "한글" on screen can be different strings underneath, because Hangul copied from a Mac is stored as separate letter components. This tool normalises before it compares, so duplicates disappear the way your eyes say they should.
Removing duplicate lines sounds trivial: walk the list, remember what you have seen, keep the first of each. That is exactly what happens, and most of the time it is all you need.
The interesting part is deciding what counts as the same. Whether case matters and whether surrounding whitespace matters are choices you can make with the options. Korean text adds one more trap that no option can settle for you.
The same Hangul, stored two ways
Unicode can hold Hangul in two forms. The syllable 한 can be one precomposed character, U+D55C, or three separate jamo — U+1112, U+1161, U+11AB — for its initial, medial and final sounds. The first is called NFC, the second NFD.
Both draw as 한 on screen, because the font composes the jamo for display. As strings they differ from the very first property: 한글 is two characters in NFC and six in NFD.
macOS stores filenames in the decomposed form, so Korean text copied out of a Mac file listing, or out of certain applications, arrives as NFD. Text typed on Windows is NFC. Merge lists from both sources and lines that are visually identical will refuse to deduplicate.
This tool normalises to NFC for the comparison only. The lines it writes out are your originals, so deduplicating never silently rewrites your characters.
Deciding what counts as the same
Turning off case sensitivity treats Apple and apple as one line, which is what you want for email addresses or tag lists. Be careful with lists containing names or proper nouns, though, since the surviving line keeps whichever capitalisation appeared first and the other spelling is gone.
Ignoring surrounding whitespace treats identical content with different indentation as one line. Merging lists exported from Excel or written by several people almost always needs this. Unlike the case option, it also trims the whitespace from the output.
You can also choose whether the first or the last occurrence survives. Either way the result keeps the original order — which is the real difference between this and sorting the list and running uniq over it. For anything where sequence carries meaning, such as a log, that distinction matters.
What the option combinations do
Input: three lines reading Apple, apple, and " Apple" with a leading space.
| Case sensitive | Ignore whitespace | Lines left | Why |
|---|---|---|---|
| On | Off | 3 | All three are distinct strings |
| On | On | 2 | The whitespace-only difference collapses |
| Off | Off | 2 | The case-only difference collapses |
| Off | On | 1 | All three become one |
Good uses for this
- •Removing repeat recipients from an email list assembled from several sources.
- •Tidying a list of keywords or hashtags.
- •Merging file listings produced on a Mac and on Windows.
- •Reducing a log to the distinct error lines it contains.
- •Finding duplicate submissions in survey responses or a signup list.
Extended FAQ
Two lines look identical but are not being deduplicated.
For Korean text this is almost always NFC and NFD mixed together, which this tool handles by normalising before comparing — just paste it in. For other text, suspect an invisible whitespace character: try ignoring surrounding whitespace, or run the extra-spaces tool over it first.
Does it reorder my list?
No. Surviving lines stay in their original order, so unlike sort-then-uniq you do not lose sequence information.
When does keeping the first differ from keeping the last?
Only when the case or whitespace options have merged lines that were not byte-identical. Then the choice decides which spelling survives.
Are blank lines removed as duplicates?
Yes — several blank lines reduce to one. If your goal is to preserve paragraph structure, the line break remover's blank-line handling fits better.
Is my list stored anywhere?
No. Everything runs in your browser and nothing is sent to a server.