How the deduplication works
The tool splits your input on line breaks, walks the lines in order, and keeps a line the first time it appears while discarding every later repeat. Order is preserved, so the output reads in the same sequence as the input with the repeats removed. The counts shown above each panel let you see immediately how many lines were dropped.
Comparison is exact and case-sensitive. Apple and apple are two different lines, and so are a line with a trailing space and the same line without one. That strictness is intentional: a case-insensitive or trimmed comparison would silently merge entries that differ meaningfully, which matters when the lines are identifiers, file paths or passwords.
Keeping the first occurrence rather than the last is the convention here. If you need to keep the last occurrence of each repeated line, reverse the text, deduplicate, and reverse it again.
Why sorting and deduplicating are different operations
The familiar command line idiom sort | uniq exists because the classic uniq only collapses adjacent duplicates, so the input has to be sorted first. The side effect is that the output comes back in sorted order, which destroys whatever sequence the file had.
This tool tracks every line it has already seen rather than only comparing with the previous one, so it removes duplicates wherever they occur and the original order survives. That is the behaviour of sort -u only in the sense of removing repeats; the equivalent of preserving order is awk '!seen[$0]++'.
The distinction matters whenever sequence carries meaning: a log file, a playlist, a priority-ordered task list, a set of steps. Sorting such a file to deduplicate it damages the data more than the duplicates did.
Whitespace and line endings, the usual sources of surprise
Two lines that look identical on screen frequently are not. A trailing space, a tab used where spaces were used elsewhere, or a non-breaking space pasted from a document all make lines distinct to an exact comparison while looking the same to you. If a duplicate you can clearly see is not being removed, invisible whitespace is the first thing to check.
Line endings are the other common cause. Text produced on Windows ends lines with a carriage return followed by a line feed. Splitting on the line feed leaves the carriage return attached to the end of each line, so a line from a Windows file and a visually identical line typed here differ by that invisible character and will both be kept. Converting the text to Unix line endings before pasting avoids this entirely.
Zero-width characters cause the same problem and are even harder to spot, since they have no width at all. Text copied out of word processors, design tools and web pages carries them regularly. If lines refuse to match for no visible reason, run the text through a zero-width character detector.
Where this is useful
Email and contact lists are the most common case. Merging exports from several sources produces repeats that will otherwise cause duplicate sends, and deduplicating before import is far easier than cleaning up afterwards. The same applies to keyword lists, URL lists and tag lists assembled from multiple places.
It is also handy for log triage. Pulling the error lines out of a log and deduplicating collapses a thousand repetitions of one failure into a single line, which makes the actual variety of problems visible in a way that reading the raw file does not.
For data that has structure, be careful about what a line represents. Deduplicating CSV rows works only when the entire row is the unit of identity; if two rows differ in one timestamp column but describe the same entity, an exact line comparison will keep both. That kind of deduplication needs a spreadsheet or a script that can compare a chosen key.