Remove duplicate lines from your text with a single click. Choose between case-sensitive or case-insensitive comparison. See statistics on how many lines were removed.
Saved Presets is a Supporter feature.
Tool History is a Supporter feature.
Tool Notes is a Supporter feature.
Enter text with duplicate lines in the input area.
Toggle case sensitivity based on your needs.
Copy the deduplicated text to your clipboard.
The tool uses JavaScript's Set data structure to track seen lines. A Set automatically ensures uniqueness and provides O(1) constant-time lookup, making it highly efficient even for large texts. As the tool iterates through lines, it checks if each line already exists in the Set. If not, the line is added to both the Set and the output array. If yes, it's skipped. This approach guarantees that only the first occurrence of each unique line is kept.
In case-insensitive mode, the tool maintains two separate representations of each line: the original casing (for output) and a lowercase version (for comparison). The Set stores lowercase versions for duplicate detection while the output array preserves original casing. This allows lines like "Apple", "APPLE", and "apple" to be recognized as duplicates while displaying whichever case appeared first in your input.
The tool splits text by newline characters (\n, \r\n, or \r) to create an array of lines. Different operating systems use different line ending conventions (Windows uses \r\n, Unix/Mac use \n), and the tool handles all formats correctly. After splitting, each line is processed individually without modification to the line content itself - only the line endings are normalized in the output.
The tool provides real-time statistics showing the number of original lines, unique lines remaining, and how many duplicates were removed. These metrics are calculated during the deduplication process: original lines = input.split('\n').length, unique lines = Set.size, removed = original - unique. This gives you immediate feedback about the effectiveness of the deduplication.
All processing happens entirely in your browser using native JavaScript array methods and Set operations. There's no server communication - the tool is a pure client-side application. For typical use cases (under 100,000 lines), performance is instant. The algorithmic complexity is O(n) where n is the number of lines, making it scale linearly with input size.
The tool scans each line of your text and keeps track of lines it has already seen using a JavaScript Set data structure, which provides O(1) lookup time for efficient duplicate detection. When it encounters a duplicate, it removes it while preserving the original order of unique lines. The first occurrence of each unique line is kept, and all subsequent duplicates are discarded.
