TToolbox
≣

Dedup & Sort

Remove duplicate lines and sort

About this tool

Cleaning text data across server logs, marketing email lists, web scraper results, and database seed tables constantly demands eliminating duplicate lines and sorting chaotic records. Manual inspection is exhausting and error-prone. This tool delivers client-side high-throughput line deduplication and natural sorting, supporting case sensitivity toggles, numerical ordering, and empty-line removal.

How to Use This Tool (Step-by-Step)

1

Paste Text List

Paste your raw multi-line list or log dump into the editor. The parser smoothly loads tens of thousands of lines.

2

Configure Deduplication & Sorting

Toggle case sensitivity (Apple vs apple), strip whitespace padding, remove empty rows, and select Ascending (A-Z) or Descending (Z-A) sort.

3

Inspect Metrics and Copy Clean List

Review real-time metrics showing original count, unique count, and removed duplicates. Copy the cleaned list with one click.

Sub-Millisecond Hash Deduplication

Powered by native JavaScript `Set` hash lookups, filtering duplicate entries across tens of thousands of lines in milliseconds.

Human-Intuitive Natural Sort

Implements true natural sorting (`Intl.Collator`) so that 'item2' correctly precedes 'item10', avoiding naive ASCII sorting bugs.

100% Client-Side Air-Gapped

Deduplication and sorting execute strictly inside your local browser memory. Sensitive customer lists and internal emails remain private.

Data Cleaning Best Practices & Sorting Tips

  • •Beware of trailing spaces: 'admin' and 'admin ' appear identical to human eyes but represent distinct strings. Always enable 'Trim' before deduplicating.
  • •Natural sort vs ASCII sort: Raw ASCII sorting places 'file10' before 'file2' because '1' < '2'. Natural sorting recognizes multi-digit numbers properly.
  • •Stable deduplication: If you need to strip duplicates while preserving the first-seen chronological sequence of items, keep sorting disabled.

Code & Transformation Examples

Raw List Deduplication and Sorting Benchmark

Raw Input with Duplicates
apple
Banana
orange
apple
banana
10_item
2_item
Deduplicated & Naturally Sorted
2_item
10_item
apple
Banana
orange

* Eliminates duplicate entries while ensuring '2_item' naturally precedes '10_item'.

JavaScript Deduplication & Natural Sort Snippet

JavaScript Implementation
function dedupAndSort(text, caseSensitive = false) {
  const lines = text.split(/\r?\n/).map(l => l.trim()).filter(Boolean);
  const seen = new Set();
  const unique = lines.filter(line => {
    const key = caseSensitive ? line : line.toLowerCase();
    if (seen.has(key)) return false;
    seen.add(key);
    return true;
  });
  // 自然排序 (适应文件名与编号)
  return unique.sort(new Intl.Collator(undefined, { numeric: true }).compare);
}
Core Benefit
借助 Set.prototype.has() 实现 O(1) 的超极速查重复杂度,结合 Intl.Collator 实现全球语言友好的自然排序。

* Intl.Collator provides native locale-sensitive numerical sorting without third-party libraries.

Frequently Asked Questions

Q: How does the tool handle case sensitivity (Apple vs apple)?

A: With case sensitivity disabled, 'Apple' and 'apple' are treated as identical and condensed to the first-seen entry. Enabling it preserves both variations.

Q: Will pasting tens of thousands of lines freeze the page?

A: Not at all. Modern V8 JavaScript engines process 50,000 text rows in under 50 milliseconds using Set lookups. The interface remains snappy.

Q: Can the tool count how many times each item repeated?

A: Yes. Toggle 'Show repetition counts' to prepend frequency tags (e.g., '3 | [email protected]'), ideal for analyzing recurring error logs.

Related Tools

All processing happens locally in your browser. Your data is never uploaded to a server.
/