Find Duplicate Files

PracticalmediumPremium

45:00

Desktop only

Coding Challenges are optimized for Desktop

Come back on a wider screen - the live editor needs room to breathe. You can still read the problem brief above.

Problem

Backup tools and deduplication systems avoid storing the same content twice by fingerprinting each file. Byte-by-byte comparison doesn't scale; a cryptographic hash lets you identify identical files with a single streaming pass. You will implement a duplicate finder that scales to large files without loading them into memory.

Return a list of groups, each containing two or more paths with identical content. Singletons are excluded. Order within groups and between groups does not matter.

Level 1 - Streaming hash

Constraints:

Don't load whole files into memory. Read in 8 KB chunks.
Use SHA-256 for content hashing.
Empty files are valid content - two empty files are duplicates of each other.
Singletons are excluded from the result.

Example:

paths = ["report.pdf", "report_copy.pdf", "summary.txt"]
# report.pdf and report_copy.pdf have the same content
# summary.txt is unique
 
find_duplicates(paths) == [["report.pdf", "report_copy.pdf"]]

Level 2 - `min_size`, `follow_symlinks`, size pre-filter

def find_duplicates(
    paths: list[str],
    min_size: int = 0,
    follow_symlinks: bool = True,
) -> list[list[str]]:

Three layered changes:

Size pre-filter. Files with a unique size cannot be duplicates of anything - skip hashing them entirely. Only hash files within a size cohort of two or more.
min_size: skip files smaller than min_size bytes. They never appear in the output, even as duplicates of each other.
follow_symlinks: when False, skip symlinks entirely (don't stat their target, don't include them in groups). When True (default), follow them.

The result contract is unchanged. Inputs are assumed to be readable files; an unreadable input (for example a followed broken symlink) propagates its OSError. Per-file error capture is the job of the Extended variant.

Example:

paths = ["a.txt", "b.txt", "c.txt", "d.txt", "e.txt"]
# a.txt == b.txt (same size and content), c.txt == d.txt, e.txt is unique
 
find_duplicates(paths) == [["a.txt", "b.txt"], ["c.txt", "d.txt"]]
# (order within each group and between groups does not matter)

Loading editor...

The live editor for this challenge unlocks with Premium.

Browse free problems

0/0 passed