[GH-ISSUE #85] Any room for performance improvement? #32

New Issue

2026-06-08T11:25:23+03:00

zhus commented

2026-06-08 11:25:23 +03:00

Originally created by @kawpii on GitHub (Mar 24, 2020).
Original GitHub issue: https://github.com/bootandy/dust/issues/85

Benchmark

hyperfine 'dutree app/'
Benchmark #1: dutree app/
  Time (mean ± σ):      57.4 ms ±   1.0 ms    [User: 19.2 ms, System: 37.8 ms]
  Range (min … max):    55.9 ms …  61.9 ms    51 runs

hyperfine 'dust app/'
Benchmark #1: dust app/
  Time (mean ± σ):      20.6 ms ±   2.8 ms    [User: 29.0 ms, System: 15.6 ms]
  Range (min … max):    17.2 ms …  27.0 ms    108 runs
 
hyperfine 'du app/'
Benchmark #1: du app/
  Time (mean ± σ):       6.3 ms ±   0.2 ms    [User: 2.6 ms, System: 3.6 ms]
  Range (min … max):     5.9 ms …   7.7 ms    401 runs

dust is ~3x slower than du but ~2.5x faster than duetree

Improvement suggestions from https://github.com/nachoparker/dutree/issues/18 may apply

Using https://github.com/BurntSushi/walkdir
Minimize # sys calls
Using pointer for the tree structure

Originally created by @kawpii on GitHub (Mar 24, 2020). Original GitHub issue: https://github.com/bootandy/dust/issues/85 **Benchmark** ``` hyperfine 'dutree app/' Benchmark #1: dutree app/ Time (mean ± σ): 57.4 ms ± 1.0 ms [User: 19.2 ms, System: 37.8 ms] Range (min … max): 55.9 ms … 61.9 ms 51 runs hyperfine 'dust app/' Benchmark #1: dust app/ Time (mean ± σ): 20.6 ms ± 2.8 ms [User: 29.0 ms, System: 15.6 ms] Range (min … max): 17.2 ms … 27.0 ms 108 runs hyperfine 'du app/' Benchmark #1: du app/ Time (mean ± σ): 6.3 ms ± 0.2 ms [User: 2.6 ms, System: 3.6 ms] Range (min … max): 5.9 ms … 7.7 ms 401 runs ``` `dust` is ~3x slower than `du` but ~2.5x faster than `duetree` Improvement suggestions from https://github.com/nachoparker/dutree/issues/18 may apply - Using https://github.com/BurntSushi/walkdir - Minimize # sys calls - Using pointer for the tree structure

zhus closed this issue

2026-06-08 11:25:23 +03:00

zhus commented

2026-06-08 11:25:23 +03:00

@bootandy commented on GitHub (Mar 25, 2020):

We moved from walkdir to jwalk because jwalk was faster for large filesystems. jwalk can run in parallel walkdir can't.

I have a branch where I have moved from jwalk to Burntsushi's ignore crate which seems to be even faster than jwalk.

I don't think I can remove any more sys calls but please feel free to show me where I can.

We already use pointers for the tree structure, we don't clone the objects

One way we could improve performance for the base case is to stop sorting the output and instead search linearly for the k largest objects but I think that improvement would be very minor in practice.

@bootandy commented on GitHub (Mar 25, 2020): We moved from walkdir to jwalk because jwalk was faster for large filesystems. jwalk can run in parallel walkdir can't. I have a branch where I have moved from jwalk to Burntsushi's ignore crate which seems to be even faster than jwalk. I don't think I can remove any more sys calls but please feel free to show me where I can. We already use pointers for the tree structure, we don't clone the objects One way we could improve performance for the base case is to stop sorting the output and instead search linearly for the k largest objects but I think that improvement would be very minor in practice.

zhus commented

2026-06-08 11:25:23 +03:00