[PR #40] [MERGED] Use jwalk instead of walkdir: parallel walking for performance boost #269

New Issue

2026-06-08T11:26:37+03:00

zhus commented

2026-06-08 11:26:37 +03:00

📋 Pull Request Information

Original PR: https://github.com/bootandy/dust/pull/40
Author: @AdminXVII
Created: 11/25/2019
Status: ✅ Merged
Merged: 12/4/2019
Merged by: @bootandy

Base: master ← Head: parallel-walk

📝 Commits (6)

86b3ccc perf(IO): use parallel walkdir (jwalk) for super faster traversal
f8ce6c9 Use more rusty patterns and preallocate enough space
bf28d42 Update Performance section of README
62ac9b6 Make sure to count the hidden directories
19a41aa Add CLI option for the number of threads to spawn
b66523c Apply clippy lints

📊 Changes

7 files changed (+257 additions, -80 deletions)

View changed files

📝 Cargo.lock (+170 -14)
📝 Cargo.toml (+1 -1)
📝 README.md (+1 -1)
📝 src/display.rs (+7 -7)
📝 src/main.rs (+32 -26)
📝 src/utils/mod.rs (+36 -26)
📝 src/utils/platform.rs (+10 -5)

📄 Description

Use a parallel walkdir implementation that is able to fetch more resources at the same time. This should greatly improve directory trees with lots of branching, at the cost of threading. Also make sure to allocate enough space as much as possible ahead of time to avoid reallocation. Lastly, remove on a vec needs to make an allocation each time, so prefer the retain method for repeated removal.

Some benchmark on my computer (galago pro 3):

Folder 1 (medium, shallow):
dust: 0.055011367s
du -sh: 0.245766881s

Folder 2 (small, shallow):
du -sh: 0.007830246s
dust: 0.025789946s

Folder 3 (large, deep):
dust: 3.421158887s
du -sh: 21.322613990s

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/bootandy/dust/pull/40 **Author:** [@AdminXVII](https://github.com/AdminXVII) **Created:** 11/25/2019 **Status:** ✅ Merged **Merged:** 12/4/2019 **Merged by:** [@bootandy](https://github.com/bootandy) **Base:** `master` ← **Head:** `parallel-walk` --- ### 📝 Commits (6) - [`86b3ccc`](https://github.com/bootandy/dust/commit/86b3cccaf6f28314d251f17ff49afbb75dd58735) perf(IO): use parallel walkdir (jwalk) for super faster traversal - [`f8ce6c9`](https://github.com/bootandy/dust/commit/f8ce6c97bf54afa5183b7d3c5fe6f79e1e6a1532) Use more rusty patterns and preallocate enough space - [`bf28d42`](https://github.com/bootandy/dust/commit/bf28d424837cba93d3c7676dcfe0d6f9daf523d2) Update Performance section of README - [`62ac9b6`](https://github.com/bootandy/dust/commit/62ac9b623adcbc7cc148c523fa8e19ca36c287e8) Make sure to count the hidden directories - [`19a41aa`](https://github.com/bootandy/dust/commit/19a41aa3825a93b64e5b6f2dd719e4df4603e469) Add CLI option for the number of threads to spawn - [`b66523c`](https://github.com/bootandy/dust/commit/b66523cff3fc871b378d1f3ca4d3a970f2ab98dc) Apply clippy lints ### 📊 Changes **7 files changed** (+257 additions, -80 deletions) <details> <summary>View changed files</summary> 📝 `Cargo.lock` (+170 -14) 📝 `Cargo.toml` (+1 -1) 📝 `README.md` (+1 -1) 📝 `src/display.rs` (+7 -7) 📝 `src/main.rs` (+32 -26) 📝 `src/utils/mod.rs` (+36 -26) 📝 `src/utils/platform.rs` (+10 -5) </details> ### 📄 Description Use a parallel walkdir implementation that is able to fetch more resources at the same time. This should greatly improve directory trees with lots of branching, at the cost of threading. Also make sure to allocate enough space as much as possible ahead of time to avoid reallocation. Lastly, `remove` on a vec needs to make an allocation each time, so prefer the retain method for repeated removal. Some benchmark on my computer (galago pro 3): Folder 1 (medium, shallow): dust: 0.055011367s du -sh: 0.245766881s Folder 2 (small, shallow): du -sh: 0.007830246s dust: 0.025789946s Folder 3 (large, deep): dust: 3.421158887s du -sh: 21.322613990s --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

zhus added the pull-request label 2026-06-08 11:26:37 +03:00

zhus closed this issue

2026-06-08 11:26:38 +03:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bootandy/archived-dust#269