[PR #74] [MERGED] Avoid opening all files for reading on windows #289

New Issue

2026-06-08T11:26:49+03:00

zhus commented

2026-06-08 11:26:49 +03:00

📋 Pull Request Information

Original PR: https://github.com/bootandy/dust/pull/74
Author: @rasmushalland
Created: 2/24/2020
Status: ✅ Merged
Merged: 3/1/2020
Merged by: @bootandy

Base: master ← Head: win-perf

📝 Commits (3)

c30f31c Avoid opening all files for reading on windows
2c58041 Clean up windows performance
efb455c Opening files on windows got a lot cheaper.

📊 Changes

2 files changed (+148 additions, -42 deletions)

View changed files

📝 src/utils/mod.rs (+50 -30)
📝 src/utils/platform.rs (+98 -12)

📄 Description

It can be very expensive to do that, especially when it causes windows defender to read the files and scan them.

Savings in orders of magnitude in terms of time, io and cpu have been observed on hdd, windows 10, some 100Ks files taking up some hundreds of GBs:
Consistently opening the file: 30 minutes.
With this optimization: 8 sec.

Hard links: Unresolved. We don't get inode/file index, so hard links count once for each link. Hopefully they are not too commonly in use on windows.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/bootandy/dust/pull/74 **Author:** [@rasmushalland](https://github.com/rasmushalland) **Created:** 2/24/2020 **Status:** ✅ Merged **Merged:** 3/1/2020 **Merged by:** [@bootandy](https://github.com/bootandy) **Base:** `master` ← **Head:** `win-perf` --- ### 📝 Commits (3) - [`c30f31c`](https://github.com/bootandy/dust/commit/c30f31c22cc89af1b50897bb2d7eea0cfa3e6e76) Avoid opening all files for reading on windows - [`2c58041`](https://github.com/bootandy/dust/commit/2c58041885232949d4a946de340dd875c0310ec8) Clean up windows performance - [`efb455c`](https://github.com/bootandy/dust/commit/efb455c73930d44d94c369cdfac2858e01d5c016) Opening files on windows got a lot cheaper. ### 📊 Changes **2 files changed** (+148 additions, -42 deletions) <details> <summary>View changed files</summary> 📝 `src/utils/mod.rs` (+50 -30) 📝 `src/utils/platform.rs` (+98 -12) </details> ### 📄 Description It can be very expensive to do that, especially when it causes windows defender to read the files and scan them. Savings in orders of magnitude in terms of time, io and cpu have been observed on hdd, windows 10, some 100Ks files taking up some hundreds of GBs: Consistently opening the file: 30 minutes. With this optimization: 8 sec. Hard links: Unresolved. We don't get inode/file index, so hard links count once for each link. Hopefully they are not too commonly in use on windows. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

zhus added the pull-request label 2026-06-08 11:26:49 +03:00

zhus closed this issue

2026-06-08 11:26:51 +03:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bootandy/archived-dust#289