mirror of
https://github.com/bootandy/dust.git
synced 2026-06-08 11:29:05 +03:00
[GH-ISSUE #495] Dust never finishes #217
Assignees
zhus (Sergey Zhukov)
Clear assignees
zhus (Sergey Zhukov)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: bootandy/archived-dust#217
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @sasq64 on GitHub (May 23, 2025).
Original GitHub issue: https://github.com/bootandy/dust/issues/495
Did
cargo installand ran it in my home directory.Quickly counted up to
Indexing: . 2835562 files, 299G ... /but then doesn't seem to get any further. Interrupted it after 5 minutes.
@Amr-Shams commented on GitHub (May 23, 2025):
could u use
dust --print-errors@sankalp-khare commented on GitHub (May 29, 2025):
Same issue on
v1.2.0(installed using homebrew).Tried with
dust --print-errorsbut it kept spinning and no error details got printed:I built
v1.1.1locally and tested it on the same filesystem, and the issue does not happen.v.1.1.1handles theInterrupted system call (os error 4)and completes execution, see https://github.com/bootandy/dust/issues/495#issuecomment-2921686085Looks like a regression introduced in
v1.2.0.@sasq64 do you mind trying this on your end? Build with tag
v.1.1.1code and see if the issue is reproducible?Is there a way I could get dust to print what file it is currently processing? I don't see a
--verboseequivalent in the flags.@Amr-Shams commented on GitHub (May 29, 2025):
well there is no verbose in the flags
yet i have tried to add it locally to my version-tbh i don't want this flag to be there for many reasons
fork my repo I have added the following changes to the latest commit
@sasq64 commented on GitHub (May 29, 2025):
My suspicion was that it tried to access a folder that it didn't have permissions for, because I was using a new terminal emulator on macOS when I got the error...
@Amr-Shams commented on GitHub (May 29, 2025):
interesting but with the flag print-errors is enabled it should show this before the spawn
@sankalp-khare commented on GitHub (May 30, 2025):
@sasq64 are you saying the issue happens only with your new terminal emulator, but not with your previous one?
@sasq64 commented on GitHub (May 30, 2025):
I have not had it happen again, but I know that if you scan your home directory from a newly installed application on macOS, it will pop up dialogs asking for permissions to read various directories.
@sankalp-khare commented on GitHub (May 30, 2025):
@Amr-Shams thanks for your commit adding
-Voption!Using the release executable built from https://github.com/Amr-Shams/dust/commit/419e6699785c0b2b71434946988a4cd64d436eec I can see the issue:
is where it's getting stuck. I am, however, able to traverse the directory using
findwithout seeing any error:Am also able to stat it
And most stunning is that I am able to run
dust -Von the directory without facing any error:Moved up a few directories and am able to see the same error now on a different subpath:
@sankalp-khare commented on GitHub (May 30, 2025):
This error is intermittent on the same directory. I don't have any Pocket process running => I don't think any changes are happening real-time to the directory or its contents. Yet repeated invocations of
duston the dir sometimes run into this error and sometimes don't.@sankalp-khare commented on GitHub (May 30, 2025):
When the "interrupted system call (os error 4)" error happens,
v1.1.1 prints it, moves past it and prints the expected final output:
but v.1.2.0 gets stuck at Indexing:
same thing with https://github.com/Amr-Shams/dust/commit/419e6699785c0b2b71434946988a4cd64d436eec
--verbose:@sankalp-khare commented on GitHub (May 30, 2025):
I added some print statements to trace the execution, and it looks to me like it gets stuck in
handle_error_and_retry()https://github.com/bootandy/dust/blob/702f0f0fe9d00641bf392a9d3a7abb6501ca7464/src/dir_walker.rs#L290 called at https://github.com/bootandy/dust/blob/702f0f0fe9d00641bf392a9d3a7abb6501ca7464/src/dir_walker.rs#L264What's curious to me is that in the 2nd, 3rd, and 4th case of entering
handle_error_and_retry, each of which happen afterError reading directory <dir>: Interrupted system call (os error 4), it does not enter the Interrupted error handling block... 🤔It make sense then that we don't see the panic at https://github.com/bootandy/dust/blob/702f0f0fe9d00641bf392a9d3a7abb6501ca7464/src/dir_walker.rs#L310 because that block isn't executing more than 1x...
@sankalp-khare commented on GitHub (May 30, 2025):
Using executable compiled from https://github.com/sankalp-khare/dust/commit/86861fb60441963398c4e0d52b4f075d622c50cd
I'm convinced that the
walk_data.errors.lock().unwrap()call is getting stuck:walk_data.errors.lock().unwrap()works fine the very first time, thereafter it seems like gets stuck.I'm not at all familiar with the concurrency situation in the execution of this logic, but I think what's happening above, in sequence, is:
(there are two occurrences of
walk_data.errors.lock().unwrap()inhandle_error_and_retry(), I will call themunwrap1https://github.com/bootandy/dust/blob/702f0f0fe9d00641bf392a9d3a7abb6501ca7464/src/dir_walker.rs#L291 andunwrap2https://github.com/bootandy/dust/blob/702f0f0fe9d00641bf392a9d3a7abb6501ca7464/src/dir_walker.rs#L307 in order of their occurrence in the function)(I use the word "thread" because I don't know what abstraction exists / what term is used in Rust for this)
handle_error_and_retry()and goes pastunwrap1(the first, and only successfulwalk_data.errors.lock().unwrap())unwrap2, in the Interrupted error match casehandle_error_and_retry()and gets stuck atunwrap1Hope this helps figure out the issue.
@bootandy commented on GitHub (Jun 3, 2025):
coo, this is interesting, I'll look into this soon.
thanks for the investigation so far
@bootandy commented on GitHub (Jun 3, 2025):
Yeah that looks a bit sus
@bootandy commented on GitHub (Jun 3, 2025):
Yes thanks @sankalp-khare - I think your summary is correct
I think the fix is simple - we just remove the second unwrap2 - I think this is simply my own dumb refactoring error:
https://github.com/bootandy/dust/pull/497
@sankalp-khare commented on GitHub (Jun 3, 2025):
@bootandy I see there's a bunch of checks run against Pull Request branches. Wonder if we could setup a test case or something which would detect this kind of issue.
Although even for the dir tree in my filesystem where the locking occurred, it happened non-deterministically... Sometimes a
dustrun for the same dir tree would complete, other times it'd encounter https://github.com/bootandy/dust/issues/495. So I'm not very sure what the test case / job would look like.@bootandy commented on GitHub (Jun 4, 2025):
released a new version, hopefully this fixes it, let me know if it doesn't
@sankalp-khare commented on GitHub (Jun 5, 2025):
With
v1.2.1I now intermittently face the below situation:Other times it finishes execution and shows results (for the same dir).
Managed to get a backtrace after many retries:
@sankalp-khare commented on GitHub (Jun 5, 2025):
@waynexia commented on GitHub (Jul 2, 2025):
Run into the same case. Can we ignore the interrupted dirs and continue, instead of retrying and panicking?
@bootandy commented on GitHub (Jul 2, 2025):
https://github.com/bootandy/dust/pull/507
Yes,
Although I'm a bit surprised this is happening, I don't see why we can't just keep retrying
@bootandy commented on GitHub (Jul 2, 2025):
Do you think we should fail after a large number of failures say 100 or 100 instead of Never ? I'm worried I might bring back the 'dust does not terminate' issue.
@bootandy commented on GitHub (Jul 7, 2025):
hopefully fixed in 1.2.2