[GH-ISSUE #369] Dust memory leak #168

New Issue

2026-06-08T11:25:58+03:00

zhus commented

2026-06-08 11:25:58 +03:00

Originally created by @MathMan05 on GitHub (Feb 16, 2024).
Original GitHub issue: https://github.com/bootandy/dust/issues/369

I attempted to use the system version of dust after someone recommended it to me in a discord server I'm in and it had a bit of a memory leak when it tried to run, I'm on version 0.9.0 Arch Linux, btrfs filesystem. I did sudo dust / just to see what that might look like it tried to scan my file system and ended up taking 30Gibs of RAM before I killed it after taking a screenshot of that. I am including a screenshot of the output of dust.

I only have a 256Gib SSD in my laptop so this should not be happening. Feel free to ask aditional questions as needed.

Originally created by @MathMan05 on GitHub (Feb 16, 2024). Original GitHub issue: https://github.com/bootandy/dust/issues/369 I attempted to use the system version of dust after someone recommended it to me in a discord server I'm in and it had a bit of a memory leak when it tried to run, I'm on version 0.9.0 Arch Linux, btrfs filesystem. I did `sudo dust /` just to see what that might look like it tried to scan my file system and ended up taking 30Gibs of RAM before I killed it after taking a screenshot of that. I am including a screenshot of the output of dust. ![image](https://github.com/bootandy/dust/assets/73901602/83829535-5ad3-4345-9b17-e8b743ca946c) I only have a 256Gib SSD in my laptop so this should not be happening. Feel free to ask aditional questions as needed.

zhus closed this issue

2026-06-08 11:25:58 +03:00

zhus commented

2026-06-08 11:25:58 +03:00

@bootandy commented on GitHub (Feb 19, 2024):

It will scan network drives and USB drives if you have anything mounted. This will result in more than your hard disk being scanned.

What does 'du' say when run from root?:
sudo du -h -d 1

Also consider, what does dust print if you let it finish ? What about if you run it on each of the folders that are one up from the root ?

@bootandy commented on GitHub (Feb 19, 2024): It will scan network drives and USB drives if you have anything mounted. This will result in more than your hard disk being scanned. What does 'du' say when run from root?: `sudo du -h -d 1 ` Also consider, what does dust print if you let it finish ? What about if you run it on each of the folders that are one up from the root ?

zhus commented

2026-06-08 11:25:58 +03:00

@MathMan05 commented on GitHub (Feb 20, 2024):

I can't let dust finish, I don't have enough ram lol. du is also stalling but using way less ram, and due to that it can finish.

sudo du -h -d 1 /
90G     /home
267M    /boot
16M     /etc
0       /lost+found
0       /mnt
3.6G    /opt
du: cannot access '/proc/2685/task/2849/fdinfo/86': No such file or directory
du: cannot access '/proc/1200736/task/1200736/fd/4': No such file or directory
du: cannot access '/proc/1200736/task/1200736/fdinfo/4': No such file or directory
du: cannot access '/proc/1200736/fd/3': No such file or directory
du: cannot access '/proc/1200736/fdinfo/3': No such file or directory
0       /proc
507M    /root
du: cannot access '/run/user/1000/doc': Permission denied
du: cannot access '/run/user/1000/gvfs': Permission denied
2.5M    /run
0       /srv
0       /sys
1.5M    /tmp
26G     /usr
23G     /var
0       /dev
4.2T    /.snapshots
0       /btrfs
4.3T    /

I just figured the problem out though, they're trying to index ./snapshots and neither tool is aware of btrfs, counting things multiple times, as you can see with /.snapshots. While this is not as much of a bug as I thought it was, dust is still having a rather large memory leak, causing it to fill my ram.

@MathMan05 commented on GitHub (Feb 20, 2024): I can't let dust finish, I don't have enough ram lol. du is also stalling but using way less ram, and due to that it can finish. ``` sudo du -h -d 1 / 90G /home 267M /boot 16M /etc 0 /lost+found 0 /mnt 3.6G /opt du: cannot access '/proc/2685/task/2849/fdinfo/86': No such file or directory du: cannot access '/proc/1200736/task/1200736/fd/4': No such file or directory du: cannot access '/proc/1200736/task/1200736/fdinfo/4': No such file or directory du: cannot access '/proc/1200736/fd/3': No such file or directory du: cannot access '/proc/1200736/fdinfo/3': No such file or directory 0 /proc 507M /root du: cannot access '/run/user/1000/doc': Permission denied du: cannot access '/run/user/1000/gvfs': Permission denied 2.5M /run 0 /srv 0 /sys 1.5M /tmp 26G /usr 23G /var 0 /dev 4.2T /.snapshots 0 /btrfs 4.3T / ``` I just figured the problem out though, they're trying to index ./snapshots and neither tool is aware of btrfs, counting things multiple times, as you can see with `/.snapshots`. While this is not as much of a bug as I thought it was, dust is still having a rather large memory leak, causing it to fill my ram.

zhus commented

2026-06-08 11:25:58 +03:00

@bootandy commented on GitHub (Feb 21, 2024):

I can't let dust finish, I don't have enough ram

So does dust crash? What's the error ? Is it killed due to an OOM Error? Do you see this: fatal runtime error: stack overflow ?

Remember dust is trying to summarize every file on your harddisk. This will use a lot of RAM and it may fallover if the OS can't allocate enough virtual memory.

In the shortterm you could try ignoring the snapshots directory:

dust -X snapshots

@bootandy commented on GitHub (Feb 21, 2024): >I can't let dust finish, I don't have enough ram So does dust crash? What's the error ? Is it killed due to an OOM Error? Do you see this: `fatal runtime error: stack overflow` ? Remember dust is trying to summarize every file on your harddisk. This will use a lot of RAM and it may fallover if the OS can't allocate enough virtual memory. In the shortterm you could try ignoring the snapshots directory: dust -X snapshots

zhus commented

2026-06-08 11:25:58 +03:00

@bootandy commented on GitHub (Feb 22, 2024):

Does running it single threaded make any difference?
export RAYON_NUM_THREADS=1; dust /

@bootandy commented on GitHub (Feb 22, 2024): Does running it single threaded make any difference? export RAYON_NUM_THREADS=1; dust /

zhus commented

2026-06-08 11:25:58 +03:00

@MathMan05 commented on GitHub (Feb 22, 2024):

Dust used 30Gib of ram, I had to force close it. Sorry for the late reply, was busy during the day. du used at most 100Mibs of RAM, while dust used 30Gib before I had to force close it as it was just about to fill up all my RAM. Though dust itself throws no error. And dust is still seeing multiple times the size on my SSD in files, due to the /.snapshots dir, as that dir is full of simlink like objects, which instead of being true simlinks, make / COW, and they are an instant in time. I don't know if it's intentional to crawl the COW dirs and count those files multiple times. The command you provided did not fix anything, though the amount of snapshots had decreased, so it only took up arround 15-17Gibs of RAM, which still seems like a memory leak to me.

@MathMan05 commented on GitHub (Feb 22, 2024): Dust used 30Gib of ram, I had to force close it. Sorry for the late reply, was busy during the day. du used at most 100Mibs of RAM, while dust used 30Gib before I had to force close it as it was just about to fill up all my RAM. Though dust itself throws no error. And dust is still seeing multiple times the size on my SSD in files, due to the /.snapshots dir, as that dir is full of simlink like objects, which instead of being true simlinks, make / COW, and they are an instant in time. I don't know if it's intentional to crawl the COW dirs and count those files multiple times. The command you provided did not fix anything, though the amount of snapshots had decreased, so it only took up arround 15-17Gibs of RAM, which still seems like a memory leak to me.

zhus commented

2026-06-08 11:25:58 +03:00

@YanWQ-monad commented on GitHub (Feb 22, 2024):

Could you run find /.snapshots | wc -c to check the total length of all the file path? Too many long paths could take up a large amount of memory, since dust will cache all the paths in the memory.

In my system, a directory with 600 MiB total length of file path, will cause dust uses up to 1.8 GiB memory.

@YanWQ-monad commented on GitHub (Feb 22, 2024): Could you run `find /.snapshots | wc -c` to check the total length of all the file path? Too many long paths could take up a large amount of memory, since dust will cache all the paths in the memory. In my system, a directory with 600 MiB total length of file path, will cause dust uses up to 1.8 GiB memory.

zhus commented

2026-06-08 11:25:58 +03:00

@bootandy commented on GitHub (Feb 24, 2024):

which instead of being true simlinks, make / COW

I've heard of some problems with COW files. They are sadly considered different files. It's annoying but as far as i understand the tooling within rust: https://doc.rust-lang.org/std/fs/struct.Metadata.html doesn't let me detect it.

If you want to use dust I think you'll have to ignore snapshots dir:

dust -X snapshots

Also try @YanWQ-monad's idea, (thanks Yan).

like a memory leak to me.

Dust is tracking all your filepaths, By some definitions this is considered a "memory leak" but personally its one I find useful. It might be that using du is better suited to your usecase Alternatively run dust on a subfolder instead of the root.

@bootandy commented on GitHub (Feb 24, 2024): >which instead of being true simlinks, make / COW I've heard of some problems with COW files. They are sadly considered different files. It's annoying but as far as i understand the tooling within rust: https://doc.rust-lang.org/std/fs/struct.Metadata.html doesn't let me detect it. If you want to use dust I think you'll have to ignore snapshots dir: >dust -X snapshots Also try @YanWQ-monad's idea, (thanks Yan). > like a memory leak to me. Dust is tracking all your filepaths, By some definitions this is considered a "memory leak" but personally its one I find useful. It might be that using du is better suited to your usecase Alternatively run dust on a subfolder instead of the root.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bootandy/archived-dust#168