bootandy/archived-dust
1
0
Fork 0
mirror of https://github.com/bootandy/dust.git synced 2026-06-08 11:29:05 +03:00
Code Issues 21 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #503] The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized } #221

New Issue
Closed
opened 2026-06-08 11:26:12 +03:00 by zhus · 22 comments
zhus commented 2026-06-08 11:26:12 +03:00
Owner
Copy Link
Copy Source

Originally created by @Joshix-1 on GitHub (Jun 14, 2025).
Original GitHub issue: https://github.com/bootandy/dust/issues/503

$ git show | head -n 1
commit 74ffd7890194bf850b75dc4ff0fd72ce358c8a00
$  rustc --version --verbose
rustc 1.87.0 (17067e9ac 2025-05-09)
binary: rustc
commit-hash: 17067e9ac6d7ecb70e50f92c1944e545188d2359
commit-date: 2025-05-09
host: x86_64-unknown-linux-gnu
release: 1.87.0
LLVM version: 20.1.1
$ RUST_BACKTRACE=1 cargo run
   Compiling du-dust v1.2.1 (/home/josh/code/dust)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.07s
     Running `target/debug/dust`

thread 'main' panicked at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:168:10:
The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5
   1: core::panicking::panic_fmt
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:75:14
   2: core::result::unwrap_failed
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/result.rs:1704:5
   3: core::result::Result<T,E>::expect
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1061:23
   4: rayon_core::registry::global_registry
             at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:166:5
   5: rayon_core::registry::Registry::current_num_threads
             at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:334:17
   6: rayon_core::current_num_threads
             at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/lib.rs:144:5
   7: <rayon::iter::par_bridge::IterBridge<Iter> as rayon::iter::ParallelIterator>::drive_unindexed
             at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/par_bridge.rs:88:27
   8: <rayon::iter::filter_map::FilterMap<I,P> as rayon::iter::ParallelIterator>::drive_unindexed
             at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/filter_map.rs:46:9
   9: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend
             at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/extend.rs:588:28
  10: rayon::iter::from_par_iter::collect_extended
             at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/from_par_iter.rs:20:5
  11: rayon::iter::from_par_iter::<impl rayon::iter::FromParallelIterator<T> for alloc::vec::Vec<T>>::from_par_iter
             at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/from_par_iter.rs:33:9
  12: rayon::iter::ParallelIterator::collect
             at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/mod.rs:2069:9
  13: dust::dir_walker::walk
             at ./src/dir_walker.rs:213:17
  14: dust::dir_walker::walk_it::{{closure}}
             at ./src/dir_walker.rs:60:24
  15: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:294:13
  16: core::iter::traits::iterator::Iterator::find_map::check::{{closure}}
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2873:32
  17: core::iter::traits::iterator::Iterator::try_fold
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2384:21
  18: core::iter::traits::iterator::Iterator::find_map
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2879:9
  19: <core::iter::adapters::filter_map::FilterMap<I,F> as core::iter::traits::iterator::Iterator>::next
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/filter_map.rs:64:9
  20: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter_nested.rs:25:32
  21: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter.rs:34:9
  22: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/mod.rs:3438:9
  23: core::iter::traits::iterator::Iterator::collect
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:1985:9
  24: dust::dir_walker::walk_it
             at ./src/dir_walker.rs:55:35
  25: dust::main
             at ./src/main.rs:280:27
  26: core::ops::function::FnOnce::call_once
             at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Fix:

diff --git a/src/main.rs b/src/main.rs
index 87cb3dd..0b9749b 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -387,15 +387,7 @@ fn print_any_errors(print_errors: bool, errors: Arc<Mutex<RuntimeErrors>>) {
     }
 }
 
-fn init_rayon(stack_size: &Option<usize>, threads: &Option<usize>) {
-    // Rayon seems to raise this error on 32-bit builds
-    // The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }
-    if cfg!(target_pointer_width = "64") {
-        let result = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads));
-        if result.is_err() {
-            eprintln!("Problem initializing rayon, try: export RAYON_NUM_THREADS=1")
-        }
-    }
+fn init_rayon(_stack_size: &Option<usize>, _threads: &Option<usize>) {
 }
 
 fn build_thread_pool(

Also doesn't happen if I set RAYON_NUM_THREADS=1 or to another up to 19

Originally created by @Joshix-1 on GitHub (Jun 14, 2025). Original GitHub issue: https://github.com/bootandy/dust/issues/503 ``` $ git show | head -n 1 commit 74ffd7890194bf850b75dc4ff0fd72ce358c8a00 $ rustc --version --verbose rustc 1.87.0 (17067e9ac 2025-05-09) binary: rustc commit-hash: 17067e9ac6d7ecb70e50f92c1944e545188d2359 commit-date: 2025-05-09 host: x86_64-unknown-linux-gnu release: 1.87.0 LLVM version: 20.1.1 $ RUST_BACKTRACE=1 cargo run Compiling du-dust v1.2.1 (/home/josh/code/dust) Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.07s Running `target/debug/dust` thread 'main' panicked at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:168:10: The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized } stack backtrace: 0: __rustc::rust_begin_unwind at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5 1: core::panicking::panic_fmt at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:75:14 2: core::result::unwrap_failed at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/result.rs:1704:5 3: core::result::Result<T,E>::expect at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1061:23 4: rayon_core::registry::global_registry at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:166:5 5: rayon_core::registry::Registry::current_num_threads at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:334:17 6: rayon_core::current_num_threads at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/lib.rs:144:5 7: <rayon::iter::par_bridge::IterBridge<Iter> as rayon::iter::ParallelIterator>::drive_unindexed at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/par_bridge.rs:88:27 8: <rayon::iter::filter_map::FilterMap<I,P> as rayon::iter::ParallelIterator>::drive_unindexed at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/filter_map.rs:46:9 9: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/extend.rs:588:28 10: rayon::iter::from_par_iter::collect_extended at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/from_par_iter.rs:20:5 11: rayon::iter::from_par_iter::<impl rayon::iter::FromParallelIterator<T> for alloc::vec::Vec<T>>::from_par_iter at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/from_par_iter.rs:33:9 12: rayon::iter::ParallelIterator::collect at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-1.10.0/src/iter/mod.rs:2069:9 13: dust::dir_walker::walk at ./src/dir_walker.rs:213:17 14: dust::dir_walker::walk_it::{{closure}} at ./src/dir_walker.rs:60:24 15: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:294:13 16: core::iter::traits::iterator::Iterator::find_map::check::{{closure}} at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2873:32 17: core::iter::traits::iterator::Iterator::try_fold at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2384:21 18: core::iter::traits::iterator::Iterator::find_map at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2879:9 19: <core::iter::adapters::filter_map::FilterMap<I,F> as core::iter::traits::iterator::Iterator>::next at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/filter_map.rs:64:9 20: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter_nested.rs:25:32 21: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter.rs:34:9 22: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/mod.rs:3438:9 23: core::iter::traits::iterator::Iterator::collect at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:1985:9 24: dust::dir_walker::walk_it at ./src/dir_walker.rs:55:35 25: dust::main at ./src/main.rs:280:27 26: core::ops::function::FnOnce::call_once at /home/josh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. ``` Fix: ```diff diff --git a/src/main.rs b/src/main.rs index 87cb3dd..0b9749b 100644 --- a/src/main.rs +++ b/src/main.rs @@ -387,15 +387,7 @@ fn print_any_errors(print_errors: bool, errors: Arc<Mutex<RuntimeErrors>>) { } } -fn init_rayon(stack_size: &Option<usize>, threads: &Option<usize>) { - // Rayon seems to raise this error on 32-bit builds - // The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized } - if cfg!(target_pointer_width = "64") { - let result = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads)); - if result.is_err() { - eprintln!("Problem initializing rayon, try: export RAYON_NUM_THREADS=1") - } - } +fn init_rayon(_stack_size: &Option<usize>, _threads: &Option<usize>) { } fn build_thread_pool( ``` Also doesn't happen if I set `RAYON_NUM_THREADS=1` or to another up to 19
zhus closed this issue 2026-06-08 11:26:12 +03:00
zhus commented 2026-06-08 11:26:12 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 16, 2025):

Nope.

I can not replicate this on my machine.
The problem here is that the thread pool has already been initialized by something else in your machine.

Also, you've removed the call to build_thread_pool which assigns the stack size correctly on low memory machines.

This part of the code is weird and tricky. The above code is an unpleasant compromise that (used) to stop the error you printed occurring.

I can't figure out why you are seeing it.

Does your code enter the if: if cfg!(target_pointer_width = "64") { ? - if so the thread pool should be built there - if not removing this 'if' will not do anything.

<!-- gh-comment-id:2974711450 --> @bootandy commented on GitHub (Jun 16, 2025): Nope. I can not replicate this on my machine. The problem here is that the thread pool has already been initialized by something else in your machine. Also, you've removed the call to `build_thread_pool` which assigns the stack size correctly on low memory machines. This part of the code is weird and tricky. The above code is an unpleasant compromise that (used) to stop the error you printed occurring. I can't figure out why you are seeing it. Does your code enter the if: `if cfg!(target_pointer_width = "64") {` ? - if so the thread pool should be built there - if not removing this 'if' will not do anything.
zhus commented 2026-06-08 11:26:12 +03:00
Author
Owner
Copy Link
Copy Source

@Joshix-1 commented on GitHub (Jun 16, 2025):

I can't figure out why you are seeing it.

Same. Doesn't really make sense.

Does your code enter the if: if cfg!(target_pointer_width = "64") { ?

Yes, otherwise my patch wouldn't do anything.

The above code is an unpleasant compromise that (used) to stop the error you printed occurring.

Seems like the problem wasn't a pointer width of 32. Other stuff can cause the same issue.

Also, you've removed the call to build_thread_pool which assigns the stack size correctly on low memory machines.

Does that really improve stuff? Are the defaults rayon provides that bad?

I see two possible solutions:

  1. Detect if this is going to panic and then not execute it. (Don't know how)
  2. Add a command line option to disable this.
<!-- gh-comment-id:2975332042 --> @Joshix-1 commented on GitHub (Jun 16, 2025): > I can't figure out why you are seeing it. Same. Doesn't really make sense. > Does your code enter the if: if cfg!(target_pointer_width = "64") { ? Yes, otherwise my patch wouldn't do anything. > The above code is an unpleasant compromise that (used) to stop the error you printed occurring. Seems like the problem wasn't a pointer width of 32. Other stuff can cause the same issue. > Also, you've removed the call to `build_thread_pool` which assigns the stack size correctly on low memory machines. Does that really improve stuff? Are the defaults rayon provides that bad? I see two possible solutions: 1. Detect if this is going to panic and then not execute it. (Don't know how) 2. Add a command line option to disable this.
zhus commented 2026-06-08 11:26:12 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 20, 2025):

Does that really improve stuff? Are the defaults rayon provides that bad?

Yes it does, we had to add this to stop exactly the same error you are seeing now:
https://github.com/bootandy/dust/issues?q=is%3Aissue%20state%3Aclosed%20ThreadPoolBuildError

Some of the previous PRs were doing (1) in your suggestion list.

I really don't understand why you see:

The global thread pool has not been initialized

That's exactly what the code you removed in your suggestion is doing - it's initalizing the global thread pool and catching the error if it causes a problem.

The error you are seeing is from the dir_walker.rs file (The first place Rayon is used to walk the filesystem). - Yet you removed code for initalizing rayon. This doesn't make any sense to me.

<!-- gh-comment-id:2989302837 --> @bootandy commented on GitHub (Jun 20, 2025): >Does that really improve stuff? Are the defaults rayon provides that bad? Yes it does, we had to add this to stop exactly the same error you are seeing now: https://github.com/bootandy/dust/issues?q=is%3Aissue%20state%3Aclosed%20ThreadPoolBuildError Some of the previous PRs were doing (1) in your suggestion list. I really don't understand why you see: >The global thread pool has not been initialized That's exactly what the code you removed in your suggestion is doing - it's initalizing the global thread pool and catching the error if it causes a problem. The error you are seeing is from the dir_walker.rs file (The first place Rayon is used to walk the filesystem). - Yet you removed code for initalizing rayon. This doesn't make any sense to me.
zhus commented 2026-06-08 11:26:12 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 20, 2025):

Seems like the problem wasn't a pointer width of 32. Other stuff can cause the same issue.

Yeah, I think you are right here, maybe we should remove the 'if'.

<!-- gh-comment-id:2989304207 --> @bootandy commented on GitHub (Jun 20, 2025): >Seems like the problem wasn't a pointer width of 32. Other stuff can cause the same issue. Yeah, I think you are right here, maybe we should remove the 'if'.
zhus commented 2026-06-08 11:26:12 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 20, 2025):

does dust -T 1 do anything ? - This should run dust with 1 thread.

<!-- gh-comment-id:2989326397 --> @bootandy commented on GitHub (Jun 20, 2025): does `dust -T 1 ` do anything ? - This should run dust with 1 thread.
zhus commented 2026-06-08 11:26:12 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 20, 2025):

What happens on your box if you use this instead:


fn init_rayon(stack_size: &Option<usize>, threads: &Option<usize>) {
    // Rayon seems to raise this error on 32-bit builds
    // The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }
    if cfg!(target_pointer_width = "64") {
        let result = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads));
        println!("{:?}", result);
        if result.is_err() {
            eprintln!("Problem initializing rayon, try: export RAYON_NUM_THREADS=1")
        }
        let result2 = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads));
        println!("{:?}", result2);
    }
}

Mine does:

 (lib)andy:(0):~/dev/rust/dust$ time target/debug/dust  ~
Ok(Ok(()))
Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }))
....

Which ^ makes sense to me, one call to init the thread pool, and the second failed but the panic is caught by the logic.

<!-- gh-comment-id:2989333928 --> @bootandy commented on GitHub (Jun 20, 2025): What happens on your box if you use this instead: ``` fn init_rayon(stack_size: &Option<usize>, threads: &Option<usize>) { // Rayon seems to raise this error on 32-bit builds // The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized } if cfg!(target_pointer_width = "64") { let result = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads)); println!("{:?}", result); if result.is_err() { eprintln!("Problem initializing rayon, try: export RAYON_NUM_THREADS=1") } let result2 = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads)); println!("{:?}", result2); } } ``` Mine does: ``` (lib)andy:(0):~/dev/rust/dust$ time target/debug/dust ~ Ok(Ok(())) Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized })) .... ``` Which ^ makes sense to me, one call to init the thread pool, and the second failed but the panic is caught by the logic.
zhus commented 2026-06-08 11:26:12 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 20, 2025):

So here's my thoughts:

I need to call build_thread_pool - but its wrapped in a panic unwrap incase it goes wrong - (which in my eyes should make it completely safe).

If you can tweak build_thread_pool or add another call to make your build work then I'll accept a PR.

(And maybe we should loose if cfg!(target_pointer_width = "64") { in init_rayon)

<!-- gh-comment-id:2989340164 --> @bootandy commented on GitHub (Jun 20, 2025): So here's my thoughts: I need to call `build_thread_pool` - but its wrapped in a panic unwrap incase it goes wrong - (which in my eyes should make it completely safe). If you can tweak `build_thread_pool` or add another call to make your build work then I'll accept a PR. (And maybe we should loose `if cfg!(target_pointer_width = "64") {` in init_rayon)
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@Joshix-1 commented on GitHub (Jun 20, 2025):

does dust -T 1 do anything ?

Works with numbers from 1 to 19 (inclusive)

What happens on your box if you use this instead:

$  cargo run
   Compiling du-dust v1.2.1 (/home/josh/code/dust)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.98s
     Running `target/debug/dust`
Ok(Err(ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }))
Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }))

thread 'main' panicked at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:168:10:
The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

cargo run -- -T 19
   Compiling du-dust v1.2.1 (/home/josh/code/dust)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.96s
     Running `target/debug/dust -T 19`
Ok(Ok(()))
Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }))

I looked a bit into it and it seems like rayon fails to create the 20th thread.

 $ cargo run -- -S 1048576 -T 100
   Compiling du-dust v1.2.1 (/home/josh/code/dust)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.95s
     Running `target/debug/dust -S 1048576 -T 100`
Ok(Ok(()))
Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }))

So it seems like it can't spawn the 20th thread because of some sort of stack limitations

<!-- gh-comment-id:2990319231 --> @Joshix-1 commented on GitHub (Jun 20, 2025): > does dust -T 1 do anything ? Works with numbers from 1 to 19 (inclusive) > What happens on your box if you use this instead: ``` $ cargo run Compiling du-dust v1.2.1 (/home/josh/code/dust) Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.98s Running `target/debug/dust` Ok(Err(ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) })) Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized })) thread 'main' panicked at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:168:10: The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ``` ``` cargo run -- -T 19 Compiling du-dust v1.2.1 (/home/josh/code/dust) Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.96s Running `target/debug/dust -T 19` Ok(Ok(())) Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized })) ``` I looked a bit into it and it seems like rayon fails to create the 20th thread. ``` $ cargo run -- -S 1048576 -T 100 Compiling du-dust v1.2.1 (/home/josh/code/dust) Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.95s Running `target/debug/dust -S 1048576 -T 100` Ok(Ok(())) Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized })) ``` So it seems like it can't spawn the 20th thread because of some sort of stack limitations
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 23, 2025):

Ok(Err(ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }))

This bit is interesting, its different to mine.

<!-- gh-comment-id:2997258681 --> @bootandy commented on GitHub (Jun 23, 2025): >Ok(Err(ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) })) This bit is interesting, its different to mine.
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 23, 2025):

So it seems like it can't spawn the 20th thread because of some sort of stack limitations

I think its because if you set the stack_size you call pool.stack_size(stack_size_param); which somehow initializers the thread library correctly

<!-- gh-comment-id:2997265427 --> @bootandy commented on GitHub (Jun 23, 2025): >So it seems like it can't spawn the 20th thread because of some sort of stack limitations I think its because if you set the stack_size you call ` pool.stack_size(stack_size_param);` which somehow initializers the thread library correctly
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 23, 2025):

cargo run -- -S 1048576

I imagine this would work for you too.

<!-- gh-comment-id:2997267043 --> @bootandy commented on GitHub (Jun 23, 2025): ` cargo run -- -S 1048576 ` I imagine this would work for you too.
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 23, 2025):

If you change init_rayon to catch your IOError and then set the default stack size and re-try and create the thread pool, does that work ?

let result = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads));
if result.is_err() {
  result = panic::catch_unwind(|| build_thread_pool(stack_size=1024 ** 3, *threads));
  if result.is_err() {
          eprintln!("Problem initializing rayon, try: export RAYON_NUM_THREADS=1")
   }
}
<!-- gh-comment-id:2997302053 --> @bootandy commented on GitHub (Jun 23, 2025): If you change init_rayon to catch your IOError and then set the default stack size and re-try and create the thread pool, does that work ? ``` let result = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads)); if result.is_err() { result = panic::catch_unwind(|| build_thread_pool(stack_size=1024 ** 3, *threads)); if result.is_err() { eprintln!("Problem initializing rayon, try: export RAYON_NUM_THREADS=1") } } ```
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@Joshix-1 commented on GitHub (Jun 23, 2025):

cargo run -- -S 1048576

Yep, that works too. But with the default for high memory (1024 ** 3) https://github.com/bootandy/dust/blob/master/src/main.rs#L414 it doesn't work:

$ cargo run -- -S 1073741824
...
Ok(Err(ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }))
Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }))

thread 'main' panicked at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:168:10:
The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

If you change init_rayon to catch your IOError and then set the default stack size and re-try and create the thread pool, does that work ?

With

fn init_rayon(stack_size: &Option<usize>, threads: &Option<usize>) {
    let mut result = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads));
    result = dbg!(result);
    if result.is_err() {
      result = panic::catch_unwind(|| build_thread_pool(Some(1024usize.pow(3)), *threads));
      result = dbg!(result);
      if result.is_err() {
              eprintln!("Problem initializing rayon, try: export RAYON_NUM_THREADS=1")
       }
    }
}

cargo run outputs:

[src/main.rs:392:14] result = Ok(
    Err(
        ThreadPoolBuildError {
            kind: IOError(
                Os {
                    code: 11,
                    kind: WouldBlock,
                    message: "Resource temporarily unavailable",
                },
            ),
        },
    ),
)

thread 'main' panicked at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:168:10:
The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<!-- gh-comment-id:2997422440 --> @Joshix-1 commented on GitHub (Jun 23, 2025): > `cargo run -- -S 1048576` Yep, that works too. But with the default for high memory (`1024 ** 3`) https://github.com/bootandy/dust/blob/master/src/main.rs#L414 it doesn't work: ``` $ cargo run -- -S 1073741824 ... Ok(Err(ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) })) Ok(Err(ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized })) thread 'main' panicked at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:168:10: The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ``` > If you change init_rayon to catch your IOError and then set the default stack size and re-try and create the thread pool, does that work ? With ```rs fn init_rayon(stack_size: &Option<usize>, threads: &Option<usize>) { let mut result = panic::catch_unwind(|| build_thread_pool(*stack_size, *threads)); result = dbg!(result); if result.is_err() { result = panic::catch_unwind(|| build_thread_pool(Some(1024usize.pow(3)), *threads)); result = dbg!(result); if result.is_err() { eprintln!("Problem initializing rayon, try: export RAYON_NUM_THREADS=1") } } } ``` `cargo run` outputs: ``` [src/main.rs:392:14] result = Ok( Err( ThreadPoolBuildError { kind: IOError( Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable", }, ), }, ), ) thread 'main' panicked at /home/josh/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rayon-core-1.12.1/src/registry.rs:168:10: The global thread pool has not been initialized.: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ```
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jun 23, 2025):

what about if it was pow(2) - as we know pow(3) was too much memory ?

edit: No that was a dumb comment.

<!-- gh-comment-id:2997706957 --> @bootandy commented on GitHub (Jun 23, 2025): >what about if it was `pow(2)` - as we know pow(3) was too much memory ? edit: No that was a dumb comment.
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jul 5, 2025):

So in summary: If we look at the build_thread_pool function. It seems that it 'works' if we set stack_size or number of threads - So it looks like we have to alter the ThreadPool in some way inorder for the code to work.

<!-- gh-comment-id:3038540396 --> @bootandy commented on GitHub (Jul 5, 2025): So in summary: If we look at the `build_thread_pool` function. It seems that it 'works' if we set stack_size or number of threads - So it looks like we have to alter the ThreadPool in some way inorder for the code to work.
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jul 5, 2025):

https://github.com/bootandy/dust/pull/511

Does this fix it ?

<!-- gh-comment-id:3038567416 --> @bootandy commented on GitHub (Jul 5, 2025): https://github.com/bootandy/dust/pull/511 Does this fix it ?
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jul 5, 2025):

Resource temporarily unavailable (os error 11)

I think this error occurs when the system doesn't have enough memory - I have seen this error when setting the stack size to be much too large.

<!-- gh-comment-id:3038812432 --> @bootandy commented on GitHub (Jul 5, 2025): >Resource temporarily unavailable (os error 11) I think this error occurs when the system doesn't have enough memory - I have seen this error when setting the stack size to be much too large.
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jul 5, 2025):

I'm optimistic about this one:
https://github.com/bootandy/dust/pull/512

<!-- gh-comment-id:3038812583 --> @bootandy commented on GitHub (Jul 5, 2025): I'm optimistic about this one: https://github.com/bootandy/dust/pull/512
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@Joshix-1 commented on GitHub (Jul 5, 2025):

With available memory of 44635316224 Bytes (41.569 GiB) I can't get 32 threads with stack size of 1073741824 Bytes (1 GiB) to work.

<!-- gh-comment-id:3038892579 --> @Joshix-1 commented on GitHub (Jul 5, 2025): With available memory of 44635316224 Bytes (41.569 GiB) I can't get 32 threads with stack size of 1073741824 Bytes (1 GiB) to work.
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@Joshix-1 commented on GitHub (Jul 5, 2025):

And ./target/release/dust -T 19 uses only a few MiB of RAM, so it's not a memory issue. There is some other limit.

The max memory size of 20Gb could be the reason for the failure. It breaks starting with -T 20

$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) 20000000
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 253262
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) 20000000
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 253262
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
<!-- gh-comment-id:3038907207 --> @Joshix-1 commented on GitHub (Jul 5, 2025): And `./target/release/dust -T 19` uses only a few MiB of RAM, so it's not a memory issue. There is some other limit. The `max memory size` of 20Gb could be the reason for the failure. It breaks starting with `-T 20` ``` $ ulimit -a real-time non-blocking time (microseconds, -R) unlimited core file size (blocks, -c) unlimited data seg size (kbytes, -d) 20000000 scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 253262 max locked memory (kbytes, -l) 8192 max memory size (kbytes, -m) 20000000 open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 253262 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ```
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jul 5, 2025):

Hmm, that'd interesting.
surprised that you can't get 32 threads running.

https://docs.rs/rayon/latest/rayon/struct.ThreadPoolBuilder.html#method.stack_size

If there are any other functions we could call in ThreadPoolBuilder let me know and we'll try it.

<!-- gh-comment-id:3038954506 --> @bootandy commented on GitHub (Jul 5, 2025): Hmm, that'd interesting. surprised that you can't get 32 threads running. https://docs.rs/rayon/latest/rayon/struct.ThreadPoolBuilder.html#method.stack_size If there are any other functions we could call in ThreadPoolBuilder let me know and we'll try it.
zhus commented 2026-06-08 11:26:13 +03:00
Author
Owner
Copy Link
Copy Source

@bootandy commented on GitHub (Jul 7, 2025):

i hope this is fixed in 1.2.2

<!-- gh-comment-id:3046330265 --> @bootandy commented on GitHub (Jul 7, 2025): i hope this is fixed in 1.2.2
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
good first issue
help wanted
pull-request
Mirrored from GitHub Pull Request
 windows help wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
zhus (Sergey Zhukov)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: bootandy/archived-dust#221
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?