[GH-ISSUE #445] Search with prebuild database #243

New Issue

zhus · 2026-04-08T16:51:26+03:00

zhus commented

2026-04-08 16:51:26 +03:00

Originally created by @freedit-dev on GitHub (Aug 31, 2024).
Original GitHub issue: https://github.com/sigoden/dufs/issues/445

Specific Demand

When working with millions of files, the search function does not work. Just like the difference of find and locate in Linux, find is a recursive search, and locate is a search based on the database. If dufs is used in a large file system, it is recommended to use locate to search.

Implement Suggestion

I found https://github.com/ngirard/lolcate-rs is very useful. And I did some try and it works well.

let search_paths = tokio::task::spawn_blocking(move || {
                let mut paths: Vec<PathBuf> = vec![];
                if let Ok(output) = std::process::Command::new("lolcate").arg(search).output() {
                    if output.status.success() {
                        let output = String::from_utf8_lossy(&output.stdout);
                        for line in output.lines() {
                            paths.push(PathBuf::from(line));
                        }
                    }
                }

                // for dir in access_paths.child_paths(&path_buf) {
                //     let mut it = WalkDir::new(&dir).into_iter();
                //     it.next();
                //     while let Some(Ok(entry)) = it.next() {
                //         if !running.load(atomic::Ordering::SeqCst) {
                //             break;
                //         }
                //         let entry_path = entry.path();
                //         let base_name = get_file_name(entry_path);
                //         let file_type = entry.file_type();
                //         let mut is_dir_type: bool = file_type.is_dir();
                //         if file_type.is_symlink() {
                //             match std::fs::symlink_metadata(entry_path) {
                //                 Ok(meta) => {
                //                     is_dir_type = meta.is_dir();
                //                 }
                //                 Err(_) => {
                //                     continue;
                //                 }
                //             }
                //         }
                //         if is_hidden(&hidden, base_name, is_dir_type) {
                //             if file_type.is_dir() {
                //                 it.skip_current_dir();
                //             }
                //             continue;
                //         }
                //         if !base_name.to_lowercase().contains(&search) {
                //             continue;
                //         }
                //         paths.push(entry_path.to_path_buf());
                //     }
                // }
                paths
            })
            .await?;
            for search_path in search_paths.into_iter() {
                if let Ok(Some(item)) = self.to_pathitem(search_path, path.to_path_buf()).await {
                    paths.push(item);
                }
            }

Originally created by @freedit-dev on GitHub (Aug 31, 2024). Original GitHub issue: https://github.com/sigoden/dufs/issues/445 ## Specific Demand When working with millions of files, the search function does not work. Just like the difference of `find` and `locate` in Linux, `find` is a recursive search, and `locate` is a search based on the database. If dufs is used in a large file system, it is recommended to use `locate` to search. ## Implement Suggestion I found https://github.com/ngirard/lolcate-rs is very useful. And I did some try and it works well. ```rust let search_paths = tokio::task::spawn_blocking(move || { let mut paths: Vec<PathBuf> = vec![]; if let Ok(output) = std::process::Command::new("lolcate").arg(search).output() { if output.status.success() { let output = String::from_utf8_lossy(&output.stdout); for line in output.lines() { paths.push(PathBuf::from(line)); } } } // for dir in access_paths.child_paths(&path_buf) { // let mut it = WalkDir::new(&dir).into_iter(); // it.next(); // while let Some(Ok(entry)) = it.next() { // if !running.load(atomic::Ordering::SeqCst) { // break; // } // let entry_path = entry.path(); // let base_name = get_file_name(entry_path); // let file_type = entry.file_type(); // let mut is_dir_type: bool = file_type.is_dir(); // if file_type.is_symlink() { // match std::fs::symlink_metadata(entry_path) { // Ok(meta) => { // is_dir_type = meta.is_dir(); // } // Err(_) => { // continue; // } // } // } // if is_hidden(&hidden, base_name, is_dir_type) { // if file_type.is_dir() { // it.skip_current_dir(); // } // continue; // } // if !base_name.to_lowercase().contains(&search) { // continue; // } // paths.push(entry_path.to_path_buf()); // } // } paths }) .await?; for search_path in search_paths.into_iter() { if let Ok(Some(item)) = self.to_pathitem(search_path, path.to_path_buf()).await { paths.push(item); } } ```

zhus closed this issue

2026-04-08 16:51:26 +03:00

zhus commented

2026-04-08 16:51:27 +03:00

@sigoden commented on GitHub (Aug 31, 2024):

we will not support this feature.

dufs must remain stateless and not rely on third-party tools; this breaks it.

@sigoden commented on GitHub (Aug 31, 2024): we will not support this feature. dufs must remain stateless and not rely on third-party tools; this breaks it.

zhus referenced this issue

2026-04-08 16:53:19 +03:00

[PR #243] [CLOSED] chore: update ci #533

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sigoden/dufs#243