[GH-ISSUE #392] Potential Memory Leak #205

New Issue

zhus · 2026-04-08T16:51:10+03:00

zhus commented

2026-04-08 16:51:10 +03:00

Originally created by @pauldeng on GitHub (May 31, 2024).
Original GitHub issue: https://github.com/sigoden/dufs/issues/392

Problem

Thank you for the great project.

I see that memory usage growing gradually and never released until I restart the application.
This could be memory leak but need your help to guide me on how to debug.

My application is data collection from thousands of embedded devices.
Embedded devices HTTP Post file to DUFS.

Memory usage increase rate is like 0.5MB per minute.
Last time, it reached 2.1GB of usage out of 4GB total and DUFS CPU usage skyrocket. Restart DUFS fix the problem temporarily.

Configuration

serve-path: '/home/ubuntu/dufs/storage/'
bind:
  - 0.0.0.0
port: A_PORT_NUMBER
auth:
  - USERNAME:PASSWORD@/:rw
allow-upload: true
allow-delete: false
allow-search: true
allow-symlink: false
allow-archive: true
log-format: '$remote_addr "$request" $status $http_user_agent'

Log

Not sure if this is the root cause of the problem:

...
2024-05-31T09:34:20+10:00 ERROR - 44.216.145.121 "PUT /1754769701/28617664.bin" 500 UBLOX-HttpClient V2.0 error reading a body from connection
...

Environment:

Dufs version: 0.41.0
OS info: Ubuntu 20.04
Proxy server: no

Originally created by @pauldeng on GitHub (May 31, 2024). Original GitHub issue: https://github.com/sigoden/dufs/issues/392 **Problem** Thank you for the great project. I see that memory usage growing gradually and never released until I restart the application. This could be memory leak but need your help to guide me on how to debug. My application is data collection from thousands of embedded devices. Embedded devices HTTP Post file to DUFS. Memory usage increase rate is like 0.5MB per minute. Last time, it reached 2.1GB of usage out of 4GB total and DUFS CPU usage skyrocket. Restart DUFS fix the problem temporarily. **Configuration** ``` serve-path: '/home/ubuntu/dufs/storage/' bind: - 0.0.0.0 port: A_PORT_NUMBER auth: - USERNAME:PASSWORD@/:rw allow-upload: true allow-delete: false allow-search: true allow-symlink: false allow-archive: true log-format: '$remote_addr "$request" $status $http_user_agent' ``` **Log** Not sure if this is the root cause of the problem: ``` ... 2024-05-31T09:34:20+10:00 ERROR - 44.216.145.121 "PUT /1754769701/28617664.bin" 500 UBLOX-HttpClient V2.0 error reading a body from connection ... ``` **Environment:** - Dufs version: 0.41.0 - OS info: Ubuntu 20.04 - Proxy server: no

zhus closed this issue

2026-04-08 16:51:10 +03:00

zhus commented

2026-04-08 16:51:10 +03:00

@sigoden commented on GitHub (May 31, 2024):

@pauldeng

The OS can take a while to release memory or may not release it until it needs too. There is nothing specific here and its likely just what your os is setup to do. Its quite common to see memory increases like this. You can try to use valgrind to see if there are actually any leaks.

The following test can prove whether dufs has a memory leak:

Run dufs with docker and limit the memory to 20m

docker run -d --name dufs -p 5000:5000 -m 20m sigoden/dufs

Generate loads

 while true; do ab -n 20000 -c 30 http://localhost:5000/; done

If dufs has a memory leak, the docker container will be killed due to OOM. But from my own actual test, this is not the case.

So dufs does not have a memory leak.

@sigoden commented on GitHub (May 31, 2024): @pauldeng The OS can take a while to release memory or may not release it until it needs too. There is nothing specific here and its likely just what your os is setup to do. Its quite common to see memory increases like this. You can try to use valgrind to see if there are actually any leaks. The following test can prove whether dufs has a memory leak: 1. Run dufs with docker and limit the memory to 20m ``` docker run -d --name dufs -p 5000:5000 -m 20m sigoden/dufs ``` 2. Generate loads ``` while true; do ab -n 20000 -c 30 http://localhost:5000/; done ``` If dufs has a memory leak, the docker container will be killed due to OOM. But from my own actual test, this is not the case. **So dufs does not have a memory leak.**

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sigoden/dufs#205