While interning, I once had an application that logged every request and response to the Linux file system. That app easily fills up our disks due to traffic. I got confused when I tried to delete log files because disk usage didn’t decrease even after the log files were deleted. So I restarted the app and finally, the disk usage decreased.
But it is ineffective because we need to kill the application which is making a downtime at that time. Recently, I learned that it was caused by something called a file descriptor
.
TL;DR
|
|
Let’s reproduce that incident.
Creating a big file
We need to create a big file, to fill our disk up. I think 3GB is enough. Well, you don’t need to make it big just to reproduce, but more than 1GB could show the difference easily in the df -h
.
|
|
Yep, now you have a 3GB file contains lorem ipsum called big-file.txt
Create a simple app
Then we need to create an app that simply read big-file.txt
. But I want to show you even if the big-file.txt
is deleted, our app still writing to it. So I make the app read and append the big-file.txt
using Golang, you can do it with whatever you prefer. Let’s call it write-to-file.go
|
|
It will wait for our input and loop forever until we terminate the app.
Check current disk usage
Well, because I put it in the home dir
, I check the size of the /home
using df
.
|
|
Tail the appended text in the file
tail -f
will follow the last 10 lines in the file and it would show this
|
|
Run the app
Let’s run our write-to-file.go
|
|
For every Enter
you input, it will append the big-file.txt
like this
|
|
|
|
Remove the file
Now let’s stop the tail -f
and remove that big-file.txt
.
|
|
Does your disk usage decreased? Unfortunately, mine is not. It might be karma working there isn’t it?
|
|
Now, how to really remove
the big-file.txt
?
For doing that we need to check the current running process that still open that big-file.txt
.
|
|
Now we got the app’s name, its PID, and we know that the Linux know the big-file.txt
is already deleted
.
Let’s open the file descriptor
of the app at /proc/<PID>/fd
.
|
|
It seems like, we have a symlink called 3
that linked to our big-file.txt
.
Now if you tail -f
to the symlink and input some Enter
to the app, it will show something like this
|
|
Interesting isn’t it? Even if the big-file.txt
is deleted, the app still writing into the symlink. That symlink held our disk space, so we need to make that symlink is empty. By simply echoing nothing to the symlink in this case echoing nothing to 3
.
|
|
Now check the disk usage it will return to normal like we really delete the big-file.txt
and the program is still running and still appending to the file.
|
|
Let’s input some Enter
to the app again then the symlink will filled up again by the app.
|
|
Thank you for reading!