How to truncate a file that is being written to
up vote
1
down vote
favorite
I have a long running process on a remote computer which I started like this:
$ nohup ./process > output &
My problem is that the output file is getting bigger and bigger fast. And of course, the only use this file has to me is monitoring the progress of the process:
$ tail -f ./output
So I would like to truncate the file from time to time to conserve the space (I'm afraid I might run out of space). So I tried this:
$ truncate -s 0 output
But it seems since the file is open and being written to, this command has no effect. As a test, once I removed the file but then there wasn't a new one created. So I had lost my progress report and I had to restart the process.
Is there any way I can truncate the file while it is being written to?
files nohup
asked Nov 25 at 11:19
Mehran
66041122
add a comment |
up vote
1
down vote
favorite
I have a long running process on a remote computer which I started like this:
$ nohup ./process > output &
My problem is that the output file is getting bigger and bigger fast. And of course, the only use this file has to me is monitoring the progress of the process:
$ tail -f ./output
So I would like to truncate the file from time to time to conserve the space (I'm afraid I might run out of space). So I tried this:
$ truncate -s 0 output
But it seems since the file is open and being written to, this command has no effect. As a test, once I removed the file but then there wasn't a new one created. So I had lost my progress report and I had to restart the process.
Is there any way I can truncate the file while it is being written to?
files nohup
asked Nov 25 at 11:19
Mehran
66041122
Is it necessary to write to a file? I mean, maybe it is enough to write to a terminal window directly (without redirection to the fileoutput).
– sudodus
Nov 25 at 11:22
1
I'm writing to file since my SSH connection might drop and I need to be able to log back in and continue monitoring again. The solution depicted above has the benefit spawning a new process which is not a child process of connection. The process might take weeks to complete.
– Mehran
Nov 25 at 11:25
How many output lines do you want to keep?
– sudodus
Nov 25 at 11:29
Maybe you can uselogrotate. See for example this link
– sudodus
Nov 25 at 11:32
1
I only care about the few last lines each time look into the file
– Mehran
Nov 25 at 11:32
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I have a long running process on a remote computer which I started like this:
$ nohup ./process > output &
My problem is that the output file is getting bigger and bigger fast. And of course, the only use this file has to me is monitoring the progress of the process:
$ tail -f ./output
So I would like to truncate the file from time to time to conserve the space (I'm afraid I might run out of space). So I tried this:
$ truncate -s 0 output
But it seems since the file is open and being written to, this command has no effect. As a test, once I removed the file but then there wasn't a new one created. So I had lost my progress report and I had to restart the process.
Is there any way I can truncate the file while it is being written to?
files nohup
asked Nov 25 at 11:19
Mehran
66041122
I have a long running process on a remote computer which I started like this:
$ nohup ./process > output &
My problem is that the output file is getting bigger and bigger fast. And of course, the only use this file has to me is monitoring the progress of the process:
$ tail -f ./output
So I would like to truncate the file from time to time to conserve the space (I'm afraid I might run out of space). So I tried this:
$ truncate -s 0 output
But it seems since the file is open and being written to, this command has no effect. As a test, once I removed the file but then there wasn't a new one created. So I had lost my progress report and I had to restart the process.
Is there any way I can truncate the file while it is being written to?
files nohup
files nohup
asked Nov 25 at 11:19
Mehran
66041122
asked Nov 25 at 11:19
Mehran
66041122
asked Nov 25 at 11:19
Mehran
66041122
asked Nov 25 at 11:19
Mehran
66041122
asked Nov 25 at 11:19
Mehran
66041122
66041122
Is it necessary to write to a file? I mean, maybe it is enough to write to a terminal window directly (without redirection to the fileoutput).
– sudodus
Nov 25 at 11:22
1
I'm writing to file since my SSH connection might drop and I need to be able to log back in and continue monitoring again. The solution depicted above has the benefit spawning a new process which is not a child process of connection. The process might take weeks to complete.
– Mehran
Nov 25 at 11:25
How many output lines do you want to keep?
– sudodus
Nov 25 at 11:29
Maybe you can uselogrotate. See for example this link
– sudodus
Nov 25 at 11:32
1
I only care about the few last lines each time look into the file
– Mehran
Nov 25 at 11:32
add a comment |
Is it necessary to write to a file? I mean, maybe it is enough to write to a terminal window directly (without redirection to the fileoutput).
– sudodus
Nov 25 at 11:22
1
I'm writing to file since my SSH connection might drop and I need to be able to log back in and continue monitoring again. The solution depicted above has the benefit spawning a new process which is not a child process of connection. The process might take weeks to complete.
– Mehran
Nov 25 at 11:25
How many output lines do you want to keep?
– sudodus
Nov 25 at 11:29
Maybe you can uselogrotate. See for example this link
– sudodus
Nov 25 at 11:32
1
I only care about the few last lines each time look into the file
– Mehran
Nov 25 at 11:32
Is it necessary to write to a file? I mean, maybe it is enough to write to a terminal window directly (without redirection to the file
output).– sudodus
Nov 25 at 11:22
Is it necessary to write to a file? I mean, maybe it is enough to write to a terminal window directly (without redirection to the file
output).– sudodus
Nov 25 at 11:22
1
1
I'm writing to file since my SSH connection might drop and I need to be able to log back in and continue monitoring again. The solution depicted above has the benefit spawning a new process which is not a child process of connection. The process might take weeks to complete.
– Mehran
Nov 25 at 11:25
I'm writing to file since my SSH connection might drop and I need to be able to log back in and continue monitoring again. The solution depicted above has the benefit spawning a new process which is not a child process of connection. The process might take weeks to complete.
– Mehran
Nov 25 at 11:25
How many output lines do you want to keep?
– sudodus
Nov 25 at 11:29
How many output lines do you want to keep?
– sudodus
Nov 25 at 11:29
Maybe you can use
logrotate. See for example this link– sudodus
Nov 25 at 11:32
Maybe you can use
logrotate. See for example this link– sudodus
Nov 25 at 11:32
1
1
I only care about the few last lines each time look into the file
– Mehran
Nov 25 at 11:32
I only care about the few last lines each time look into the file
– Mehran
Nov 25 at 11:32
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
While a full size solution may involve logrotate, it might work with a simpler solution according to the following demo example.
I started a process, #1, that writes the current date and time to the file output.
while true;do LC_TIME=en_US date>>output;sleep 10;done &
Then I started a process, #2, that copies output, and uses tail to truncate the redirected copy back to output. This keeps writing from process #1 alive. It might work in your case too.
while true;do cat output > ttt;tail ttt > output;cat output;echo '----------------------------';sleep 60;done &
Example of output from process #2,
Sun Nov 25 14:31:40 CET 2018
----------------------------
Sun Nov 25 14:31:10 CET 2018
Sun Nov 25 14:31:20 CET 2018
Sun Nov 25 14:31:30 CET 2018
Sun Nov 25 14:31:40 CET 2018
Sun Nov 25 14:31:50 CET 2018
Sun Nov 25 14:32:00 CET 2018
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
----------------------------
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
Sun Nov 25 14:32:50 CET 2018
Sun Nov 25 14:33:00 CET 2018
Sun Nov 25 14:33:10 CET 2018
Sun Nov 25 14:33:20 CET 2018
Sun Nov 25 14:33:30 CET 2018
Sun Nov 25 14:33:40 CET 2018
----------------------------
I also tested how to make a demo example that does not hangup, if the window/connection is closed,
nohup bash -c 'while true;do LC_TIME=en_US date>>output;sleep 10;done' &
and
nohup bash -c 'while true;do cat output > ttt;tail ttt > output;sleep 60;done' &
which can be monitored with LANG=C tail -f ./output from another window/connection,
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
tail: ./output: file truncated
Sun Nov 25 15:00:35 CET 2018
Sun Nov 25 15:00:45 CET 2018
Sun Nov 25 15:00:55 CET 2018
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
Sun Nov 25 15:02:15 CET 2018
Sun Nov 25 15:02:25 CET 2018
Sun Nov 25 15:02:35 CET 2018
Sun Nov 25 15:02:45 CET 2018
answered Nov 25 at 14:08
sudodus
22.1k32871
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
While a full size solution may involve logrotate, it might work with a simpler solution according to the following demo example.
I started a process, #1, that writes the current date and time to the file output.
while true;do LC_TIME=en_US date>>output;sleep 10;done &
Then I started a process, #2, that copies output, and uses tail to truncate the redirected copy back to output. This keeps writing from process #1 alive. It might work in your case too.
while true;do cat output > ttt;tail ttt > output;cat output;echo '----------------------------';sleep 60;done &
Example of output from process #2,
Sun Nov 25 14:31:40 CET 2018
----------------------------
Sun Nov 25 14:31:10 CET 2018
Sun Nov 25 14:31:20 CET 2018
Sun Nov 25 14:31:30 CET 2018
Sun Nov 25 14:31:40 CET 2018
Sun Nov 25 14:31:50 CET 2018
Sun Nov 25 14:32:00 CET 2018
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
----------------------------
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
Sun Nov 25 14:32:50 CET 2018
Sun Nov 25 14:33:00 CET 2018
Sun Nov 25 14:33:10 CET 2018
Sun Nov 25 14:33:20 CET 2018
Sun Nov 25 14:33:30 CET 2018
Sun Nov 25 14:33:40 CET 2018
----------------------------
I also tested how to make a demo example that does not hangup, if the window/connection is closed,
nohup bash -c 'while true;do LC_TIME=en_US date>>output;sleep 10;done' &
and
nohup bash -c 'while true;do cat output > ttt;tail ttt > output;sleep 60;done' &
which can be monitored with LANG=C tail -f ./output from another window/connection,
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
tail: ./output: file truncated
Sun Nov 25 15:00:35 CET 2018
Sun Nov 25 15:00:45 CET 2018
Sun Nov 25 15:00:55 CET 2018
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
Sun Nov 25 15:02:15 CET 2018
Sun Nov 25 15:02:25 CET 2018
Sun Nov 25 15:02:35 CET 2018
Sun Nov 25 15:02:45 CET 2018
answered Nov 25 at 14:08
sudodus
22.1k32871
add a comment |
up vote
1
down vote
While a full size solution may involve logrotate, it might work with a simpler solution according to the following demo example.
I started a process, #1, that writes the current date and time to the file output.
while true;do LC_TIME=en_US date>>output;sleep 10;done &
Then I started a process, #2, that copies output, and uses tail to truncate the redirected copy back to output. This keeps writing from process #1 alive. It might work in your case too.
while true;do cat output > ttt;tail ttt > output;cat output;echo '----------------------------';sleep 60;done &
Example of output from process #2,
Sun Nov 25 14:31:40 CET 2018
----------------------------
Sun Nov 25 14:31:10 CET 2018
Sun Nov 25 14:31:20 CET 2018
Sun Nov 25 14:31:30 CET 2018
Sun Nov 25 14:31:40 CET 2018
Sun Nov 25 14:31:50 CET 2018
Sun Nov 25 14:32:00 CET 2018
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
----------------------------
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
Sun Nov 25 14:32:50 CET 2018
Sun Nov 25 14:33:00 CET 2018
Sun Nov 25 14:33:10 CET 2018
Sun Nov 25 14:33:20 CET 2018
Sun Nov 25 14:33:30 CET 2018
Sun Nov 25 14:33:40 CET 2018
----------------------------
I also tested how to make a demo example that does not hangup, if the window/connection is closed,
nohup bash -c 'while true;do LC_TIME=en_US date>>output;sleep 10;done' &
and
nohup bash -c 'while true;do cat output > ttt;tail ttt > output;sleep 60;done' &
which can be monitored with LANG=C tail -f ./output from another window/connection,
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
tail: ./output: file truncated
Sun Nov 25 15:00:35 CET 2018
Sun Nov 25 15:00:45 CET 2018
Sun Nov 25 15:00:55 CET 2018
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
Sun Nov 25 15:02:15 CET 2018
Sun Nov 25 15:02:25 CET 2018
Sun Nov 25 15:02:35 CET 2018
Sun Nov 25 15:02:45 CET 2018
answered Nov 25 at 14:08
sudodus
22.1k32871
add a comment |
up vote
1
down vote
up vote
1
down vote
While a full size solution may involve logrotate, it might work with a simpler solution according to the following demo example.
I started a process, #1, that writes the current date and time to the file output.
while true;do LC_TIME=en_US date>>output;sleep 10;done &
Then I started a process, #2, that copies output, and uses tail to truncate the redirected copy back to output. This keeps writing from process #1 alive. It might work in your case too.
while true;do cat output > ttt;tail ttt > output;cat output;echo '----------------------------';sleep 60;done &
Example of output from process #2,
Sun Nov 25 14:31:40 CET 2018
----------------------------
Sun Nov 25 14:31:10 CET 2018
Sun Nov 25 14:31:20 CET 2018
Sun Nov 25 14:31:30 CET 2018
Sun Nov 25 14:31:40 CET 2018
Sun Nov 25 14:31:50 CET 2018
Sun Nov 25 14:32:00 CET 2018
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
----------------------------
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
Sun Nov 25 14:32:50 CET 2018
Sun Nov 25 14:33:00 CET 2018
Sun Nov 25 14:33:10 CET 2018
Sun Nov 25 14:33:20 CET 2018
Sun Nov 25 14:33:30 CET 2018
Sun Nov 25 14:33:40 CET 2018
----------------------------
I also tested how to make a demo example that does not hangup, if the window/connection is closed,
nohup bash -c 'while true;do LC_TIME=en_US date>>output;sleep 10;done' &
and
nohup bash -c 'while true;do cat output > ttt;tail ttt > output;sleep 60;done' &
which can be monitored with LANG=C tail -f ./output from another window/connection,
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
tail: ./output: file truncated
Sun Nov 25 15:00:35 CET 2018
Sun Nov 25 15:00:45 CET 2018
Sun Nov 25 15:00:55 CET 2018
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
Sun Nov 25 15:02:15 CET 2018
Sun Nov 25 15:02:25 CET 2018
Sun Nov 25 15:02:35 CET 2018
Sun Nov 25 15:02:45 CET 2018
answered Nov 25 at 14:08
sudodus
22.1k32871
While a full size solution may involve logrotate, it might work with a simpler solution according to the following demo example.
I started a process, #1, that writes the current date and time to the file output.
while true;do LC_TIME=en_US date>>output;sleep 10;done &
Then I started a process, #2, that copies output, and uses tail to truncate the redirected copy back to output. This keeps writing from process #1 alive. It might work in your case too.
while true;do cat output > ttt;tail ttt > output;cat output;echo '----------------------------';sleep 60;done &
Example of output from process #2,
Sun Nov 25 14:31:40 CET 2018
----------------------------
Sun Nov 25 14:31:10 CET 2018
Sun Nov 25 14:31:20 CET 2018
Sun Nov 25 14:31:30 CET 2018
Sun Nov 25 14:31:40 CET 2018
Sun Nov 25 14:31:50 CET 2018
Sun Nov 25 14:32:00 CET 2018
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
----------------------------
Sun Nov 25 14:32:10 CET 2018
Sun Nov 25 14:32:20 CET 2018
Sun Nov 25 14:32:30 CET 2018
Sun Nov 25 14:32:40 CET 2018
Sun Nov 25 14:32:50 CET 2018
Sun Nov 25 14:33:00 CET 2018
Sun Nov 25 14:33:10 CET 2018
Sun Nov 25 14:33:20 CET 2018
Sun Nov 25 14:33:30 CET 2018
Sun Nov 25 14:33:40 CET 2018
----------------------------
I also tested how to make a demo example that does not hangup, if the window/connection is closed,
nohup bash -c 'while true;do LC_TIME=en_US date>>output;sleep 10;done' &
and
nohup bash -c 'while true;do cat output > ttt;tail ttt > output;sleep 60;done' &
which can be monitored with LANG=C tail -f ./output from another window/connection,
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
tail: ./output: file truncated
Sun Nov 25 15:00:35 CET 2018
Sun Nov 25 15:00:45 CET 2018
Sun Nov 25 15:00:55 CET 2018
Sun Nov 25 15:01:05 CET 2018
Sun Nov 25 15:01:15 CET 2018
Sun Nov 25 15:01:25 CET 2018
Sun Nov 25 15:01:35 CET 2018
Sun Nov 25 15:01:45 CET 2018
Sun Nov 25 15:01:55 CET 2018
Sun Nov 25 15:02:05 CET 2018
Sun Nov 25 15:02:15 CET 2018
Sun Nov 25 15:02:25 CET 2018
Sun Nov 25 15:02:35 CET 2018
Sun Nov 25 15:02:45 CET 2018
answered Nov 25 at 14:08
sudodus
22.1k32871
answered Nov 25 at 14:08
sudodus
22.1k32871
answered Nov 25 at 14:08
sudodus
22.1k32871
answered Nov 25 at 14:08
sudodus
22.1k32871
22.1k32871
add a comment |
add a comment |
Thanks for contributing an answer to Ask Ubuntu!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1095889%2fhow-to-truncate-a-file-that-is-being-written-to%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Is it necessary to write to a file? I mean, maybe it is enough to write to a terminal window directly (without redirection to the file
output).– sudodus
Nov 25 at 11:22
1
I'm writing to file since my SSH connection might drop and I need to be able to log back in and continue monitoring again. The solution depicted above has the benefit spawning a new process which is not a child process of connection. The process might take weeks to complete.
– Mehran
Nov 25 at 11:25
How many output lines do you want to keep?
– sudodus
Nov 25 at 11:29
Maybe you can use
logrotate. See for example this link– sudodus
Nov 25 at 11:32
1
I only care about the few last lines each time look into the file
– Mehran
Nov 25 at 11:32