Why is .bash_history periodically wiped?





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}







3















This is the second time it's happened.
I just tried grep-ing some historical commands and came up empty. a look at my bash history shows that it's practically empty. Like it's been wiped clean.
I use iTerm2 and on MacOS 10.13.6.
I see there's a .bash_sessions directory with some sessions saved but I guess this is probably an iTerm2 thing to preserve sessions for some reason or another.










share|improve this question























  • unix.stackexchange.com/questions/163371/…

    – JBis
    Mar 25 at 11:32


















3















This is the second time it's happened.
I just tried grep-ing some historical commands and came up empty. a look at my bash history shows that it's practically empty. Like it's been wiped clean.
I use iTerm2 and on MacOS 10.13.6.
I see there's a .bash_sessions directory with some sessions saved but I guess this is probably an iTerm2 thing to preserve sessions for some reason or another.










share|improve this question























  • unix.stackexchange.com/questions/163371/…

    – JBis
    Mar 25 at 11:32














3












3








3


1






This is the second time it's happened.
I just tried grep-ing some historical commands and came up empty. a look at my bash history shows that it's practically empty. Like it's been wiped clean.
I use iTerm2 and on MacOS 10.13.6.
I see there's a .bash_sessions directory with some sessions saved but I guess this is probably an iTerm2 thing to preserve sessions for some reason or another.










share|improve this question














This is the second time it's happened.
I just tried grep-ing some historical commands and came up empty. a look at my bash history shows that it's practically empty. Like it's been wiped clean.
I use iTerm2 and on MacOS 10.13.6.
I see there's a .bash_sessions directory with some sessions saved but I guess this is probably an iTerm2 thing to preserve sessions for some reason or another.







terminal bash iterm






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Mar 25 at 10:33









Dark Star1Dark Star1

249211




249211













  • unix.stackexchange.com/questions/163371/…

    – JBis
    Mar 25 at 11:32



















  • unix.stackexchange.com/questions/163371/…

    – JBis
    Mar 25 at 11:32

















unix.stackexchange.com/questions/163371/…

– JBis
Mar 25 at 11:32





unix.stackexchange.com/questions/163371/…

– JBis
Mar 25 at 11:32










1 Answer
1






active

oldest

votes


















6














There can be multiple reasons as to why this happens - I'll try to outline how it works below.



However, I can say that the other answer you have received here is not correct. HISTFILESIZE and HISTSIZE will not cause your history to "sometimes" being entirely wiped out or almost wiped out. Only by setting them to 0 you would get nothing in the files - but it would happen every time, and not by chance. In addition, what you describe with the file being "almost wiped out" cannot happen due to it being set to 0.



You're actually on to the right thing yourself by mentioning .bash_sessions. That is not an iTerm2 "thing", but rather how it works by default on a standard macOS install. Apple has built-in per-session history on top of a regular bash install.



This means that if you have multiple terminals running (for example multiple tabs), each of those will have a seperate history tracked in .bash_sessions. If you reboot your Mac and the terminal windows are restored, you'll find that each still have their own history - and only their own history.



When you close down a bash session, Apple's system will merge the history for that specific session into the global .bash_history file. Then when you open a new terminal (and thus bash session), it will start with that merged history containing history from potentially multiple sessions.



This is all handled by the /etc/bashrc_Apple_Terminal script.



Now that you know how it works, here are some possible causes for what you're seeing:




  • You may accidentially have removed the history yourself (history -c)


  • Your Mac could be infected with malware and/or hacked, and someone else is removing your .bash_history file


  • You have software installed that periodically empties/removes .bash_history



These reasons are not that likely, I would say.



Now there's one last possible cause:



There's a known race-condition in handling the bash histories. If you close down multiple bash sessions at once (for example closing down a terminal window with multiple tabs), you might loose part of or the whole history.



This happens if the computer is "slow" while reading/writing parts of the history. Basically how it works is that your currently history is read, the file is deleted/moved away, and a new file is created, where all the history lines are written to. If this process is running multiple times at the same time, you might see that one process removes the old history, the other process read an empty history, then the first process writes out the whole history again, and then the other process deletes that and writes out a very short history containing only few recent lines - as it didn't read anything in. This bug shows up as exactly what you have described.



The fix is simply to close one bash session at a time to let the system have time to write out everything correctly.






share|improve this answer
























  • Thanks for your explanation I will see if I can replicate this given your answer and get back to you. I am aware that it may not necessarily happen on the first try, but I'd like to be sure it is replicable.

    – Dark Star1
    Mar 25 at 14:18











  • Also just want to add to the malware part - a common OS X RAT known as Eggshell has a macOS payload that actually executes history -wc as part of its code after the main payload, such that all traces of the payload injection command are wiped. Not saying its necessary Eggshell but it could be something common if someone wanted to hide their traces...

    – QuickishFM
    Mar 25 at 14:31











  • This has only happened to me twice and I doubt it is that malware but thanks for the info. Ill look out for it just in case

    – Dark Star1
    Mar 25 at 15:16











  • As a workaround, I have this in my bashrc: export PROMPT_COMMAND='echo $(date +%Y-%m-%dT%H:%M:%S) $(pwd): "$(history 1 | head -c 512 | sed "s/^[ ]*[0-9]+[ ]*//" )" >> ~/.bash_history_full; echo -ne "33]0;${PWD##*/}07"' which will store each command after it is executed.

    – jpa
    Mar 25 at 19:18











  • The disadvantage there is thought that it won't seperate sessions for you - so it will be hard to make sense of stuff afterwards if you have many sessions (for example some local and some remote).

    – jksoegaard
    Mar 25 at 19:42



















1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









6














There can be multiple reasons as to why this happens - I'll try to outline how it works below.



However, I can say that the other answer you have received here is not correct. HISTFILESIZE and HISTSIZE will not cause your history to "sometimes" being entirely wiped out or almost wiped out. Only by setting them to 0 you would get nothing in the files - but it would happen every time, and not by chance. In addition, what you describe with the file being "almost wiped out" cannot happen due to it being set to 0.



You're actually on to the right thing yourself by mentioning .bash_sessions. That is not an iTerm2 "thing", but rather how it works by default on a standard macOS install. Apple has built-in per-session history on top of a regular bash install.



This means that if you have multiple terminals running (for example multiple tabs), each of those will have a seperate history tracked in .bash_sessions. If you reboot your Mac and the terminal windows are restored, you'll find that each still have their own history - and only their own history.



When you close down a bash session, Apple's system will merge the history for that specific session into the global .bash_history file. Then when you open a new terminal (and thus bash session), it will start with that merged history containing history from potentially multiple sessions.



This is all handled by the /etc/bashrc_Apple_Terminal script.



Now that you know how it works, here are some possible causes for what you're seeing:




  • You may accidentially have removed the history yourself (history -c)


  • Your Mac could be infected with malware and/or hacked, and someone else is removing your .bash_history file


  • You have software installed that periodically empties/removes .bash_history



These reasons are not that likely, I would say.



Now there's one last possible cause:



There's a known race-condition in handling the bash histories. If you close down multiple bash sessions at once (for example closing down a terminal window with multiple tabs), you might loose part of or the whole history.



This happens if the computer is "slow" while reading/writing parts of the history. Basically how it works is that your currently history is read, the file is deleted/moved away, and a new file is created, where all the history lines are written to. If this process is running multiple times at the same time, you might see that one process removes the old history, the other process read an empty history, then the first process writes out the whole history again, and then the other process deletes that and writes out a very short history containing only few recent lines - as it didn't read anything in. This bug shows up as exactly what you have described.



The fix is simply to close one bash session at a time to let the system have time to write out everything correctly.






share|improve this answer
























  • Thanks for your explanation I will see if I can replicate this given your answer and get back to you. I am aware that it may not necessarily happen on the first try, but I'd like to be sure it is replicable.

    – Dark Star1
    Mar 25 at 14:18











  • Also just want to add to the malware part - a common OS X RAT known as Eggshell has a macOS payload that actually executes history -wc as part of its code after the main payload, such that all traces of the payload injection command are wiped. Not saying its necessary Eggshell but it could be something common if someone wanted to hide their traces...

    – QuickishFM
    Mar 25 at 14:31











  • This has only happened to me twice and I doubt it is that malware but thanks for the info. Ill look out for it just in case

    – Dark Star1
    Mar 25 at 15:16











  • As a workaround, I have this in my bashrc: export PROMPT_COMMAND='echo $(date +%Y-%m-%dT%H:%M:%S) $(pwd): "$(history 1 | head -c 512 | sed "s/^[ ]*[0-9]+[ ]*//" )" >> ~/.bash_history_full; echo -ne "33]0;${PWD##*/}07"' which will store each command after it is executed.

    – jpa
    Mar 25 at 19:18











  • The disadvantage there is thought that it won't seperate sessions for you - so it will be hard to make sense of stuff afterwards if you have many sessions (for example some local and some remote).

    – jksoegaard
    Mar 25 at 19:42
















6














There can be multiple reasons as to why this happens - I'll try to outline how it works below.



However, I can say that the other answer you have received here is not correct. HISTFILESIZE and HISTSIZE will not cause your history to "sometimes" being entirely wiped out or almost wiped out. Only by setting them to 0 you would get nothing in the files - but it would happen every time, and not by chance. In addition, what you describe with the file being "almost wiped out" cannot happen due to it being set to 0.



You're actually on to the right thing yourself by mentioning .bash_sessions. That is not an iTerm2 "thing", but rather how it works by default on a standard macOS install. Apple has built-in per-session history on top of a regular bash install.



This means that if you have multiple terminals running (for example multiple tabs), each of those will have a seperate history tracked in .bash_sessions. If you reboot your Mac and the terminal windows are restored, you'll find that each still have their own history - and only their own history.



When you close down a bash session, Apple's system will merge the history for that specific session into the global .bash_history file. Then when you open a new terminal (and thus bash session), it will start with that merged history containing history from potentially multiple sessions.



This is all handled by the /etc/bashrc_Apple_Terminal script.



Now that you know how it works, here are some possible causes for what you're seeing:




  • You may accidentially have removed the history yourself (history -c)


  • Your Mac could be infected with malware and/or hacked, and someone else is removing your .bash_history file


  • You have software installed that periodically empties/removes .bash_history



These reasons are not that likely, I would say.



Now there's one last possible cause:



There's a known race-condition in handling the bash histories. If you close down multiple bash sessions at once (for example closing down a terminal window with multiple tabs), you might loose part of or the whole history.



This happens if the computer is "slow" while reading/writing parts of the history. Basically how it works is that your currently history is read, the file is deleted/moved away, and a new file is created, where all the history lines are written to. If this process is running multiple times at the same time, you might see that one process removes the old history, the other process read an empty history, then the first process writes out the whole history again, and then the other process deletes that and writes out a very short history containing only few recent lines - as it didn't read anything in. This bug shows up as exactly what you have described.



The fix is simply to close one bash session at a time to let the system have time to write out everything correctly.






share|improve this answer
























  • Thanks for your explanation I will see if I can replicate this given your answer and get back to you. I am aware that it may not necessarily happen on the first try, but I'd like to be sure it is replicable.

    – Dark Star1
    Mar 25 at 14:18











  • Also just want to add to the malware part - a common OS X RAT known as Eggshell has a macOS payload that actually executes history -wc as part of its code after the main payload, such that all traces of the payload injection command are wiped. Not saying its necessary Eggshell but it could be something common if someone wanted to hide their traces...

    – QuickishFM
    Mar 25 at 14:31











  • This has only happened to me twice and I doubt it is that malware but thanks for the info. Ill look out for it just in case

    – Dark Star1
    Mar 25 at 15:16











  • As a workaround, I have this in my bashrc: export PROMPT_COMMAND='echo $(date +%Y-%m-%dT%H:%M:%S) $(pwd): "$(history 1 | head -c 512 | sed "s/^[ ]*[0-9]+[ ]*//" )" >> ~/.bash_history_full; echo -ne "33]0;${PWD##*/}07"' which will store each command after it is executed.

    – jpa
    Mar 25 at 19:18











  • The disadvantage there is thought that it won't seperate sessions for you - so it will be hard to make sense of stuff afterwards if you have many sessions (for example some local and some remote).

    – jksoegaard
    Mar 25 at 19:42














6












6








6







There can be multiple reasons as to why this happens - I'll try to outline how it works below.



However, I can say that the other answer you have received here is not correct. HISTFILESIZE and HISTSIZE will not cause your history to "sometimes" being entirely wiped out or almost wiped out. Only by setting them to 0 you would get nothing in the files - but it would happen every time, and not by chance. In addition, what you describe with the file being "almost wiped out" cannot happen due to it being set to 0.



You're actually on to the right thing yourself by mentioning .bash_sessions. That is not an iTerm2 "thing", but rather how it works by default on a standard macOS install. Apple has built-in per-session history on top of a regular bash install.



This means that if you have multiple terminals running (for example multiple tabs), each of those will have a seperate history tracked in .bash_sessions. If you reboot your Mac and the terminal windows are restored, you'll find that each still have their own history - and only their own history.



When you close down a bash session, Apple's system will merge the history for that specific session into the global .bash_history file. Then when you open a new terminal (and thus bash session), it will start with that merged history containing history from potentially multiple sessions.



This is all handled by the /etc/bashrc_Apple_Terminal script.



Now that you know how it works, here are some possible causes for what you're seeing:




  • You may accidentially have removed the history yourself (history -c)


  • Your Mac could be infected with malware and/or hacked, and someone else is removing your .bash_history file


  • You have software installed that periodically empties/removes .bash_history



These reasons are not that likely, I would say.



Now there's one last possible cause:



There's a known race-condition in handling the bash histories. If you close down multiple bash sessions at once (for example closing down a terminal window with multiple tabs), you might loose part of or the whole history.



This happens if the computer is "slow" while reading/writing parts of the history. Basically how it works is that your currently history is read, the file is deleted/moved away, and a new file is created, where all the history lines are written to. If this process is running multiple times at the same time, you might see that one process removes the old history, the other process read an empty history, then the first process writes out the whole history again, and then the other process deletes that and writes out a very short history containing only few recent lines - as it didn't read anything in. This bug shows up as exactly what you have described.



The fix is simply to close one bash session at a time to let the system have time to write out everything correctly.






share|improve this answer













There can be multiple reasons as to why this happens - I'll try to outline how it works below.



However, I can say that the other answer you have received here is not correct. HISTFILESIZE and HISTSIZE will not cause your history to "sometimes" being entirely wiped out or almost wiped out. Only by setting them to 0 you would get nothing in the files - but it would happen every time, and not by chance. In addition, what you describe with the file being "almost wiped out" cannot happen due to it being set to 0.



You're actually on to the right thing yourself by mentioning .bash_sessions. That is not an iTerm2 "thing", but rather how it works by default on a standard macOS install. Apple has built-in per-session history on top of a regular bash install.



This means that if you have multiple terminals running (for example multiple tabs), each of those will have a seperate history tracked in .bash_sessions. If you reboot your Mac and the terminal windows are restored, you'll find that each still have their own history - and only their own history.



When you close down a bash session, Apple's system will merge the history for that specific session into the global .bash_history file. Then when you open a new terminal (and thus bash session), it will start with that merged history containing history from potentially multiple sessions.



This is all handled by the /etc/bashrc_Apple_Terminal script.



Now that you know how it works, here are some possible causes for what you're seeing:




  • You may accidentially have removed the history yourself (history -c)


  • Your Mac could be infected with malware and/or hacked, and someone else is removing your .bash_history file


  • You have software installed that periodically empties/removes .bash_history



These reasons are not that likely, I would say.



Now there's one last possible cause:



There's a known race-condition in handling the bash histories. If you close down multiple bash sessions at once (for example closing down a terminal window with multiple tabs), you might loose part of or the whole history.



This happens if the computer is "slow" while reading/writing parts of the history. Basically how it works is that your currently history is read, the file is deleted/moved away, and a new file is created, where all the history lines are written to. If this process is running multiple times at the same time, you might see that one process removes the old history, the other process read an empty history, then the first process writes out the whole history again, and then the other process deletes that and writes out a very short history containing only few recent lines - as it didn't read anything in. This bug shows up as exactly what you have described.



The fix is simply to close one bash session at a time to let the system have time to write out everything correctly.







share|improve this answer












share|improve this answer



share|improve this answer










answered Mar 25 at 13:14









jksoegaardjksoegaard

20.2k2150




20.2k2150













  • Thanks for your explanation I will see if I can replicate this given your answer and get back to you. I am aware that it may not necessarily happen on the first try, but I'd like to be sure it is replicable.

    – Dark Star1
    Mar 25 at 14:18











  • Also just want to add to the malware part - a common OS X RAT known as Eggshell has a macOS payload that actually executes history -wc as part of its code after the main payload, such that all traces of the payload injection command are wiped. Not saying its necessary Eggshell but it could be something common if someone wanted to hide their traces...

    – QuickishFM
    Mar 25 at 14:31











  • This has only happened to me twice and I doubt it is that malware but thanks for the info. Ill look out for it just in case

    – Dark Star1
    Mar 25 at 15:16











  • As a workaround, I have this in my bashrc: export PROMPT_COMMAND='echo $(date +%Y-%m-%dT%H:%M:%S) $(pwd): "$(history 1 | head -c 512 | sed "s/^[ ]*[0-9]+[ ]*//" )" >> ~/.bash_history_full; echo -ne "33]0;${PWD##*/}07"' which will store each command after it is executed.

    – jpa
    Mar 25 at 19:18











  • The disadvantage there is thought that it won't seperate sessions for you - so it will be hard to make sense of stuff afterwards if you have many sessions (for example some local and some remote).

    – jksoegaard
    Mar 25 at 19:42



















  • Thanks for your explanation I will see if I can replicate this given your answer and get back to you. I am aware that it may not necessarily happen on the first try, but I'd like to be sure it is replicable.

    – Dark Star1
    Mar 25 at 14:18











  • Also just want to add to the malware part - a common OS X RAT known as Eggshell has a macOS payload that actually executes history -wc as part of its code after the main payload, such that all traces of the payload injection command are wiped. Not saying its necessary Eggshell but it could be something common if someone wanted to hide their traces...

    – QuickishFM
    Mar 25 at 14:31











  • This has only happened to me twice and I doubt it is that malware but thanks for the info. Ill look out for it just in case

    – Dark Star1
    Mar 25 at 15:16











  • As a workaround, I have this in my bashrc: export PROMPT_COMMAND='echo $(date +%Y-%m-%dT%H:%M:%S) $(pwd): "$(history 1 | head -c 512 | sed "s/^[ ]*[0-9]+[ ]*//" )" >> ~/.bash_history_full; echo -ne "33]0;${PWD##*/}07"' which will store each command after it is executed.

    – jpa
    Mar 25 at 19:18











  • The disadvantage there is thought that it won't seperate sessions for you - so it will be hard to make sense of stuff afterwards if you have many sessions (for example some local and some remote).

    – jksoegaard
    Mar 25 at 19:42

















Thanks for your explanation I will see if I can replicate this given your answer and get back to you. I am aware that it may not necessarily happen on the first try, but I'd like to be sure it is replicable.

– Dark Star1
Mar 25 at 14:18





Thanks for your explanation I will see if I can replicate this given your answer and get back to you. I am aware that it may not necessarily happen on the first try, but I'd like to be sure it is replicable.

– Dark Star1
Mar 25 at 14:18













Also just want to add to the malware part - a common OS X RAT known as Eggshell has a macOS payload that actually executes history -wc as part of its code after the main payload, such that all traces of the payload injection command are wiped. Not saying its necessary Eggshell but it could be something common if someone wanted to hide their traces...

– QuickishFM
Mar 25 at 14:31





Also just want to add to the malware part - a common OS X RAT known as Eggshell has a macOS payload that actually executes history -wc as part of its code after the main payload, such that all traces of the payload injection command are wiped. Not saying its necessary Eggshell but it could be something common if someone wanted to hide their traces...

– QuickishFM
Mar 25 at 14:31













This has only happened to me twice and I doubt it is that malware but thanks for the info. Ill look out for it just in case

– Dark Star1
Mar 25 at 15:16





This has only happened to me twice and I doubt it is that malware but thanks for the info. Ill look out for it just in case

– Dark Star1
Mar 25 at 15:16













As a workaround, I have this in my bashrc: export PROMPT_COMMAND='echo $(date +%Y-%m-%dT%H:%M:%S) $(pwd): "$(history 1 | head -c 512 | sed "s/^[ ]*[0-9]+[ ]*//" )" >> ~/.bash_history_full; echo -ne "33]0;${PWD##*/}07"' which will store each command after it is executed.

– jpa
Mar 25 at 19:18





As a workaround, I have this in my bashrc: export PROMPT_COMMAND='echo $(date +%Y-%m-%dT%H:%M:%S) $(pwd): "$(history 1 | head -c 512 | sed "s/^[ ]*[0-9]+[ ]*//" )" >> ~/.bash_history_full; echo -ne "33]0;${PWD##*/}07"' which will store each command after it is executed.

– jpa
Mar 25 at 19:18













The disadvantage there is thought that it won't seperate sessions for you - so it will be hard to make sense of stuff afterwards if you have many sessions (for example some local and some remote).

– jksoegaard
Mar 25 at 19:42





The disadvantage there is thought that it won't seperate sessions for you - so it will be hard to make sense of stuff afterwards if you have many sessions (for example some local and some remote).

– jksoegaard
Mar 25 at 19:42



Popular posts from this blog

How to change which sound is reproduced for terminal bell?

Can I use Tabulator js library in my java Spring + Thymeleaf project?

Title Spacing in Bjornstrup Chapter, Removing Chapter Number From Contents