Bash history loss when using histappend

I like to keep a lot of history, so I have histappend set in my .bashrc. Most of the time everything works fine, with history built up from many shells appending. However, every once and a while, I'll start a new shell and find that I've lost the entire history - and it often only contains some of the commands from the last shell to exit (i.e. it's not just overwriting instead of appending). Because of this, I'm suspicious it's happening at shell exit, rather than from some other process killing the .bash_history file. Supporting this conclusion, I have history command numbers in my prompt, and I've never seen them jump down.

Anyone ever run into a similar problem? Or even just have suggestions how to track down the problem?

Cascabel

Posted 2009-08-11T01:24:44.720

Reputation: 1 240

See: patch #8676: Fix truncating .bash_history

– kenorb – 2015-05-28T10:35:36.363

Answers

Sorry to answer my own question, but none of the other answers really address the problem.

I've finally figured out that this only happens when closing gnome-terminal itself (i.e. file > exit, the 'x' button, alt+F4), and even then generally only when closing several terminals in quick succession. It never happens when using ctrl-D to close the shell, letting the terminal follow.

If I can pin it down well enough, I'll file a gnome-terminal bug report. In the meantime, perhaps this will help some other people who get here from google!

Cascabel

Posted 2009-08-11T01:24:44.720

Reputation: 1 240

No idea why this happens, but maybe you can circumvent the problem by forcing bash to write to its history file each time it displays a prompt:

PROMPT_COMMAND="history -a; history -n"

This will write (-a) and then re-read (-n) the history file each time bash prompts for the next command. Additional benefit: you'll get command X in shell 1 in the history of shell 2.

innaM

Posted 2009-08-11T01:24:44.720

Reputation: 9 208

Doesn't work on GNU bash, version 3.00.15(1)-release (i686-redhat-linux-gnu) – David Mackintosh – 2009-08-11T12:39:35.990

2Can you explain what "doesn't work" means? – innaM – 2009-08-11T13:02:55.293

3The additional benefit you cite is in many cases a downside. It's not the behavior I'm looking for, as I may be carrying out two completely separate tasks in separate shells, and do not want to intermingle their history. This would also likely not help anything. When the history disappears, it is deleting content of .bash_history - I don't expect it'd matter whether they'd been written at shell exit or by PROMPT_COMMAND. – Cascabel – 2009-08-11T22:09:28.947

Sure. If I start shell A, and shell B, then type "echo Bookie" into shell A, that command doesn't get into the history of shell B. – David Mackintosh – 2009-08-11T23:37:20.500

Right, but if you ever want to reexecute a series of commands - in shell C started after A and B, or perhaps you realize you want to turn it into a script - it's been mixed. – Cascabel – 2009-08-12T00:36:11.550

5history -n is flaky. It's more reliable to do history -a; history -c; history -r. To explain this, I'll first note that history -a does the right thing - your .bash_history will contain all your commands that you typed, in the order that you typed them - assuming that you run history -a after every command. The challenge is to keep the shell's idea of history in sync with the .bash_history file. This is easy with -c and -r, the problem is that it could be slow if it's large. -n can break because it incorrectly identifies which lines are new. (I'm running out of space here!) – Aaron McDaid – 2012-12-01T13:10:31.527

4(... if you use -n) Imagine you execute a command in shell 1: ls. Then in another shell, Shell Two, you execute cd. Now, the history in the .bash_history is correct because of the history -a in your PROMPT_COMMAND - it will contain ls \n cd \n. Next, you return to shell One and type pwd. Shell One thinks that there was only command n the history (ls). Now it thinks there are two commands (ls and pwd) in the history. When you do -n it thinks (I have two commands in my history, and there are two commands in .bash_history, therefore I'm up to date.) – Aaron McDaid – 2012-12-01T13:20:44.237

This answer doesn't work for me either (in gnome-terminal on ubuntu 12.10), but Aaron's modification sorta kinda works (it seems to update 1 command later) – wim – 2013-02-26T07:04:42.177

My experience was that shells updated the history file at exit time. So a shell's initial "history" depended on the most recently exited shell's view of the history.

The result of this is that you can get commands coming and going from the history, depending on how other shells started and stopped.

David Mackintosh

Posted 2009-08-11T01:24:44.720

Reputation: 3 728

2I understand very well how the history file is written - this is why I specified in my question that I'm using histappend. The problem is not unexpected content, but a total loss of previously stored content. – Cascabel – 2009-08-11T22:14:34.113

This explains why I was losing my history... – B Seven – 2012-08-20T23:13:54.280

I have seen this happen before but it was a problem with disk errors that were happening in increasing frequency. I would run a scan on the drive. If it turns out the drive is fine, I would check to see if this file is not surpassing an arbitrary shell history limit.

Something that might be able to keep that from happening would be to keep pruning the file back to 80 lines or however many commands you want the history to be.

Axxmasterr

Posted 2009-08-11T01:24:44.720

Reputation: 7 584

I don't have root access on the machine this is happening on, but I'm fairly confident the drive is ok. My home directory is stored on a server in our lab (plenty of RAID, I believe) and nfs-mounted.

What do you mean by "arbitrary shell history limit"? This is all happening well below HISTSIZE and HISTFILESIZE, and though I have both set large, they're well below the int bash stores them as. – Cascabel – 2009-08-11T22:13:26.597

I would have to say that David Mackintosh's entry is probably what is happening. – Axxmasterr – 2009-08-11T22:40:15.047

1I am entirely sure that it is not. I should never end up with only two commands in my history, when the last shell to exit had a couple dozen commands, the history file had several hundred, and HISTSIZE/HISTFILESIZE are set to 10000. – Cascabel – 2009-08-12T00:37:38.977