Atomically copying and truncating a file being appended to

reno_rita

I have a log file that is being appended to with a line-buffered writer, and each line is written in a single call so at any given time, the log file is consistent. How do I copy it to a second file and then truncate the first file without the risk of losing writes in the window between the copy and truncation? Essentially, I want to perform these two operations atomically without relying on "foo" being closed. shutil.copyfile("foo", "bar")...

Originally asked by forest on Stack Exchange (unix) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

A common solution is to use rename() instead of copy + truncate. Rename the active log file to bar , then immediately create a new foo . The writer can keep its existing file descriptor, while new writes continue safely to the old inode. However, this only works if the writer opens/reopens the log appropriately. If it must keep writing to the same open inode, copy + truncate cannot be made truly atomic without coordinating with the writer. A short pause or proper log-rotation mechanism is safer than risking lost lines.

Full answer by Prashant Kumar: https://unix.stackexchange.com/a/807435 (CC BY-SA 4.0).

Need it done for your business? UphillTech.