Grammar caching causes EOFError and race condition when used as pre-commit hook

[a-gardner1]

Grammar caching causes EOFError and race condition when used as pre-commit hook

[kamahen]

The problem seems to be in Grammar.dump() in file yapf/third_party/yapf_third_party/_ylib2to3/pgen2/grammar.py - it opens the file directly instead of creating a temporary name, writing to the file, then renaming it (this works on Unix, where rename is an atomic operation (as are open, close, linke, etc.), so it's safe if two processes do the same thing at the same time). I don't know if this technique works on Windows.

Something like this:

def dump(self, filename):
  """Dump the grammar tables to a pickle file."""
  with tempfile.NamedTemporaryFile(mode='wb') as f:
    pickle.dump(self.__dict__, f, pickle.HIGHEST_PROTOCOL)
    os.link(f.name, filename)

[a-gardner1]

Agreed, I think that would work on Unix systems. To my knowledge, there is no documented, guaranteed atomic write operation on Windows. The python-atomicwrites library provides a best-effort atomic write operation for Windows; there may or may not be something better out there.

[kamahen]

There's an alternative method that runs a small risk of leaving a temporary file lying around, but might work better on Windows [this code is untested; and it probably should have a try/except to deal with cleanup on error]:

  with tempfile.NamedTemporaryFile(mode='wb', delete=False, delete_on_close=False) as f:
    tmp_file_name = f.name
    pickle.dump((self.__dict__, f, pickle.HIGHEST_PROTOCOL)
  os.rename(tmp_file_name, filename)