Giter Site home page Giter Site logo

Comments (23)

marxin avatar marxin commented on September 26, 2024

Hello.

It seems to me as an unusual temporary directory (expected would be C:\Users\USERNAME\AppData\Local\Temp). Can you please check the usual one?

Unfortunately, I don't have a Windows machine I can test it on.

from cvise.

jketema avatar jketema commented on September 26, 2024

It seems to me as an unusual temporary directory

Yeah, now you point that out, that is odd. c:\\Users\\xxx\\Desktop\\help is the directory in which the source file and script calling
the compiler live. %TEMP% and %TMP% do point to C:\Users\xxx\AppData\Local\Temp. I'll have a look to see why it's not picking up on that.

from cvise.

jketema avatar jketema commented on September 26, 2024

Hi again,

It seems that Windows 11 doesn't like a file to be opened twice. The following fails with the same error:

import tempfile
import shutil

with tempfile.NamedTemporaryFile(mode='w+') as x:
     shutil.copy(some_file_that_exists, x.name)

In addition temp directory is not being used due to the following line:

tmp = os.path.dirname(test_case)

from cvise.

marxin avatar marxin commented on September 26, 2024

Interesting! Anyway, it's something we can probably address with the new API provided by Python 3.12:
https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile
where we can set delete_on_close=False and then call tmp_file.close() in the with block.

Any chance you could test the suggested change on your system, please?

from cvise.

jketema avatar jketema commented on September 26, 2024

Yes, that works if I close both tmp_file after the flush and backup sometime before shutil.copy(test_case, backup.name).

from cvise.

jketema avatar jketema commented on September 26, 2024

I also need to make similar changes to the clang and clex passes, otherwise those passes hang.

from cvise.

marxin avatar marxin commented on September 26, 2024

Great, then let me prepare a proper PR during the weekend that would address that.

from cvise.

jketema avatar jketema commented on September 26, 2024

Thanks for your help. Here's what I did:

diff --git a/cvise/passes/clang.py b/cvise/passes/clang.py
index 1247c88..1e714f1 100644
--- a/cvise/passes/clang.py
+++ b/cvise/passes/clang.py
@@ -21,7 +21,7 @@ class ClangPass(AbstractPass):
 
     def transform(self, test_case, state, process_event_notifier):
         tmp = os.path.dirname(test_case)
-        with tempfile.NamedTemporaryFile(mode='w', delete=False, dir=tmp) as tmp_file:
+        with tempfile.NamedTemporaryFile(mode='w', delete=False, dir=tmp, delete_on_close=False) as tmp_file:
             args = [
                 self.external_programs['clang_delta'],
                 f'--transformation={self.arg}',
@@ -36,9 +36,11 @@ class ClangPass(AbstractPass):
             stdout, _stderr, returncode = process_event_notifier.run_process(cmd)
             if returncode == 0:
                 tmp_file.write(stdout)
+                tmp_file.close()
                 shutil.move(tmp_file.name, test_case)
                 return (PassResult.OK, state)
             else:
+                tmp_file.close()
                 os.unlink(tmp_file.name)
                 if returncode == 255 or returncode == 1:
                     return (PassResult.STOP, state)
diff --git a/cvise/passes/clex.py b/cvise/passes/clex.py
index 64b9000..352fc52 100644
--- a/cvise/passes/clex.py
+++ b/cvise/passes/clex.py
@@ -20,14 +20,16 @@ class ClexPass(AbstractPass):
 
     def transform(self, test_case, state, process_event_notifier):
         tmp = os.path.dirname(test_case)
-        with tempfile.NamedTemporaryFile(mode='w', delete=False, dir=tmp) as tmp_file:
+        with tempfile.NamedTemporaryFile(mode='w', delete=False, dir=tmp, delete_on_close=False) as tmp_file:
             cmd = [self.external_programs['clex'], str(self.arg), str(state), test_case]
             stdout, _stderr, returncode = process_event_notifier.run_process(cmd)
             if returncode == 51:
                 tmp_file.write(stdout)
+                tmp_file.close()
                 shutil.move(tmp_file.name, test_case)
                 return (PassResult.OK, state)
             else:
+                tmp_file.close()
                 os.unlink(tmp_file.name)
                 return (
                     PassResult.STOP if returncode == 71 else PassResult.ERROR,
diff --git a/cvise/passes/lines.py b/cvise/passes/lines.py
index 2eb0d48..6c67c59 100644
--- a/cvise/passes/lines.py
+++ b/cvise/passes/lines.py
@@ -15,9 +15,10 @@ class LinesPass(AbstractPass):
     def __format(self, test_case, check_sanity):
         tmp = os.path.dirname(test_case)
 
-        with tempfile.NamedTemporaryFile(mode='w+', dir=tmp) as backup, tempfile.NamedTemporaryFile(
-            mode='w+', dir=tmp
+        with tempfile.NamedTemporaryFile(mode='w+', dir=tmp, delete_on_close=False) as backup, tempfile.NamedTemporaryFile(
+            mode='w+', dir=tmp, delete_on_close=False
         ) as tmp_file:
+            backup.close()
             with open(test_case) as in_file:
                 try:
                     cmd = [self.external_programs['topformflat'], self.arg]
@@ -29,6 +30,7 @@ class LinesPass(AbstractPass):
                 if not line.isspace():
                     tmp_file.write(line)
             tmp_file.flush()
+            tmp_file.close()
 
             # we need to check that sanity check is still fine
             if check_sanity:

from cvise.

jketema avatar jketema commented on September 26, 2024

There's also a similar problem in the clangbinarysearch pass, which I had missed initially, as I ran with a test case that was already reduced.

from cvise.

marxin avatar marxin commented on September 26, 2024

Can you please test the pull request I've just created?

from cvise.

jketema avatar jketema commented on September 26, 2024

On your PR I'm seeing it hanging in ClangPass::remove-unused-function that probably means there's still some file that cannot be accessed. Before any of mine or your changes I also saw this with ClangBinarySearchPass, but that works with your PR, so there must be some subtle difference between the passes. It's not clear to me what though, the code seems pretty much identical.

from cvise.

marxin avatar marxin commented on September 26, 2024

Thanks for the testing effort! Yeah, the passes seem very much the same. Can you please terminate the program and show me the corresponding back-traces?

from cvise.

jketema avatar jketema commented on September 26, 2024

The backtrace I'm getting is:

Exception ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\managers.py", line 873, in _decref
    conn = _Client(token.address, authkey=authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 507, in Client
    answer_challenge(c, authkey)
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 935, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 215, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 304, in _recv_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt:
00:00:28 INFO Exiting now ...

from cvise.

jketema avatar jketema commented on September 26, 2024

Note: eventually cvise does continue after printing:

00:05:42 WARNING ClangPass::remove-unused-function has encountered a non fatal bug: pass got stuck

from cvise.

marxin avatar marxin commented on September 26, 2024

Can you please add more debugging output to the pass:

diff --git a/cvise/passes/clang.py b/cvise/passes/clang.py
index 990184d..c6cbc4b 100644
--- a/cvise/passes/clang.py
+++ b/cvise/passes/clang.py
@@ -33,7 +33,8 @@ class ClangPass(AbstractPass):
 
             logging.debug(' '.join(cmd))
 
-            stdout, _, returncode = process_event_notifier.run_process(cmd)
+            stdout, stderr, returncode = process_event_notifier.run_process(cmd)
+            print(stdout[:64], stderr, returncode)
             if returncode == 0:
                 tmp_file.write(stdout)
                 tmp_file.close()

and test it?

from cvise.

jketema avatar jketema commented on September 26, 2024

The output is:

Error: The counter value exceeded the number of transformation i  1

Where print(cmd) gives:

['C:/Users/xxx/Desktop/cvise/tools/cvise2/libexec\\cvise\\clang_delta.EXE', '--transformation=remove-unused-function', '--counter=505', 'C:\\Users\\xxx\\AppData\\Local\\Temp\\cvise-ClangPass-remove-unused-function-leyd0zb3\\cvise-yp5abkhu\\test.cpp']

And test.cpp contains:

template <class a, class...> bool b = __is_nothrow_constructible(a);
                                        class c;
                                        class B {
                                     template <typename... d> B() noexcept(b<c, d...>);
                                   };
                                        class e : B {
                                  };

from cvise.

marxin avatar marxin commented on September 26, 2024
Error: The counter value exceeded the number of transformation i  1

Yeah, that's expected output, but I don't see why should it become stuck. Please attach a full reduction log: (with --debug option).

from cvise.

jketema avatar jketema commented on September 26, 2024

There's very little log data:

...
00:00:15 INFO ===< ClangBinarySearchPass::remove-unused-function >===
00:00:15 DEBUG available transformation opportunities for c++98: 1, took: 0.02 s
00:00:15 DEBUG available transformation opportunities for c++11: 1, took: 0.03 s
00:00:15 DEBUG available transformation opportunities for c++14: 1, took: 0.02 s
00:00:15 DEBUG available transformation opportunities for c++17: 1, took: 0.02 s
00:00:15 DEBUG available transformation opportunities for c++20: 1, took: 0.02 s
00:00:15 DEBUG available transformation opportunities for c++2b: 1, took: 0.02 s
00:00:15 INFO using C++ standard: c++2b with 1 transformation opportunities
00:00:16 DEBUG Creating pass root folder: C:\Users\xxx\AppData\Local\Temp\cvise-ClangPass-remove-unused-function-0sj3kcmo
00:00:16 INFO ===< ClangPass::remove-unused-function >===
00:06:38 WARNING ClangPass::remove-unused-function has encountered a non fatal bug: pass got stuck
00:06:38 DEBUG Please consider tarring up cvise_bug_2 and creating an issue at https://github.com/marxin/cvise/issues and we will try to fix the bug.
00:06:51 DEBUG Creating pass root folder: C:\Users\xx\AppData\Local\Temp\cvise-BalancedPass-curly-ze7mdy62
00:06:51 INFO ===< BalancedPass::curly >===
...

from cvise.

jketema avatar jketema commented on September 26, 2024

The problem goes away when I close the file in the location where the unlink was at the beginning of the else branch in the clang pass. And similarly for the clex pass.

from cvise.

marxin avatar marxin commented on September 26, 2024

Oh, got catch!

Apparently, it's something that produces an exception:
https://docs.python.org/3/library/os.html#os.remove

On Windows, attempting to remove a file that is in use causes an exception to be raised; on Unix, the directory entry is removed but the storage allocated to the file is not made available until the original file is no longer in use.

Please try the latest version of the #139.

from cvise.

jketema avatar jketema commented on September 26, 2024

The latest version works for me!

from cvise.

marxin avatar marxin commented on September 26, 2024

Great, thanks for testing!

from cvise.

jketema avatar jketema commented on September 26, 2024

Thanks for fixing!

from cvise.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.