frerich/clcache

A compiler cache for MSVC, much like ccache for gcc

clcache.py - a compiler cache for Microsoft Visual Studio

clcache.py is a little Python script which attempts to avoid unnecessary recompilation by reusing previously cached object files if possible. It is meant to be called instead of the original 'cl.exe' executable. The script analyses the command line to decide whether source code is to be compiled. If so, a cache will be queried for a previously stored object file.

If the script is called in an unsupported way (e.g. if the compiler is called for linking), the script will simply relay the invocation to the real 'cl.exe' program.

Installation

Please see the Wiki for instructions on how to install clcache and different approaches on how to integrate it into a build system.

Options

--help

Print usage information

-s

Print some statistics about the cache (cache hits, cache misses, cache size etc.)

-c

Clean the cache: trim the cache size to 90% of its maximum by removing the oldest objects.

-C

Clear the cache: remove all cached objects, but keep the cache statistics (hits, misses, etc.).

-z

Reset the cache statistics, i.e. number of cache hits, cache misses etc.. Doesn’t actually clear the cache, so the number of cached objects and the cache size will remain unchanged.

-M <size>

Sets the maximum size of the cache in bytes. The default value is 1073741824 (1 GiB).

compiler

It is, optionally, possible to specify the full path to the compiler as the first argument on the command line, in the style of ccache, instead of using the CLCACHE_CL environment variable or searching the path for cl.exe

Environment Variables

CLCACHE_DIR

If set, points to the directory within which all the cached object files should be stored. This defaults to %HOME%\clcache

CLCACHE_CL

Can be set to the actual 'cl.exe' executable to use. If this variable is not set, the 'clcache.py' script will scan the directories listed in the PATH environment variable for 'cl.exe'. In case this is just a file name (as opposed to an absolute path), 'clcache.py' will scan the directories mentioned by the %PATH% environment variable to compute the absolute path.

CLCACHE_LOG

If this variable is set, a bit of diagnostic information is printed which can help with debugging cache problems.

CLCACHE_DISABLE

Setting this variable will disable 'clcache.py' completely. The script will relay all calls to the real compiler.

CLCACHE_HARDLINK

If this variable is set, cached object files won’t be copied to their final location. Instead, hard links pointing to the cached object files will be created. This is more efficient (faster, and uses less disk space) but doesn’t work if the cache directory is on a different drive than the build directory.

CLCACHE_COMPRESS

If true, clcache will compress object files it puts in the cache. If the cache was filled without compression it can’t be used with compression and vice versa (i.e. you have to clear the cache when changing this setting). The default is false.

CLCACHE_COMPRESSLEVEL

This setting determines the level at which clcache will compress object files. It only has effect if compression is enabled. The value defaults to 6, and must be no lower than 1 (fastest, worst compression) and no higher than 9 (slowest, best compression).

CLCACHE_NODIRECT

Disable direct mode. If this variable is set, clcache will always run preprocessor on source file and will hash preprocessor output to get cache key. Use this if you experience problems with direct mode or if you need built-in macroses like _TIME_ to work correctly.

CLCACHE_BASEDIR

Has effect only when direct mode is on. Set this to path to root directory of your project. This allows clcache to cache relative paths, so if you move your project to different directory, clcache will produce cache hits as before.

CLCACHE_OBJECT_CACHE_TIMEOUT_MS

Overrides the default ObjectCacheLock timeout (Default is 10 * 1000 ms). The ObjectCacheLock is used to give exclusive access to the cache, which is used by the clcache script. You may override this variable if you are getting ObjectCacheLockExceptions with return code 258 (which is the WAIT_TIMEOUT return code).

CLCACHE_PROFILE

If this variable is set, clcache will generate profiling information about how the runtime is spent in the clcache code. For each invocation, clcache will generate a file with a name similiar to 'clcache-<hashsum>.prof'. You can aggregate these files and generate a report by running the 'showprofilereport.py' script.

CLCACHE_SERVER

Setting this environment variable will make clcache use (and expect) a running clcachesrv.py script which takes care of caching file hashes. This greatly improves performance of cache hits, but only has an effect in direct mode (i.e. when CLCACHE_NODIRECT is not set).

CLCACHE_MEMCACHED

This variable can be used to make clcache use a memcached[https://memcached.org/] backend for saving and restoring cached data. The variable is assumed to hold the host and port information of the memcached server, e.g. 127.0.0.1:11211.

Known limitations

• INCLUDE and LIBPATH environment variables are not supported.

How clcache works

clcache.py was designed to intercept calls to the actual cl.exe compiler binary. Once an invocation has been intercepted, the command line is analyzed for whether it is a command line which just compiles a single source file into an object file. This means that all of the following requirements on the command line must be true:

• The /link switch must not be present

• The /c switch must be present

• The /Zi switch must not be present (/Z7 is okay though)

If multiple source files are given on the command line, clcache.py wil invoke itself multiple times while respecting an optional /MP switch.

If all the above requirements are met, clcache forwards the call to the preprocessor by replacing /c with /EP in the command line and then invoking it. This will cause the complete preprocessed source code to be printed. clcache then generates a hash sum out of

• The complete preprocessed source code

• The `normalized' command line

• The file size of the compiler binary

• The modification time of the compiler binary

The `normalized' command line is the given command line minus all switches which either don’t influence the generated object file (such as /Fo) or which have already been covered otherwise. For instance, all switches which merely influence the preprocessor can be skipped since their effect is already implicitly contained in the preprocessed source code.

Once the hash sum is computed, it is used as a key (actually, a directory name) in the cache (which is a directory itself). If the cache entry exists already, it is supposed to contain a file with the stdout output of the compiler as well as the previously generated object file. clcache will copy the previously generated object file to the designated output path and then print the contents of the stdout text file. That way, the script behaves as if the actual compiler was invoked.

If the hash sum is not yet used in the cache, clcache will forward the invocation to the actual compiler. Once the real compiler successfully finished its work, the generated object file (as well as the output printed by the compiler) is copied to the cache.

Caveats

For known caveats, please see the Caveats wiki page.

License Terms

The source code of this project is - unless explicitly noted otherwise in the respective files - subject to the BSD 3-Clause License.

Credits

clcache.py was written by Frerich Raabe with a lot of help by Slava Chigrin, Simon Warta, Tim Blechmann, Tilo Wiedera and other contributors.

This program was heavily inspired by ccache, a compiler cache for the GNU Compiler Collection.

clcache's Issues

option to ignore /Zi

There should be an option (enabled with an environment variable perhaps) to ignore (eat) unsupported options like /Zi.

Perhaps this could work with /Yu too (precompiled headers)

Clean/Rebuild deletes stats and cache objects

Apologies if this has been brought up before. I did not find mention of it in the currently open issues and pull requests.

I'm using Visual Studio 2013 Update 5 with CMake-generated vcxproj/sln files. I am able to get reliable cache hit rates from what I see in clcache.exe -s. However, one unfortunate behavior is that anytime I select Clean, all the cache stats get reset to zero, apparently because stats.txt has been removed, and several cache objects are removed as well.

This unfortunately applies to the Rebuild command too, or even Project Only -> Rebuild Only .

It appears that the cleaning mechanism in Visual Studio 2013 works by tracking any files written by cl.exe so it can know how to do a reliable clean. This would not be a problem for nmake projects, and I'm not sure if this nifty mechanism was in earlier versions of Visual Studio.

Has anyone else encountered this? Any solutions or practical workarounds?

Or if you are using Visual 2013 or 2015 and have not seen this problem, that would be good to know. I do have one extension installed (VSCommands) but I doubt that it would affect the cleaning behavior.

What is the /vm argument?

In optionsWithParameter, there is a /vm argument listed, which I cannot find in the MSVS documentation.

Instead there are a bunch of switches with that prefix: /vmb /vmg /vmm /vms /vmv, which are not parameters of /vm but independent arguments.

Can /vm go?

python3: encoding error

when using clcache/master via python 3, i encountered the following error:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python34\lib\threading.py", line 920, in _bootstrap_inner
  File "C:\Python34\lib\threading.py", line 868, in run
  File "C:\Python34\lib\subprocess.py", line 1171, in _readerthread
  File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 161099: character maps to <undefined>

Traceback (most recent call last):
  File "clcache.py", line 1200, in <module>
  File "clcache.py", line 1100, in main
  File "clcache.py", line 1166, in processCompileRequest
  File "clcache.py", line 1026, in processNoManifestMiss
  File "clcache.py", line 769, in invokeRealCompiler
  File "C:\Python34\lib\subprocess.py", line 960, in communicate
  File "C:\Python34\lib\subprocess.py", line 1235, in _communicate
IndexError: list index out of range

it seems to be related to the fact that the compiler is using a german locale :(

[Error 183] Cannot create a file when that file already exists

I'm using Windows 7 + VS2012 Express + Python 2.7. Some of my project got below error, some get passed.

I'm following the setup process for Visual Studio, not sure what kind of problem will cause this issue.

1> Traceback (most recent call last):
1> File "clcache.py", line 1376, in
1> File "clcache.py", line 1250, in main
1> File "clcache.py", line 1281, in processCompileRequest
1> File "clcache.py", line 1343, in processDirect
1> File "clcache.py", line 1340, in
1> File "clcache.py", line 1181, in postprocessNoManifestMiss
1> File "clcache.py", line 1092, in addObjectToCache
1> File "clcache.py", line 284, in setEntry
1> File "clcache.py", line 617, in copyOrLink
1> WindowsError: [Error 183] Cannot create a file when that file already exists
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V110\Microsoft.CppCommon.targets(347,5): error MSB6006: "CL.exe" exited with code 255.

clcache is packaged into conda-forge

Hi, I just want to inform you that I packaged clcache in conda-forge.

This is the URL of the feedstock: https://github.com/conda-forge/clcache-feedstock

Best,

Global cache lock hurts cache hits during heavily concurrent builds

Hi!
I tried to use clcache for my project and found speed up only on 35% (23 min vs 35 min). This result greatly embarrassed me because ccache at Linux and MacOS boost build time more then 10 times. I try change msbuild to ninja, but it gave similar results.
Then I suggested problem in long waits at lock (at other machine I even got crashes by timeouts). I try cut all lock logic and stats file overwriting. See diff below. Then build time became less then 7 min! It's more then 3 times faster. Looks like absolutely unacceptable overhead for writing statistics.
I'm not sure is access to manifests needs lock too, but looks if it is, we can do it more optimal. And about statistics, I think we can write simple server for writing it with locks, and clcache processes will just sent stat to server and exit without waiting responce. Or we can add option to disable statistics (and lock respectively) as quick fix.

--- a/clcache.py
+++ b/clcache.py
@@ -123,6 +123,7 @@ class ObjectCacheLock(object):
     WAIT_ABANDONED_CODE = 0x00000080

     def __init__(self, mutexName, timeoutMs):
+        return
         mutexName = 'Local\\' + mutexName
         self._mutex = windll.kernel32.CreateMutexW(
             wintypes.INT(0),
@@ -133,17 +134,21 @@ class ObjectCacheLock(object):
         assert self._mutex

     def __enter__(self):
+        return
         if not self._acquired:
             self.acquire()

     def __exit__(self, typ, value, traceback):
+        return
         if self._acquired:
             self.release()

     def __del__(self):
+        return
         windll.kernel32.CloseHandle(self._mutex)

     def acquire(self):
+        return
         result = windll.kernel32.WaitForSingleObject(
             self._mutex, wintypes.INT(self._timeoutMs))
         if result not in [0, self.WAIT_ABANDONED_CODE]:
@@ -154,6 +159,7 @@ class ObjectCacheLock(object):
         self._acquired = True

     def release(self):
+        return
         windll.kernel32.ReleaseMutex(self._mutex)
         self._acquired = False

@@ -373,8 +394,9 @@ class PersistentJSONDict(object):

     def save(self):
         if self._dirty:
-            with open(self._fileName, 'w') as f:
-                json.dump(self._dict, f, sort_keys=True, indent=4)
+            pass
+            #with open(self._fileName, 'w') as f:
+            #    json.dump(self._dict, f, sort_keys=True, indent=4)

     def __setitem__(self, key, value):
         self._dict[key] = value

analyzeCommandLine() bug

When finding output filename in arguments in 'analyzeCommandLine()', should be arg[1:3] == 'Fo', not arg[1:] == 'Fo'.

ex) cl.exe -Fosrc-dir/a.cpp ...

A couple of bugs

As far as I can see, the cachelock isn't actually acquired. I believe you need to actually call lock.acquire() and lock.release() since FileLock's auto-lock only engages on a 'with' statement.

Also, the expandCommandLine function is dodgy because it splits on space. This is bad when you have spaces in filepaths. It would probably be good to use shlex.split instead.

CommandLineAnalyzer interface change

Due to historic reasons, The CommandLineAnalyzer is a collection of static functions (analyze() and it's helpers). This is okay, but there are some points which are not optimal:

1. Since there is no internal state (= no members), the options variable is passed around all the time.
2. The name "analyze" is not helping a lot
3. The output if analyze() is an unnamed tupel
4. We cannot test the content of options
5. static functions cause long caller code (e.g. CommandLineAnalyzer._outputFileFromArgument)

Thus I'd like to propose an Object-style interface

• Rename CommandLineAnalyzer to CommandLineAnalysis
• Store options in the self.arguments member
• Store sourceFiles in the self.sourceFiles member
• Store outputFile in the self.outputFile member
• Make the analyze() body the initializer of CommandLineAnalysis

The caller code then looks like this in clcache

analysis = CommandLineAnalysis(cmdLine)
sourceFiles = analysis.sourceFiles
outputFile = analysis.outputFile

where the local variables sourceFiles/outputFile could be avoided by just using analysis.sourceFiles, analysis.outputFile in the 5 places to come.

The code could be tested via

analysis = CommandLineAnalysis(cmdLine)
arguments = analysis.arguments

Executable generated with pyinstaller is slower than cx_freeze

We found that cx_freeze generates faster executables.

For example, with a hot cache of 1248 targets:

• executable generated with "pyinstaller --onefile clcache.py" takes 99 seconds
• executable generated with "C:\Python34\Scripts\cxfreeze clcache.py" takes 56 seconds

The downside is that cx_freeze cannot generate single file executables but the speedup is well worth it.

3-clause bsd license applicable for the whole project?

Hi,

first of all thanks for this great project. Is the whole project under 3-clause BSD license or only the files which have a copyright notice on the top? If so, would you mind adding a LICENSE file? This would ease a contribution I would like to make that will help with #78

Thanks!

INCLUDE & LIBPATH environment variables are not respected

clcache does not respect any of the CL Environment Variables. Most notably,

• CL can be used to specify arguments to be prepended to the command line.
• _CL_ can be used to specify arguments to be appended to the command line.

I suspect INCLUDE should be honoured as well, but I'm not sure whether it overrides or is overridden by paths specified on the command line. Same goes for LIBPATH.

Performance improvement unexpectedly small

After resolving my last issues I got back to do some performance tests on my main project and I see a surprisingly small performance improvement: I am building with ninja on a 24 core machine on a SSD. A build with empty cache takes about 48 min. The rebuild with a full cache takes about 30min. Here are the cache statistics for a cold and warm build.

clcache statistics:
  current cache dir         : D:\j2/clcache
  cache size                : 4,354,861,756 bytes
  maximum cache size        : 10,073,741,824 bytes
  cache entries             : 6476
  cache hits                : 6394
  cache misses
    total                      : 6476
    evicted                    : 0
    header changed             : 40
    source changed             : 6436
  passed to real compiler
    called w/ invalid argument : 0
    called for preprocessing   : 762
    called for linking         : 8
    called for external debug  : 22
    called w/o source          : 0
    called w/ multiple sources : 0
    called w/ PCH              : 0

I profiled both runs: prof4-cold.txt and prof4-warm.txt
If I read the 'warm' one correctly, most of the time is spend in computing file hashes, over 2 million times. Are these all the headers from all files? If this is correct, what I do not understand then, why the 'warm' case has a small amount of more calls to getFileHash() and why it takes more than double the time.
Any ideas or input how I might improve clcaches performance improvements?

FYI: My still unfinished lockfile mode does not change these numbers significantly. Also using xcopy instead of shutil.copyfile() or using readinto() instead of read() in getFileHash() does not change the situation.

ObjectCache.clean is not safe for concurrent builds

Running multiple builds simultaneously causes the occasional python crashes, due to ObjectCache.clean trying to stat() or rmtree() missing files.

I patched it thus:

def clean(self, stats, maximumSize):
    currentSize = stats.currentCacheSize()
    if currentSize < maximumSize:
        return

    objects = [os.path.join(root, "object")
            for root, folder, files in os.walk(self.dir)
            if "object" in files]

    objectInfos = []
    for fn in objects:
        try:
            objectInfos.append( (os.stat(fn), fn) )
        except:
            pass
    objectInfos.sort(key=lambda t: t[0].st_atime, reverse=True)

    for stat, fn in objectInfos:
        try:
            rmtree(os.path.split(fn)[0])
        except:
            pass
        currentSize -= stat.st_size
        if currentSize < maximumSize:
            break

    stats.setCacheSize(currentSize)

clcache.py can not deal with args with double \

when dealing with some args like /Fo"Release", clcache.py will faild to regnizing the 2nd quote.

cache directory and stats go out of sync

in my configuration, the size of the cache dir is configured to be 200GB, but after some time, the cache directory exceeds the stats.txt file: the file reports something like 160GB usage, but the real disk utilisation is about 1TB.

Why pickle?

Is there any good reason, we're using pickle to store the manifests?

As far as I can see, pickle does nothing else than storing and loading python structures like

[
    [
        'path/to/include1',
        'path/to/include2',
    ],
    [
        'include1Hash',
        'include2Hash',
    ]
]

in text format with some binary separator magic.

Wouldn't it be much more maintainable to store those manifests as JSON? JSON is a format, we already use, everyone is familiar with and that is highly human readable. There is a good chance that the empty includes lists found in #58 would have been noted earlier when the storage format was easier to debug.

Visual Studio 2010

I am trying to run clcache with Visual Studio 2010.

I have tried putting it inthe PATH of my user, of the whole system and even replacing my cl.exe with this launcher and speicyfing CLCACHE_CL to the correct compiler (renamed to cl.real.exe). My code let me specify where to find clcache.py, plus I am having problems with py2exe and my windows is x86_64.

In the current setup I have these variable set

CLCACHE_DIR = C:\clcache
CLCACHE_CL = C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\cl.real.exe
CLCACHE_HARDLINK = 1
CLCACHE_LOG = 1
CLCACHE_PATH = C:\sources\clcache

#include <string>
#include <stdlib.h>

int main( int argc, char* argv[] )
{
    // Checking standard variables
    char *CLCACHE_PATH = getenv("CLCACHE_PATH");
    std::string cmd = "python "; 
    if (CLCACHE_PATH)
    {
        cmd += CLCACHE_PATH;
        cmd += '/';
    }
    cmd += "clcache.py ";
    for ( int i = 1; i < argc; ++i )
    {
        cmd += ' ';
        cmd += argv[i];
    }
    return system(cmd.c_str());
}

with the PATH modifications it wasn't being called, with this launcher I can manage to make it work if I manually call it, but visual studio will return me

The system cannot execute the specified program.
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Platforms\Win32\Microsoft.Cpp.Win32.Targets(62,5): error MSB6006: "CL.exe" exited with code 1.

How can I setup Visual Studio 2010 to actually use clcache?

All cache misses after recompilation if using Visual Studio directly for compiling instead of cmake

Compling from cmake in Windows actually works quite well and there are lots of cache hits for recompilations.
The tricky thing is that if instead I generate the VS solution and then compile. Then I remove the solution and generate it back again and try to build the solution again from VS, I always get cache misses.

Another case: If I compile with the cmake, then delete the project and recompile with cmake, there are lots of cache hits.
If now I delete the project and generate the visual studio solution, open it and build solution, I get 0 cache hits, all cache misses.

Integrationtests fail when Python path contains space

With a Python installation at C:\Program Files\Python35\python.exe, integrationtests fail in TestPrecompiledHeaders.testSampleproject

c1xx: fatal error C1083: Cannot open source file: 'doesnotexist.cpp': No such fi
le or directory
clcache: preprocessor failed
.       C:\Program Files\Python35\python.exe C:\workspace\clcache\clcache.py /no
logo /EHsc /c /W3 /Gs /MT /Ycstable.h applib.cpp
Der Befehl "C:\Program" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.
NMAKE : fatal error U1077: 'C:\Program' : return code '0x1'
Stop.
Eminimal.cpp
#line 1 "tests\\integrationtests\\minimal.cpp"
void f() {}
minimal.cpp

This is because CLCACHE_CMD is joined using spaces, so the Python path is not quoted in the shell.

This is not important for me right now. Just wanted to document it.

clcache with Visual Studio 12 on AppVeyor

Hi! I have tried several different ways, both reasonable and unreasonable :) to set up clcache with Visual Studio 12 on AppVeyor to build FreeCAD, but it is still not working... Would someone have an idea what I am doing wrong or what else I could try? :\

Here is the build history, from build 1.0.131 onward https://ci.appveyor.com/project/sasobadovinac/freecad/history

Concurrency issues during clean operation

Hi there,

I run clcache in a multi-threaded environment. For my particular machine I can have up to four build jobs going at once. When using clcache, I'm now getting frequent errors due to hitting the cache size limit and clcache trying to clean the same directories at the same time. I get errors such as the following:

Traceback (most recent call last):
File "/.../clcache.py", line 405, in
cache.clean(stats, cfg.maximumCacheSize())
File "/l.../clcache.py", line 59, in clean
objectInfos = [(os.stat(fn), fn) for fn in objects]
OSError: [Errno 2] No such file or directory: '/home/.../clcache/7f/7f7b89c5cca052d07b843df89b86cb9110557bec/object'

I'm guessing it needs some sort of mutex object to ensure multiple processes don't clean the same files, or clean too many files.

Cheers,

tvStatic

clcache statistics seem wrong

I'm using clcache together with CMake and ninja, and noted that it is printing weird statistics.

Here is the workflow:

> mkdir build
> cd build
> set CC=clcache
> set CXX=clcache
> cmake -G Ninja ..
> clcache -C
> clcache -z
> ninja clean
> ninja
> clcache -s

THe clcache -C is just to ensure a clean cache in this verification process.
I noticed that in the end the cache is correctly populated with 55 objects, but the statistics only show 4 cache entries:

clcache statistics:
  current cache dir         : C:\Users\tadeu\clcache
  cache size                : 73,960,833 bytes
  maximum cache size        : 34,359,738,368 bytes
  cache entries             : 4
  cache hits                : 0
  cache misses
    total                      : 4
    evicted                    : 0
    header changed             : 0
    source changed             : 0
  passed to real compiler
    called for linking         : 0
    called for external debug  : 0
    called w/o source          : 0
    called w/ multiple sources : 0
    called w/ PCH              : 0

After a new ninja clean and ninja (which is a lot faster, due to clcache hits), the clcache -s statistics show:

clcache statistics:
  current cache dir         : C:\Users\tadeu\clcache
  cache size                : 73,960,833 bytes
  maximum cache size        : 34,359,738,368 bytes
  cache entries             : 4
  cache hits                : 5
  cache misses
    total                      : 4
    evicted                    : 0
    header changed             : 0
    source changed             : 0
  passed to real compiler
    called for linking         : 0
    called for external debug  : 0
    called w/o source          : 0
    called w/ multiple sources : 0
    called w/ PCH              : 0

Failing tests do not cause CI failures

See https://ci.appveyor.com/project/frerich/clcache/build/job/l9tugke3vgrehcn8

There we have a test failing (testBackslashBeforeQuote) but the CI is still green. There must be some error code missing since the AppVeyor UI change

clcache.py run too many jobs

when the cl.exe with many source files, clcache.py make one process for one source files. I get a project that call cl.exe with more than 2000 source files, and in that case, clcache.py run more than 2000 processes and the memory run out.

"no source file found" with double backslash in the path

Hi!

I am trying to make clcache work with my cmake-based project. I have been able to actually make bypass an error in clcache just by removing \Zi flags that looks not to be supported.

Unfortunately, looks like there is another problem when using clcache in my project: it complains about being unable to find each single source file for actually caching it for later compilations. I get tons of "passed to the real compiler" hits in the clcache stats. The error is always the same: no source file found.

This is a sample of the kind of log I get from clcache:

C:\dev\clcache\dist\clcache.py Cannot cache invocation as ['/nologo', u'/c', u'/I"C:\dev\project\common\include"' /Fd"Bullet2FileLoader.dir\Debug\vc120.pdb" /Gd /TP /wd4201 /wd4512 /analyze- /errorReport:queue "C:\dev\project\src\Bullet3Serialize\Bullet2FileLoader\b3Serializer.cpp"']: no source file found

Any idea how to fix this issue or if there is actually an issue in clcache itself?

Response file (.rsp) referencing multiple files

Not really an issue of clcache, but how do I force msvc to produce single file rsp files or single file command line for cl.exe? Is it possible for msvc 2010?
Otherwise clcache is of no use to me, because I get only CallsWithMultipleSourceFiles.

Untitled

Add CI build badge to README

At AppVeyor, we can get a build status badge for the README, which shows the sanity of the code at first glance: Screenshot

The link target should be https://ci.appveyor.com/project/frerich/clcache/branch/master. If you share the SVG url, I could prepare a PR. Or you can just do it yourself.

Don't strip out /D flags before running the preprocessor

Creates a significantly wrong output. For debug builds with boost, it also fails the build (e.g. /RTC is on but /D_DEBUG isn't, causes boost/config/auto_link.hpp to emit #error, stopping the pre-processing)

Lock Timeout even with 60s

Before starting to work on the lockfile based locking again, I wanted to see how clcache has evolved since 3.0.3. Also I wanted to get some reference timing data for later comparison with another locking strategy and also wanted to use it on our internal main project.
We have a build driven by ninja (generated by cmake). I am testing on a 4 core VM so there should be at max 4 clcache instances running at the same time. I am using the 3.3.0 release. A complete build generates about 6500 cache entries.
During the build I get a Lock Timeout Error with the 10s default timeout value and even with a 60s timeout.
I then activated profiling to see where the time might be spent. But with profiling on I am not getting any timeouts. The profiling result of a completed run is attached. prof.txt
Any hints on how to diagnose this further? I started looking at the clcache code but it seams that much has changed since I last looked at the code.

vs2012 WindowsError: [Error 5] Access is denied

Hi,

I hvae
renamed cl.exe to cl_real.exe
renamed cl.exe.config to cl_real.exe.config
copied clcache.exe to cl.exe

I have set the enviroment variable
CLCACHE_DIR = C:\clcache
CLCACHE_CL = C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\cl_real.exe

Then i open Developer Command Prompt for VS2012 ,run cl , the output is
C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin>cl
Traceback (most recent call last):
File "", line 1094, in
File "", line 996, in main
File "", line 1033, in processCompileRequest
File "", line 706, in invokeRealCompiler
File "E:\PyInstaller-2.1\clcache\build\clcache\out00-PYZ.pyz\subprocess", line
486, in call
File "E:\PyInstaller-2.1\clcache\build\clcache\out00-PYZ.pyz\subprocess", line
672, in init
File "E:\PyInstaller-2.1\clcache\build\clcache\out00-PYZ.pyz\subprocess", line
882, in _execute_child
WindowsError: [Error 5] Access is denied

Please help me to solve the problem....

Convert all names to snake_case?

According to the PEP 8 style guide, function names in Python should be written in camel_case. The same is true for different kind or variable names. clcache historically uses camelCase.

I have no string opinion about that matter but there are some minor points for and against applying that rule

• most symbols in the standard library are snake_case
• pylint default configuration expects snake_case (probably because of PEP 8)
• experienced Python developers are often used to snake_case

on the other hand

• renaming requires some amount of work
• git blame is going to be spammed by renaming changes

I would not put readability on any list because I think both are equally good when you're used to.

clcache sometimes crash

Hi,
i'm running clcache, most of time it does the job, but sometimes it just crash with this output:
Traceback (most recent call last):
File "clcache.py", line 1093, in
File "clcache.py", line 995, in main
File "clcache.py", line 1060, in processCompileRequest
File "clcache.py", line 951, in processNoManifestMiss
File "clcache.py", line 883, in addObjectToCache
File "clcache.py", line 221, in setEntry
File "clcache.py", line 88, in enter
File "clcache.py", line 105, in acquire
main.ObjectCacheLockException: Error! WaitForSingleObject returns 258, last error 0

parseIncludesList regex only working for english locale

the english compiler output is "Note: including file", though the german language output is "Hinweis: Einlesen der Datei" ... not idea about non-european languages ...

cotire seems to have some code, which might help:
https://github.com/facebook/hhvm/blob/master/CMake/cotire.cmake#L898

Error when restoring cache object with CLCACHE_NODIRECT=1; output.txt missing.

I encountered some errors after setting CLCACHE_NODIRECT to 1.

17>  Traceback (most recent call last):
17>    File "clcache.py", line 1443, in <module>
17>    File "clcache.py", line 1328, in main
17>    File "clcache.py", line 1355, in processCompileRequest
17>    File "clcache.py", line 1431, in processNoDirect
17>    File "clcache.py", line 1193, in processCacheHit
17>    File "clcache.py", line 328, in cachedCompilerOutput
17>  FileNotFoundError: [Errno 2] No such file or directory: 'D:\\clcache\\objects\\98\\98a70d87d6a357b65abc721fabbe7640\\output.txt'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler source files: []
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler object file: .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Reusing cached object for key 98a70d87d6a357b65abc721fabbe7640 for output file .obj
17>  Failed to execute script clcache
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_original.exe'
17>  Traceback (most recent call last):
17>    File "clcache.py", line 1443, in <module>
17>    File "clcache.py", line 1328, in main
17>    File "clcache.py", line 1355, in processCompileRequest
17>    File "clcache.py", line 1431, in processNoDirect
17>    File "clcache.py", line 1193, in processCacheHit
17>    File "clcache.py", line 328, in cachedCompilerOutput
17>  FileNotFoundError: [Errno 2] No such file or directory: 'D:\\clcache\\objects\\77\\77d53a5273822136e5759776a3a3c9cb\\output.txt'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Arguments we care about: '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler source files: []
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler object file: .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Reusing cached object for key 77d53a5273822136e5759776a3a3c9cb for output file .obj
17>  Failed to execute script clcache
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_original.exe'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Arguments we care about: '[]'
17>  Traceback (most recent call last):
17>    File "clcache.py", line 1443, in <module>
17>    File "clcache.py", line 1328, in main
17>    File "clcache.py", line 1355, in processCompileRequest
17>    File "clcache.py", line 1431, in processNoDirect
17>    File "clcache.py", line 1193, in processCacheHit
17>    File "clcache.py", line 328, in cachedCompilerOutput
17>  FileNotFoundError: [Errno 2] No such file or directory: 'D:\\clcache\\objects\\f0\\f00f4fdb42aae261cc1b449917497c7c\\output.txt'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler source files: []
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler object file: .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Reusing cached object for key f00f4fdb42aae261cc1b449917497c7c for output file .obj
17>  Failed to execute script clcache
17>  
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Child: []'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Child: []'
17>  Traceback (most recent call last):
17>    File "clcache.py", line 1443, in <module>
17>    File "clcache.py", line 1328, in main
17>    File "clcache.py", line 1355, in processCompileRequest
17>    File "clcache.py", line 1431, in processNoDirect
17>    File "clcache.py", line 1193, in processCacheHit
17>    File "clcache.py", line 328, in cachedCompilerOutput
17>  FileNotFoundError: [Errno 2] No such file or directory: 'D:\\clcache\\objects\\46\\4698db611e6f2bee6314c804ab111bce\\output.txt'
17>  Failed to execute script clcache
17>  Traceback (most recent call last):
17>    File "clcache.py", line 1443, in <module>
17>    File "clcache.py", line 1328, in main
17>    File "clcache.py", line 1355, in processCompileRequest
17>    File "clcache.py", line 1431, in processNoDirect
17>    File "clcache.py", line 1193, in processCacheHit
17>    File "clcache.py", line 328, in cachedCompilerOutput
17>  FileNotFoundError: [Errno 2] No such file or directory: 'D:\\clcache\\objects\\91\\918f34a29c459335f40ad03c91bdad4c\\output.txt'
17>  Failed to execute script clcache
17>  Traceback (most recent call last):
17>    File "clcache.py", line 1443, in <module>
17>    File "clcache.py", line 1328, in main
17>    File "clcache.py", line 1355, in processCompileRequest
17>    File "clcache.py", line 1431, in processNoDirect
17>    File "clcache.py", line 1193, in processCacheHit
17>    File "clcache.py", line 328, in cachedCompilerOutput
17>  FileNotFoundError: [Errno 2] No such file or directory: 'D:\\clcache\\objects\\41\\41ccbefdf6a9a9e7bc30276932410f92\\output.txt'
17>  Failed to execute script clcache
17>  Traceback (most recent call last):
17>    File "clcache.py", line 1443, in <module>
17>    File "clcache.py", line 1328, in main
17>    File "clcache.py", line 1355, in processCompileRequest
17>    File "clcache.py", line 1431, in processNoDirect
17>    File "clcache.py", line 1193, in processCacheHit
17>    File "clcache.py", line 328, in cachedCompilerOutput
17>  FileNotFoundError: [Errno 2] No such file or directory: 'D:\\clcache\\objects\\11\\1150c14a794b5ab9ff6bdf9b8bf99738\\output.txt'
17>  Failed to execute script clcache
17>  Traceback (most recent call last):
17>    File "clcache.py", line 1443, in <module>
17>    File "clcache.py", line 1328, in main
17>    File "clcache.py", line 1355, in processCompileRequest
17>    File "clcache.py", line 1431, in processNoDirect
17>    File "clcache.py", line 1193, in processCacheHit
17>    File "clcache.py", line 328, in cachedCompilerOutput
17>  FileNotFoundError: [Errno 2] No such file or directory: 'D:\\clcache\\objects\\8a\\8a135e981b561ed3cc206b76f169756e\\output.txt'
17>  Failed to execute script clcache
17>  Traceback (most recent call last):
17>    File "clcache.py", line 1443, in <module>
17>    File "clcache.py", line 1328, in main
17>    File "clcache.py", line 1355, in processCompileRequest
17>    File "clcache.py", line 1431, in processNoDirect
17>    File "clcache.py", line 1193, in processCacheHit
17>    File "clcache.py", line 328, in cachedCompilerOutput
17>  FileNotFoundError: [Errno 2] No such file or directory: 'D:\\clcache\\objects\\78\\78ee10533f6188c37a57c46f0c7238db\\output.txt'
17>  Failed to execute script clcache
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_original.exe'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Arguments we care about: '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler source files: []'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler object file: .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Reusing cached object for key 4698db611e6f2bee6314c804ab111bce for output file .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_original.exe'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Arguments we care about: '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler source files: []'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler object file: .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Reusing cached object for key 918f34a29c459335f40ad03c91bdad4c for output file .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_original.exe'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Arguments we care about: '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler source files: []'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler object file: .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Reusing cached object for key 41ccbefdf6a9a9e7bc30276932410f92 for output file .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_original.exe'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Arguments we care about: '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler source files: []'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler object file: .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Reusing cached object for key 1150c14a794b5ab9ff6bdf9b8bf99738 for output file .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_original.exe'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Arguments we care about: '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler source files: []'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler object file: .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Reusing cached object for key 8a135e981b561ed3cc206b76f169756e for output file .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_original.exe'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Arguments we care about: '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[]'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler source files: []'
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler object file: .obj
17>  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Reusing cached object for key 78ee10533f6188c37a57c46f0c7238db for output file .obj
17>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets(341,5): error MSB6006: "CL.exe" exited with code -1.

I use Visual Studio 2013 with clcache 3.2.0.

Cannot open compiler generated file: '"Debug\\".obj' with double backslash in the path

User @sowsem found this error when compiling with /Fo"Debug\\":

ClCompile:
  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\CL.exe /c /nologo /W0 /WX- /Od /Oy- /D _MBCS /Gm- /EHsc /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Fo"Debug\\" /Fd"Debug\vc120.pdb" /Gd /TC /analyze- /errorReport:queue CMakeCCompilerId.c
  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_real.exe'
  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Parsing given commandline '['@C:\\Users\\davido\\AppData\\Local\\Temp\\tmpf22c581ab5514910a81a2a62e85395c1.rsp']'
  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Expanded commandline '[u'/c', u'/nologo', u'/W0', u'/WX-', u'/Od', u'/Oy-', u'/D', u'_MBCS', u'/Gm-', u'/EHsc', u'/RTC1', u'/MDd', u'/GS', u'/fp:precise', u'/Zc:wchar_t', u'/Zc:forScope', u'/Fo"Debug\\\\"', u'/Fd"Debug\\vc120.pdb"', u'/Gd', u'/TC', u'/analyze-', u'/errorReport:queue', u'CMakeCCompilerId.c']'
  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Compiler output file: 'Debug\\'
  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Invoking real compiler as 'C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl_real.exe /showIncludes /c /nologo /W0 /WX- /Od /Oy- /D _MBCS /Gm- /EHsc /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Fo"Debug\\" /Fd"Debug\vc120.pdb" /Gd /TC /analyze- /errorReport:queue CMakeCCompilerId.c'
  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Real compiler returned code 1
  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\clcache.py Finished. Exit code 1
  CMakeCCompilerId.c
C:\dev\projects\projectx-client\projects\projectx_win_tests\CMakeFiles\3.4.3\CompilerIdC\CMakeCCompilerId.c : fatal error C1083: Cannot open compiler generated file: '"Debug\\".obj': Invalid argument [C:\dev\projects\projectx-client\projects\projectx_win_tests\CMakeFiles\3.4.3\CompilerIdC\CompilerIdC.vcxproj]

from https://gist.github.com/sowsem/5080d43be4b32fea828a8f62cf8055ab

This revealed by a change in #105. The actual bug is that

        # Strip quotes around file names; seems to happen with source files
        # with spaces in their names specified via a response file generated
        # by Visual Studio.
        if outputFile.startswith('"') and outputFile.endswith('"'):
            outputFile = outputFile[1:-1]

does not happen in the correct code branch in analyzeCommandLine(). So /Fo"Debug\\" does not become output file Debug\CMakeCCompilerId.obj but "Debug\".

Nothing is cached by clcache

• Environment :
  • Visual Studio 2013 Express
  • Windows 7 64 pro
  • clcache master and 3.0.1, (tried both)
• Issue :
  • When i build a project, nothing is cached by clcache.
• Statistics :
  C:\Users\cosmo>"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl.exe" -s
  clcache statistics:
  current cache dir : C:\clcache
  cache size : 0 bytes
  maximum cache size : 1048576000 bytes
  cache entries : 0
  cache hits : 0
  cache misses : 0
  called for linking : 0
  called w/o sources : 4
  calls w/ multiple sources: 0
  calls w/ PCH: 0
  evicted misses : 0
  header changed misses : 0
  source changed misses : 0
• The clache folder is almost empty :
  Only two empty folders and a stats.txt file.
  c:\clcache>tree
  C:.
  ├───manifests
  └───objects

py2exe on windows 8 and Python 2.7.10 produces crashing exe

Running python setup.py py2exe lists the following output after compiling the bytecode, and produces a clcache.exe that crashes when executed. Any ideas how to debug this?

*** copy extensions ***
*** copy dlls ***
copying C:\Python27\lib\site-packages\py2exe\run.exe -> C:\clcache-master\dist\clcache.exe
Adding python27.dll as resource to C:\clcache-master\dist\clcache.exe
warning: py2exe: Version Info will not be included:
  could not parse version number '3.0.3-dev'

The following modules appear to be missing
['_scproxy', '_sysconfigdata', 'dummy.Process']

*** binary dependencies ***
Your executable(s) also depend on these dlls which are not included,
you may or may not need to distribute them.

Make sure you have the license if you distribute any of them, and
make sure you don't distribute files belonging to the operating system.

   OLEAUT32.dll - C:\Windows\system32\OLEAUT32.dll
   USER32.dll - C:\Windows\system32\USER32.dll
   SHELL32.dll - C:\Windows\system32\SHELL32.dll
   ole32.dll - C:\Windows\system32\ole32.dll
   ADVAPI32.dll - C:\Windows\system32\ADVAPI32.dll
   WS2_32.dll - C:\Windows\system32\WS2_32.dll
   GDI32.dll - C:\Windows\system32\GDI32.dll
   KERNEL32.dll - C:\Windows\system32\KERNEL32.dll

C:\clcache-master>dist\clcache.exe

Harmonize naming in source code, introduce dedicated types

We're being somewhat inconsistent with the naming in the clcache source code, which is quite confusing. We recently started introducing manifest sections as well as cache sections to permit implementing a finer-grained locking strategy, but that didn't exactly make things any easier.

This ticket is meant to serve as a discussion forum for where we may want to go. Here's a moonshot idea:

The birds-eye view is that a Cache is made of an ArtifactRepository in which some sort of 'key' is mapped to an CompilerArtifacts value. And that value then provides access to whatever we cache. For the sake of finer-grained locking, we may have ArtifactSections which roughly correspond to CacheSections. So the API may be something like (read :: as has type and a -> b as a function mapping a value of type a to a value of type b):

class Cache:
  artifactRepository :: ArtifactRepository
  manifestRepository :: ManifestRepository
  stats :: Statistics
  config :: Configuration

  clean :: void -> void

// Hmm, this class doesn't pull its own weight...
class ArtifactRepository:
    sections :: Generator<ArtifactSection>
    section :: Key -> ArtifactSection

// This class closely resembles a dictionary, maybe we could make good use of some Python-specific tricks such that we can write '<key> in <section>' or 'section[key]'.
class ArtifactSection:
    lock :: CacheLock
    hasArtifact :: Key -> Bool
    getArtifact :: Key -> CompilerArtifacts
    setArtifact :: (Key, CompilerArtifacts) -> void

class CompilerArtifacts:
    objectFilePath :: Path
    compilerStdout :: String
    compilerStderr :: String

class ManifestRepository:
  sections :: Generator<ManifestSection>
  section :: Key -> ManifestSection

class ManifestSection:
    lock :: CacheLock
    hasManifest :: Key -> Bool
    getManifest :: Key -> Manifest
    setManifest :: (Key, Manifest) -> void

class Configuration:
  // same as CacheConfiguration

class Statistics:
  // same as CacheStatistics

Not sure if it would work out, but I think there are two interesting observations here:

1. The *Repository classes don't really add much value. They just provide access to a sequence of sections (useful for operations which affect all sections, e.g. when cleaning the cache), and a shortcut for accessing a specific section. Right now, you can also clean both repositories independently - but is there a good reason for that other than just having a separate function for each? This could be achieved without having two classes, too...
2. The *Section classes are very similiar, and very dictionary-like. I wonder whether we really only need a generic CacheSection class which is somehow parametrised on the type of value it stores, and then that single class is used for both manifest as well as artifact sections.

compile would run slower with clcache, or?

I tried the following: compiling one of my .c files using cl.exe without clcache. It took 0.7 seconds. I then tried compiling again with the /e command line option. And again with the /ep command line option. These 'compiles' took 1.4 and 1.8 seconds respectively. Based on this evidence, how does clcache speed up the compile if running cl.exe with the /ep option causes cl.exe to run twice as slow? Have I missed the point somehow? Please note that I didn't actually use clcache for my little performance test... I was just testing the theory to see if clcache could be faster...

error on cache cleanup

after guarding the stats cache in my personal fork of clcache, i'm getting this error when the compiler cache is cleaned:

Traceback (most recent call last):
  File "clcache.py", line 1322, in <module>
  File "clcache.py", line 1224, in main
  File "clcache.py", line 1265, in processCompileRequest
  File "clcache.py", line 1316, in processNoDirect
  File "clcache.py", line 1071, in addObjectToCache
  File "clcache.py", line 176, in clean
WindowsError: [Error 3] The system cannot find the path specified: u'D:\\clcache\\objects\\00\\00e10ae9ecf9db5b0fea0c55fa252e67\\object'

Minimal Python version?

Hi everyone,

is there any official minimum Python version for this project?

If we could use Python 2.7+ would be good because it allows nice stiff like new string formatting or argparse.

Thanks.

cl.exe invocations with /Zi are not supported.

Hi all,

In our company, /Zi is enabled by default for our development builds.
Is there a way clcache (and/or our workflow) can be adapted so we can use it, or has any of the existing forks already done this?

Thanks a lot!
Broes

memcached support

I've been following ccache and there's an upcoming pull request to add support for caching on memcached https://github.com/jrosdahl/ccache/pull/58.
I think it's a great feature to add to clcache.

clcache does not work if /Fo argument contain a directory

When /Fo argument contain a directory I have this error:

1>------ Build started: Project: test_clcache, Configuration: Debug Win32 ------
1>Compiling...
1>test_clcache.cpp
1>C:\clcache\clcache.py Found real compiler binary at 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\cl.exe'
1>C:\clcache\clcache.py Parsing given commandline '['@d:\my_projects\test_clcache\Debug\RSP00000D27521588.rsp', '/nologo', '/errorReport:prompt']'
1>C:\clcache\clcache.py Expanded commandline '[u'/Od', u'/D', u'"WIN32"', u'/D', u'"_DEBUG"', u'/D', u'"_CONSOLE"', u'/D', u'"_UNICODE"', u'/D', u'"UNICODE"', u'/FD', u'/EHsc', u'/RTC1', u'/MDd', u'/Fo"Debug\"', u'/Fd"Debug\a.pdb"', u'/W3', u'/c', u'/ZI', u'/TP', u'.\test_clcache.cpp', '/nologo', '/errorReport:prompt']'
1>C:\clcache\clcache.py Compiler output file: 'Debug'
1>C:\clcache\clcache.py Reusing cached object for key ad9deaeb7eeabaebdcda4b719b9b57bbbd76e9d1 for output file Debug
1>c:\python27\lib\shutil.py:63: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
1> os.path.normcase(os.path.abspath(dst)))
1>Traceback (most recent call last):
1> File "C:\clcache\clcache.py", line 469, in
1> copyfile(cache.cachedObjectName(cachekey), outputFile)
1> File "c:\python27\lib\shutil.py", line 82, in copyfile
1> with open(dst, 'wb') as fdst:
1>IOError: [Errno 2] No such file or directory: u'Debug\'
1>Project : error PRJ0002 : Error result 1 returned from 'C:\clcache\cl.exe'.

It seems that we must strip quotes around file names before starting work with the passed directory.

Visual Studio does not invoke cl.bat

Hello!
I have visual studio 2008 SP1 installed. I did all that discribed in your readme file about installation.
But visual studio does not invoke cl.bat. I used process monitor and saw that the studio runs the compiler as "cl.exe", not just "cl". So I wrote a small program in c + +, which does the same thing as your cl.bat. And it worked !

include <stdlib.h>

include

include <Windows.h>

std::string GetExeDir()
{
char filename[MAX_PATH];
GetModuleFileNameA(GetModuleHandleA(NULL), filename, MAX_PATH);

char* pszFile = strrchr(filename, '\\');
pszFile++;    // Moves on from \

// Get path
char szPath[MAX_PATH] = "";
strncat(szPath, filename, pszFile - filename);

return szPath;

}

int main(int argc, char* argv[])
{
std::string cmd = "python ";
cmd += GetExeDir() + "\clcache.py";
for (int i = 1; i < argc; ++i)
{
cmd += " ";
cmd += argv[i];
}

// Execute command
return system(cmd.c_str());

}

Incremental builds with MSBuild/Visual Studio are broken

I think it problem related with #33, but a bit more complicated.
When I try incremental build, all solution fully rebuilds. I found it not happens if I comment body of PersistentJSONDict.save function. As I understand, MSBuild tracks writing a statistics file and mark projects in solution as outdated.
Affects only solutions with two or more projects with build dependencies between it!

Compiler output and stderr not taken into account for cache size computation

The size of the output.txt and stderr.txt files are not taken into account when computing the size of the artifacts repository. This allows the cache to be bigger than configured.

This is a minor issue as those files are not huge.