sccache fails to compile Qt: hash collision between precompiled headers and object files

[jimis]

sccache fails to compile Qt: hash collision between precompiled headers and object files

[Xuanwo]

Interesting. We already included langauge in the hash calculation:

sccache/src/compiler/c.rs

Lines 717 to 750 in 7074753

	pub fn hash_key(
	compiler_digest: &str,
	language: Language,
	arguments: &[OsString],
	extra_hashes: &[String],
	env_vars: &[(OsString, OsString)],
	preprocessor_output: &[u8],
	plusplus: bool,
	) -> String {
	// If you change any of the inputs to the hash, you should change `CACHE_VERSION`.
	let mut m = Digest::new();
	m.update(compiler_digest.as_bytes());
	// clang and clang++ have different behavior despite being byte-for-byte identical binaries, so
	// we have to incorporate that into the hash as well.
	m.update(&[plusplus as u8]);
	m.update(CACHE_VERSION);
	m.update(language.as_str().as_bytes());
	for arg in arguments {
	arg.hash(&mut HashToDigest { digest: &mut m });
	}
	for hash in extra_hashes {
	m.update(hash.as_bytes());
	}
	for &(ref var, ref val) in env_vars.iter() {
	if CACHED_ENV_VARS.contains(var.as_os_str()) {
	var.hash(&mut HashToDigest { digest: &mut m });
	m.update(&b"="[..]);
	val.hash(&mut HashToDigest { digest: &mut m });
	}
	}
	m.update(preprocessor_output);
	m.finish()
	}

[jimis]

Isn't language the input language? Both input files are C++ code. The different is in the output format.

According to my experiments, it's the parameter -x c++-header that makes GCC generate GCH file.

[jimis]

GCC's logic that determines the output format is a bit complicated, and depends on the extension of the main input file and on the -x option. According to GCC docs:

-x language

Specify explicitly the language for the following input files (rather than letting the compiler choose a default based on the file name suffix). This option applies to all following input files until the next -x option.

After experimenting, here is GCC's behaviour, without sccache involved:

• gcc -o blah -c cmake_pch.hxx.cxx generates object code
• gcc -o blah -c cmake_pch.hxx.hxx generates precompiled header file
• gcc -x c++-header -o blah -c whatever_filename_and_extension_you_like generates precompiled header file

I guess Sccache doesn't take -x into consideration?
EDIT: scratch that, apparently it does:

sccache/src/compiler/gcc.rs

Lines 609 to 620 in 7074753

	fn language_to_gcc_arg(lang: Language) -> Option<&'static str> {
	match lang {
	Language::C => Some("c"),
	Language::CHeader => Some("c-header"),
	Language::Cxx => Some("c++"),
	Language::CxxHeader => Some("c++-header"),
	Language::ObjectiveC => Some("objective-c"),
	Language::ObjectiveCxx => Some("objective-c++"),
	Language::Cuda => Some("cu"),
	Language::GenericHeader => None, // Let the compiler decide
	}
	}

[jimis]

Another thing to note is that all the .cpp files that get mis-compiled are empty i.e. contain only comments (class documentation). And the cmake_pch.hxx.cxx file is also empty, contains only a single comment line.

So here my current theory:

1. Sccache, for all command lines, does pre-compilation (-E) first, with output to stdout
2. GCC, for all the offending files, when fed the -E parameter, outputs the same preprocessed source regardless of the -x argument it has on the command line
3. Sccache follows with a "local compile" step, which is kind of opaque. All I can see in the (local) server logs, for both the PCH generation and the object code generation steps, is:

   [2023-08-01T16:20:57Z TRACE sccache::compiler::gcc] compile
   [2023-08-01T16:20:57Z DEBUG sccache::compiler::compiler] [cmake_pch.hxx.gch]: Compiling locally
   

4. The result of "compile" is different based on the -x argument. But sccache stores them under the same hash key.
   • Indeed for each case I cleared the cache and re-run the compilation. The result file was correct. Running the same command with populated cache resulted to a possibly wrong output file, because of the hash collision.