Cache riml_includes between compile_files

[luke-gru]

Cache riml_includes between compile_files

[luke-gru]

This is a great idea, thanks! Also, it's easy to implement 😄. I am going to have to wrap a mutex around the cache though, with the way things work now.

How much faster is compilation right now with Riml.compile_files vs shelling out? I'm curious because it's compiling each file that's given to Riml.compile_files in its own thread, and if it's just 2 files then thread creation overhead may negate any speed benefits. Regardless, most of the speed benefits probably come from not creating new shells.

I'm also going to use yard to document all public API methods like this, to make it easier to use Riml outside of a command-line setting like you're doing here.

I'll let you know when this is ready :)

[dsawardekar]

How much faster is compilation right now with Riml.compile_files vs shelling out?

Quite a bit. It eliminates the boot time of ruby. And it's saving is proportional to the size of the suite. 😄

I'm curious because it's compiling each file that's given to Riml.compile_files in its own thread, and if it's just 2 files then thread creation overhead may negate any speed benefits.

I suspect I'm not seeing this as I am passing in one file at a time to compile_files. I have a Rake FileList rather than an array, so looping over it with each.

[dsawardekar]

I noticed a boost in compile times, Looks like this is working in master?

[luke-gru]

Hey @dsawardekar, this is indeed working in master, I'm glad you noticed a speed increase 😄
I'll make a release tomorrow off, and this will be included.

Thanks again for the suggestion.

[dsawardekar]

Thanks. I will update Speckle, this is an awesome update! :)

[dsawardekar]

@luke-gru About this update, It shaved about 20 seconds of the build time of Portkey! So this was a really nice update.

Not trying to be greedy, But I think you can do even better! The time taken for a complete compile of Portkey is about 5 seconds. About 1 second for plugin.riml and 4 seconds for portkey.riml which is the autoloaded main application.

Yet the whole test suite compilation takes about 40-50 seconds. Some of the tests include the entire app with portkey_inc.riml which a file that includes all the individual classes. It still feels like it is compiling the individual includes.

In contrast running the test suite in Speckle zips by in under 1 second, compiling about 10+ files. Speckle does not have any nested includes.

Does the the riml_include cache take into account nested includes?

[luke-gru]

Hmm, this is interesting, as I suspected it did this already, but I didn't benchmark anything at all, so I kind of tested it the lazy way.

If you take a look at https://github.com/luke-gru/riml/blob/master/test/integration/riml_commands/compiler_test.rb#L287, the mocked cache object I'm using during testing is apparently called with the nested includes, but something must be wrong if it's taking so long to compile.

I'll take a look at this, and make sure to add some benchmarks to the testing process, as the compile times are a little worrying right now, 40-50 seconds is pretty bad. I'm sure we can do better than that 😄

EDIT: I didn't know about Portkey! Looks awesome, I'm going to check it out.

[dsawardekar]

I'll take a look at this, and make sure to add some benchmarks to the testing process, as the compile times are a little worrying right now, 40-50 seconds is pretty bad. I'm sure we can do better than that 😄

Would assembly after compilation take too much times? If you run rake in the Portkey project you'll see it will start off pretty fast with the initial files, then when it reaches the controllers it's slows down. These are the tests that include portkey_inc.riml.

You can even see this isolated with,

$ speckle spec/controllers/commands/alternate_file_command_spec.riml spec/controllers/commands/related_file_command_spec.riml

These are E2E tests. Both these specs are very similar in size. Compiling the alternate command should compile the entire app. The related command should then be just a quick cache lookup + the tests in the file itself which is only a handful of methods.

The related command still takes a 2-3 seconds to compile. Adding a number of these commands is what is adding up to 40+ seconds in the whole suite.

[dsawardekar]

@luke-gru Compile times now down to 12 seconds in Portkey in master. Awesome work! Are you certain there were no sleep(10) calls in there? 😉

[luke-gru]

haha, thanks. No sleep calls 😄, it turns out it was still doing lots of unnecessary ATS rewrites though to pick up classes during riml_include. I made the caching better, and then to top it off there was some low-hanging fruit optimizations that I made regarding Riml.source_path and Riml.include_path caching as well. Before, whenever the compiler hit a riml_source or riml_include line, it would go over each directory in the path and check the filesystem to see if the files existed. All the filenames in both paths are now cached when they're set, so this saves some time too.

[dsawardekar]

Neat stuff. I'm glad I kept asking for more. At 12 seconds it's around the same time it takes the full suite to run. Can't ask for more now! :)