What’s new in ElectricAccelerator 9.0?

Just a couple months ago, in October 2016, we released ElectricAccelerator 9.0. This version includes some really exciting new functionality and unlocks even more amazing performance than ever before. For the first time since 2008 we added support for a new build tool: ninja, an ultra-fast new make-like build tool and the workhorse at the center of the build for both chromium and Android (yes, that Android). And we’ve continued to expand the JobCache feature — a generalization of the parse avoidance feature introduced in Accelerator 7.0. With Accelerator 9.0 you can cache more types of work, including GCC/G++ compiles, clang compiles, Microsoft cl compiles, javac and javadoc, and Google’s new Jack compiler for Java code. Even better, you can share cached results with other developers to amplify the gains across an entire team. Read on for details.

Ninja emulation

Accelerator 9.0 introduces support for the ninja-based builds. Ninja is a very interesting build tool: conceptually similar to make, but radically simplified (at least so far!). Gone are things like built-in functions, pattern rules, vpath, conditional directives, and all the other things that make it hard to parse and evaluate makefiles quickly. This enables the ninja parser to evaluate “ninja files” unbelievably quickly, but at the cost of making ninja files verbose and ill-suited for creation by hand. Instead, ninja files are typically generated from some other process, such as CMake. The benefit to the end user then is extremely fast incremental builds: for example, in Android 6.0, using the original make-based build system, a no-touch build could take as much as a minute to run even though there’s no work to be done. In Android 7.0, using the new ninja-based build system, the same build can be completed in about 5 seconds!

ElectricAccelerator’s emulation of ninja is, I think, remarkably anticlimactic: to execute a ninja build, simply invoke emake –emake-emulation=ninja. That’s it. Here’s a very simple “Hello, world!” ninja file:

And the result of running this with emake –emake-emulation=ninja:

As I said, it’s utterly uninteresting, which, quixotically, makes it very interesting: the integration is seamless and it “just works”. Even better, by running your ninja build with ElectricAccelerator you automatically and instantly take advantage of all the advanced acceleration and correctness features you’ve come to love about Accelerator: conflict detection, history, schedule optimization, annotation, even jobcache. It all just works.

JobCache Enhancements

In Accelerator 7.0 we introduced parse avoidance, a mechanism for caching the result of makefile parsing in one build in order to accelerate subsequent builds. Once we had shown that this type of caching could dramatically improve build performance we refactored the code behind parse avoidance to create a general purpose caching framework dubbed JobCache and in subsequent releases we’ve steadily expanded the types of work to which jobcache can be applied:

• Accelerator 7.1: jobcache for Javadoc generation
• Accelerator 8.0: jobcache for C/C++ compiles using clang/gcc/g++ (comparable to, but better than, ccache)
• Accelerator 8.1: jobcache for C/C++ compiles using Microsoft cl

In Accelerator 9.0 we’ve expanded the reach of jobcache in two ways. First, we added support for caching javac and Jack compiles. Next, we added shared jobcache, which enables a team of developers to leverage jobcache collectively and reliably, eliminating redundant work across the entire team.

With shared jobcache, the team designates a “blessed” or “golden” build process to populate the cache — typically the nightly or continuous integration builds. This build simply uses jobcache as normal, using –emake-assetdir to specify a location on a shared filesystem to host the cache. Then, each developer explicitly requests to use the shared cache by adding –emake-shared-assetdir to the command-line when they invoke emake, specifying the same location. Once enabled, emake uses both the shared cache and the private cache during the build. For each job that uses jobcache:

1. Check the shared jobcache for a matching entry.
   1. If a match is found in the shared jobcache, use it. Done!
   2. If a match is not found in the shared jobcache, continue.
2. Check the private jobcache for a matching entry.
   1. If a match is found in the private jobcache, use it. Done!
   2. If a match is not found in the private jobcache, continue.
3. Run the job as normal
4. Save the result to the private cache

Note that the shared cache is never written to by the developers’ builds: updates are only saved in the private cache. In this way we can ensure that developers’ builds to not litter the shared cache with one-off or user-specific cache entries. Typically we expect that developers will see very good cache hit rates against the shared cache, perhaps 95% or better, since each developer modifies only a small fraction of the total source code at once. Thus shared jobcache multiples the savings from jobcache by the size of the team.

Dynamic file patching

The final feature of interest in Accelerator 9.0 is dynamic file patching. This is a mechanism by which emake can patch files on the fly as they are referenced during the build, based on the name, size and MD5 checksum of the original. This feature enables users to tweak build scripts or makefiles in order to improve performance or compatibility with Accelerator — critical in environments where there is limited ability to modify the original files directly.

Looking forward to 9.1

Accelerator 9.0 contains some really tremendous new features: the first new build tool emulation in almost a decade; shared jobcache; on-the-fly patching for those challenging environments where no other option will do. But as always, my eye is already on the next horizon: Accelerator 9.1. We have some big plans relating to performance and ease-of-use. It will require a lot of hard work but I think we have the right team to do it. Stay tuned.

Accelerator 9.0 is available immediately for existing customers — support@electric-cloud.com to get the bits. New users can download ElectricAccelerator Huddle to take it for a test drive, or contact sales@electric-cloud.com for an evaluation of the enterprise edition.