What’s new in ElectricAccelerator 6.0

Last week Electric Cloud announced the release of ElectricAccelerator 6.0. The most exciting development in this version is the addition of ElectricAccelerator Developer Edition, which enables users to leverage the rapidly increasing horsepower on their multicore workstations to accelerate their builds, as well as the cluster agents we’ve always supported. Even better, Developer Edition allows you to use emake to accelerate builds even when disconnected from the network, with all of the benefits you’ve come to depend on: reliable parallel builds, accurate incrementals — all that good stuff.

But there’s much more to Accelerator 6.0 than just Developer Edition. Several improvements make this the most robust, secure and easy-to-use release ever. Here are the major new features:

Electrify command-line interface overhaul

electrify is the alternate front-end to the Accelerator cluster that enables users to accelerate non-make-based builds (like SCons and JAM) as well as general purpose cluster parallel computing tasks. Accelerator 5.0 was the first release to include electrify, but that version was relatively unpolished — a minimal viable product strategy that we hoped would allow us to prove the technology in the field and provide feedback to guide the technical direction of electrify over subsequent releases. As a result of that feedback we made significant improvements to both the functionality and performance of electrify in the 5.2 and 5.4 releases, but electrify still retained the “charm” of its original clumsy user interface. At long last, we’ve taken the first big step towards addressing that in 6.0, which incorporates a significantly streamlined electrify interface. I won’t bother showing the “old clunky way” here — if you’ve been using electrify you already know, and if you haven’t, I’d rather you never see it. But I’m very pleased to show you how easy it is to accelerate a SCons build with electrify in Accelerator 6.0:

electrify --emake-cm=eacm --electrify-remote=gcc:ld scons -j 8

There are still a few warts, but then, if we fixed everything what would people complain about?

Kerberos authentication

For environments with heightened security requirements, Accelerator 6.0 supports Kerberos authentication. This improvement ensures the identity of the individual running emake and the agents that participate in the build. That means that emake can be certain that any agents it connects to are legimiate agents, rather than trojans set up to capture potentially sensitive information from emake. At the same time, it means that the agents can be certain of the authenticity of the connection from emake, so malicious users cannot spoof the connection to the agent.

Note that at this time Accelerator only provides authentication of the connction between emake and an agent. It does not encrypt traffic once the connection is established.

Submake usage reporting

One thing that occassionally causes trouble for people switching to emake is the so-called submake stub problem. The issue arises when a build uses constructs like this:

foo.a:
	$(MAKE) -C sub foo.a && cp sub/foo.a ./foo.a

With emake, the recursive make invocation is not processed inline, as you might normally expect. Instead, to maximize parallel performance across all the recursive makes in the build, a make stub captures the invocation context, relays it to the top-level emake for incorporation into the overall build graph, and exits immediately. Because the cp call is part of the same command, it runs immediately after the stub, but since none of the work of the submake has actully executed yet, the cp will fail.

Correcting this requires a trivial makefile change. Instead of chaining the recursive make and the cp together in a single command, this rule can be rewritten to explicitly use two distinct commands. With this minor adjustment, emake is able to treat the cp as a separate job that logically occurs after the work in the recursive make:

foo.a:
	$(MAKE) -C sub foo.a
	cp sub/foo.a ./foo.a

Although it’s generally straightforward to remedy the problem, it’s not always easy to determine that a build has run into a submake stub problem. Thus in Accelerator 6.0 we added a feature that will make it easier to identify these situations: if you enable file- or lookup-level annotation detail, emake will record an explicit submake usage operation in the job that invoked the recursive make, so you can tell exactly when the submake occurs relative to other file usage in the job. In the example above, the usage log for the job will contain something like this:

<op type="submake" file="/home/ericm/build/sub"/>
<op type="lookup" file="/home/ericm/build/sub/foo.a" found="0"/>

The presence of file usage after the submake operation is a warning flag that the job may have a submake stub vulnerability.

Optionally enable directory read conflicts

Another long-standing thorn-in-the-side for users switching to emake is builds that depend on strict accuracy of directory listing operations. Briefly, although emake can ensure that directory listings are 100% correct throughout the build, for performance reasons emake deliberately ignores the directory read conflicts that would provide that assurance. For a thorough explanation of the problem and the rational for that design decision, see my previous post on Exceptions to conflict detection in ElectricMake. For now, suffice to say that Accelerator 6.0 includes a new command-line option that forces emake to honor directory read conflicts: –emake-readdir-conflicts=1.

Agent log rotation

Last, but certainly not least, after only 10 years Accelerator finally incorporates a feature that’s been standard on Unix operating systems probably since the dawn of the epoch: the Accelerator agent now monitors the size of its debug log, and automatically rolls over to a new logfile when the log gets too big. That’s sure to be welcome news to anybody that’s had to enable agent debug logging while working with our support team.

Looking forward

As you can see, there’s a lot of great improvements in Accelerator 6.0. As always, I’m tremendously proud of the effort my engineering team has put into the product. Of course, a product like Accelerator is never really done. We’re already working on the next release. Stay tuned for updates on what we’re doing next.

Accelerator 6.0 is available immediately for current customers — contact support@electric-cloud.com for details. New users contact sales@electric-cloud.com.