Electric Cloud Customer Summit 2011 by the Numbers

Earlier this month, Electric Cloud hosted the fourth annual Electric Cloud Customer Summit. By any measure it was a fantastic success, with more people, more content, and lots of enthusiastic and intelligent customers. I thought it would be fun to look at some statistics from this year’s event.

How many people showed up?

The most obvious metric is simply the count of attendees. In 2011, there were 146 attendees (excluding Electric Cloud employees). That’s literally double the number that showed up for the first summit in 2008:

This was the first summit for the majority of those present, but a significant minorty — nearly 25% — had been to at least one previous summit. Several are “Summit All-Stars”, having attended all four!

Who presented?

Another way to measure the growth of the summit is to look at the number of presentations each year, and the proportion of those that were given by customers or partners, rather than by Electric Cloud employees. In 2011, a healthy 40% of the presentations were given by customers and partners, including two panel sessions, and a keynote from GE about how Electric Cloud enabled the transition to agile development:

Where did they come from?

The vast majority of participants were from the United States, but several braved international travel to attend. Here are the countries represented:

Within the United States, 14 states were represented:

How many companies were represented?

This year’s summit was a fantastic place to network, with nearly 60 companies represented, across a wide range of industries. This tag cloud shows the industries, scaled by the number of people from each:

One thing that surprised me is the number of people sent by each company. I expected that most companies would send only one person, but in fact most companies sent at least two. Three companies sent ten or more!

When did attendees register?

I thought it might be interesting to see how far in advance people registered for the summit. It’s not surprising that there’s a spike the week before, although the magnitude of the jump is less than I expected. In fact, less than 25% of the registrations occurred the week before and the week of the summit:

Looking forward to 2012

I had a lot of fun at the 2011 Customer Summit. It was great to finally put faces to the names of people I’ve collaborated with, sometimes for years before meeting face-to-face. And it was a pleasure to see so many familiar faces as well. Here’s hoping the 2012 summit is just as fruitful.

One final thought: if you have any suggestions for additional statistics that might be interesting here, let me know in the comments.

What’s new in ElectricAccelerator 6.0

Last week Electric Cloud announced the release of ElectricAccelerator 6.0. The most exciting development in this version is the addition of ElectricAccelerator Developer Edition, which enables users to leverage the rapidly increasing horsepower on their multicore workstations to accelerate their builds, as well as the cluster agents we’ve always supported. Even better, Developer Edition allows you to use emake to accelerate builds even when disconnected from the network, with all of the benefits you’ve come to depend on: reliable parallel builds, accurate incrementals — all that good stuff.

But there’s much more to Accelerator 6.0 than just Developer Edition. Several improvements make this the most robust, secure and easy-to-use release ever. Here are the major new features:

Electrify command-line interface overhaul

electrify is the alternate front-end to the Accelerator cluster that enables users to accelerate non-make-based builds (like SCons and JAM) as well as general purpose cluster parallel computing tasks. Accelerator 5.0 was the first release to include electrify, but that version was relatively unpolished — a minimal viable product strategy that we hoped would allow us to prove the technology in the field and provide feedback to guide the technical direction of electrify over subsequent releases. As a result of that feedback we made significant improvements to both the functionality and performance of electrify in the 5.2 and 5.4 releases, but electrify still retained the “charm” of its original clumsy user interface. At long last, we’ve taken the first big step towards addressing that in 6.0, which incorporates a significantly streamlined electrify interface. I won’t bother showing the “old clunky way” here — if you’ve been using electrify you already know, and if you haven’t, I’d rather you never see it. But I’m very pleased to show you how easy it is to accelerate a SCons build with electrify in Accelerator 6.0:

electrify --emake-cm=eacm --electrify-remote=gcc:ld scons -j 8

There are still a few warts, but then, if we fixed everything what would people complain about?

Kerberos authentication

For environments with heightened security requirements, Accelerator 6.0 supports Kerberos authentication. This improvement ensures the identity of the individual running emake and the agents that participate in the build. That means that emake can be certain that any agents it connects to are legimiate agents, rather than trojans set up to capture potentially sensitive information from emake. At the same time, it means that the agents can be certain of the authenticity of the connection from emake, so malicious users cannot spoof the connection to the agent.

Note that at this time Accelerator only provides authentication of the connction between emake and an agent. It does not encrypt traffic once the connection is established.

Submake usage reporting

One thing that occassionally causes trouble for people switching to emake is the so-called submake stub problem. The issue arises when a build uses constructs like this:

	$(MAKE) -C sub foo.a && cp sub/foo.a ./foo.a

With emake, the recursive make invocation is not processed inline, as you might normally expect. Instead, to maximize parallel performance across all the recursive makes in the build, a make stub captures the invocation context, relays it to the top-level emake for incorporation into the overall build graph, and exits immediately. Because the cp call is part of the same command, it runs immediately after the stub, but since none of the work of the submake has actully executed yet, the cp will fail.

Correcting this requires a trivial makefile change. Instead of chaining the recursive make and the cp together in a single command, this rule can be rewritten to explicitly use two distinct commands. With this minor adjustment, emake is able to treat the cp as a separate job that logically occurs after the work in the recursive make:

	$(MAKE) -C sub foo.a
	cp sub/foo.a ./foo.a

Although it’s generally straightforward to remedy the problem, it’s not always easy to determine that a build has run into a submake stub problem. Thus in Accelerator 6.0 we added a feature that will make it easier to identify these situations: if you enable file- or lookup-level annotation detail, emake will record an explicit submake usage operation in the job that invoked the recursive make, so you can tell exactly when the submake occurs relative to other file usage in the job. In the example above, the usage log for the job will contain something like this:

<op type="submake" file="/home/ericm/build/sub"/>
<op type="lookup" file="/home/ericm/build/sub/foo.a" found="0"/>

The presence of file usage after the submake operation is a warning flag that the job may have a submake stub vulnerability.

Optionally enable directory read conflicts

Another long-standing thorn-in-the-side for users switching to emake is builds that depend on strict accuracy of directory listing operations. Briefly, although emake can ensure that directory listings are 100% correct throughout the build, for performance reasons emake deliberately ignores the directory read conflicts that would provide that assurance. For a thorough explanation of the problem and the rational for that design decision, see my previous post on Exceptions to conflict detection in ElectricMake. For now, suffice to say that Accelerator 6.0 includes a new command-line option that forces emake to honor directory read conflicts: –emake-readdir-conflicts=1.

Agent log rotation

Last, but certainly not least, after only 10 years Accelerator finally incorporates a feature that’s been standard on Unix operating systems probably since the dawn of the epoch: the Accelerator agent now monitors the size of its debug log, and automatically rolls over to a new logfile when the log gets too big. That’s sure to be welcome news to anybody that’s had to enable agent debug logging while working with our support team.

Looking forward

As you can see, there’s a lot of great improvements in Accelerator 6.0. As always, I’m tremendously proud of the effort my engineering team has put into the product. Of course, a product like Accelerator is never really done. We’re already working on the next release. Stay tuned for updates on what we’re doing next.

Accelerator 6.0 is available immediately for current customers — contact support@electric-cloud.com for details. New users contact sales@electric-cloud.com.

2011 Customer Summit “Conflicts” Handout

As promised, here’s the corrected handout to accompany the presentation I gave at the 2011 Electric Cloud Customer Summit, “Conflict Detection in ElectricMake”. This content is also available as a pair of blog articles:

  1. How ElectricMake guarantees reliable parallel builds
  2. Exceptions to conflict detection in ElectricMake

Thanks to everybody who came to the talk! I really enjoyed giving it and answering your questions. Looking forward to seeing you next year!

Exceptions to conflict detection in ElectricMake

In a previous article I covered the basic conflict detection algorithm in ElectricMake. It’s surprisingly simple, which is one of its strengths. But if ElectricMake strictly adhered to the simple definition of a conflict, many builds would be needlessly serialized, sapping performance. Over the years we’ve made a variety of tweaks to the core algorithm, adding support for special cases to improve performance. Here are some of those special cases.

Non-existence conflicts

One obvious enhancement is to ignore conflicts when the two versions are technically different, but effectively the same. The simplest example is when there are two versions of a file which both indicate non-existence, such as the initial version and the version created by job C in this chain for file foo:

Suppose that job D, which falls between C and E in serial order, runs before any other jobs finish. At runtime, D sees the initial version, but strictly speaking, if it had run in serial order it would have seen the version created by job C. But the two versions are functionally identical — both indicate that the file does not exist. From the perspective of the commands run in job D, there is no detectable difference in behavior regardless of which of these two versions was used. Therefore emake can safely ignore this conflict.

Directory creation conflicts

A common make idiom is mkdir -p $(dir $@) — that is, create the directory that will contain the output file, if it doesn’t already exist. This idiom is often used like so:

$(OUTDIR)/foo.o: foo.cpp
	@mkdir -p $(dir $@)
	@g++ -o $@ $^

Suppose that the directory does not exist when the build starts, and several jobs that employ this idiom start at the same time. At runtime they will each see the same filesystem state — namely, that the output directory does not exist. Each job will therefore create the directory. But in reality, had these jobs run serially, only the first job would have created the directory; the others would have seen the version created by the first job, and done nothing with the directory themselves. According to the simple definition of a conflict, all but the first (serial order) job would be considered in conflict. For builds without a history file expressing the dependency between the later jobs and the first, the performance impact would be disastrous.

Prior to Accelerator 5.4, there were two options for avoiding this performance hit: use a good history file, or arrange for the directories to be created before the build runs. Accelerator 5.4 introduced a refinement to the conflict detection algorithm which enables emake to suppress the conflict between jobs that both attempt to create the same directory, so even builds with no history file will not get conflicts in this scenario, without sacrificing correctness. (NB: you need not take special action to enjoy the benefits of this improvement).

Appending to files

Another surprisingly common idiom is to append error messages to a log file as the build proceeds:

$(OUTDIR)/foo.o: foo.cpp
	@g++ -o $@ $^ 2>> err.log

Implicitly, each append operation is dependent on the previous appends to the file — after all, how will you know which offset the new content should be written to if you don’t know how big the file was to begin with? In terms of file versions, you can imagine a naive implementation treating each append to the file as creating a complete new version of the file:

The problem of course is that you’ll get conflicts if you try to run all of these jobs in parallel. Suppose all three jobs, A, B and C start at the same time. They will each see the initial version, an empty file, but if run serially, only A would have seen that version. B would have seen the version created by A; C would have seen the version created by B.

This example is particularly interesting because emake cannot sort this out on its own: as long as the usage reported for err.log is the very generic “this file was modified, here’s the new content” message normally used for changes to the content of an existing file, emake has no choice but to declare conflicts and serialize these jobs. Fortunately, emake is not limited to that simple usage record. The EFS can detect that each modification is strictly appending to the file, with no regard to the prior contents, and include that detail in the usage report. Thus informed, emake can record fragments of the file, rather than the entire file content:

Since emake now knows that the jobs are not dependent on the prior content of the file, it need not declare conflicts between the jobs, even if they run in parallel. As emake commits the modifications from each job, it stitches the fragments together into a single file, with each fragment in the correct order relative to the other pieces.

Directory read conflicts

Directory read operations are interesting from the perspective of conflict detection. Consider: what does it mean to read a directory? The directory has no content of its own, not in the way that a file does. Instead, the “content” of a directory is the list of files in that directory. To check for conflicts on a directory read, emake must check whether the list of files that the reader job actually saw matches the list that it would have seen had it run in serial order — in essence, doing a simple conflict check on each of the files in the directory.

That’s conceptually easy to do, but the implications of doing so are significant: it means that emake will declare a conflict on the directory read anytime any other job creates or deletes any file in that directory. Compare that to reads on ordinary files: you only get a conflict if the read happens before a write operation on the same file. With directories, you can get a conflict for modifications to other files entirely.

This is particularly dangerous because many tools actually perform directory reads under-the-covers, and often those tools are not actually concerned with the complete directory contents. For example, a job that enumerates files matching *.obj in a directory is only interested in files ending with .obj. The creation of a file named foo.a in that directory should not affect the job at all.

Another nasty example comes from utilities that implement their own version of the getcwd() system call. If you’re going to roll your own version, the algorithm looks something like this:

  1. Let cwd = “”
  2. Let current = “.”
  3. Let parent = “./..”
  4. stat current to get its inode number.
  5. read parent until an entry matching that inode number is found.
  6. add the name from that entry to cwd
  7. Set current = parent.
  8. Set parent = parent + “/..”
  9. Repeat starting with step 4.

By following this algorithm the program can construct an absolute path for the current working directory. The problem is that the program has a read operation on every directory between the current directory and the root of the filesystem. If emake strictly adhered to conflict checking on directory reads, a job that used such a tool would be serialized against every job that created or deleted any file in any of those directories.

For this reason, emake deliberately ignores conflicts on directory read operations by default. Most of the time this is safe to do, surprisingly — often tools do not need a completely accurate list of the files in the directory. And in every case I’ve seen, even if the tool does require a perfectly correct list, the tool follows the directory read with reads of the files it finds. That means that you can ensure correct behavior by running the build one time with a single agent, to ensure the directory contents are correct when the job runs. That run will produce history based on the file reads, so subsequent builds can run with many agents and still produce correct results.

Starting with Accelerator 6.0, you can also use –emake-readdir-conflicts=1 to force emake to honor directory read conflicts.


Getting parallel builds that are fast is easy: just add -j to your make invocation. Getting parallel builds that are both fast and reliable is another story altogether. As you’ve seen, the core conflict detection algorithm in ElectricMake is simple, but after many years and hundreds of thousands of builds, we’ve enhanced that simple algorithm in a variety of special cases to provide even better performance. Future releases of ElectricAccelerator will include even more refinements to the algorithm.