Passing the ISUCON6 qualifiers with C++

Under the team name "Anago," @iwiwi, @zuisou, and @imos—the three of us—entered the ISUCON6 qualifiers, and luckily we managed to advance in 3rd place on Day 1. @iwiwi had been saying, "I want to enter ISUCON in C++!", and wanting to back him up with everything we had is what got us to participate.

What we did before the day of the contest

It is obvious that writing everything in C++ would be the fastest, but C++ is not only unavailable as a reference implementation—it is also a language not designed for writing HTTP servers, so preparation was a struggle. For writing an HTTP server in C++, the options we could think of were to write a plugin for an existing web server (e.g., nginx, Apache, H2O), to implement it as FastCGI, or to write it fully from scratch. Because we wanted to avoid inter-process communication and go blazingly fast, we wrote and open-sourced an HTTP server called nineserver in advance. The philosophy of nineserver is concisely shown by sql.cc:

make it so that you do not need to touch the same file, so that development does not conflict even when working together;
provide functions specific to SQL and the web (URL encoding, HTML escaping) and the like; and
give up on reimplementing everything, so that you can implement things little by little.

Writing an HTTP server fully from scratch had a lot to learn and was extremely fun. The thread-pool approach did not deliver enough speed, so we changed it to a design where all threads wait on accept; but that revealed a problem where threads become exhausted when connections are not closed, so in the end it became a design where half the threads wait on accept and the remaining threads wait on poll. Because we repeatedly added unforeseen changes, the code became messy, so someday I would like to rewrite it.

What we did on the day

First, we set up an environment where C++ could be built. While falling into the trap of being unable to install Java 9 via apt-get install, we installed Java 8 instead. We changed the port of the initial implementation, set up a configuration like nginx → C++ → nginx → PHP, and ran profiling. The first benchmark score was not very stable—occasionally it even returned 0 points—but I think it was around 2000 points.

Digging the timeline out of the commit log, it went something like this.

Around 10:30 … The first profiling finished, and each of us started reading the code.
Around 11:30 … Made it possible to modify the reference implementation little by little in the repository (imos)
Around 12:00 … Installed phpMyAdmin (imos)
13:07 … SQL tuning (zuisou)
13:54 … Stored sessions in cookies instead of storage (imos)
14:22 … Also stored user_name in the session (imos)
14:28 … Fully implemented /stars in C++ (iwiwi)
16:55 … Finished implementing the / and /keywords/* caches (zuisou)
17:21 … Finished implementing HTMLify in C++ using a trie (iwiwi)
17:37 … Completely eliminated the PHP HTMLify (iwiwi)
17:46 … Fully implemented /keywords/* in C++ (imos)

We solved persistence by keeping the INSERTs into SQL, making it a design where data is read from SQL on first startup and when /initialize is called. Because we could not get systemd to work well, the auto-start mechanism ended up being a design where cron launches it every minute (from the second time onward the port cannot be opened so it automatically exits, which prevents multiple instances from starting).

Figure: The score situation at final submission

Reflections

We had trouble with the final submission, and although we had originally intended to remove the frontmost nginx and disable debug mode, we could not find the time. Setting up an environment where the reference implementation can be developed in the repository took up roughly the first 2 hours, so that is another point we would like to improve. I feel we could have fully implemented / in C++ with just about 30 more minutes to spare, so I have a slight lingering regret on that point, but since we were able to advance through the qualifiers in 3rd place, I am satisfied.