Spanner's commit wait caps throughput only for serialized writes

Spanner promises external consistency: if transaction T1 commits before T2 starts, T1 gets the smaller commit timestamp [1]. Its clocks know the current time only to within a few milliseconds, so it adds commit wait: the commit takes a timestamp no earlier than the latest possible current time and stays invisible until that time has definitely passed. Every read-write transaction pays it, since Spanner can't see which later transactions depend on this one; that dependency can run through a user who saw the acknowledgment.

A wait on every write sounds like a ceiling on the write rate. Little's law says otherwise. In a stable system, L=λWL = \lambda W: the number of writes in flight equals throughput times the time each spends inside, so throughput is λ=L/W\lambda = L/W. Commit wait raises WW, but a waiting write holds nothing except its locks, so more writes overlap, LL rises with WW, and throughput stays wherever the real bottleneck puts it. In the paper's benchmark on a single replica, commit wait raised write latency from 9.4 to 14.4 ms while throughput held at about 4,000 writes per second.

The same formula says where the wait does cost rate: where writes are serialized, so only one is in flight and LL can't grow.

Commit wait on independent and serialized writesTwo timelines over 100 ms. Independent writes arrive every 5 ms, two in flight at a time; adding a 5 ms commit wait to each 9.4 ms write puts three in flight, and they still complete at 200 per second. Serialized writes start only when the previous one commits; the wait stretches each from 9.4 to 14.4 ms, cutting the rate from 106 to 69 per second.Independent writesa new one every 5 msno wait200/swith wait200/sSerialized writeseach starts when the last commitsno wait106/swith wait69/s050100 ms
  • write (9.4 ms)
  • commit wait (5 ms)
The same wait on every write. Independent writes overlap more and keep their rate; serialized writes can't overlap, so each wait delays everything behind it.

A contended key serializes writes on the server. Spanner uses two-phase locking and releases write locks only after commit wait ends and the commit is applied, so the lock is held through the wait. The key's write rate is at most one over the lock hold time, and commit wait lengthens the hold.

A causal chain serializes them in the client: each write starts only after the previous one is acknowledged. That was my objection, and it holds; the chain runs at one write per commit latency. Its fix is on the client side: writes known together can go into one transaction, even across shards, and pay the wait once.

References

  1. Spanner: Google's Globally-Distributed Database [PDF]
    Corbett, J. C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J. J., Ghemawat, S., Heydon, A., Hochschild, P., Hsieh, W., Kanthak, S., Kleiman, E., Larus, J. C., Rastogi, R., Srikumar, R. and Vanhoucke, V., 2012. Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI).