Why Go Sometimes Fails

I've been using Go since v0.7, at numerous companies. Go is a huge success! Go mostly works as advertised which is refreshing and reassuring. Specifically:

• Go isn't weird. People understand Go. People understand very well written Go. People understand too-clever Go. People understand junk Go.
• Go delivers. Go is pretty fast. Go is pretty good about memory. Go code doesn't blow up with a segfault because you've done something no one else has ever done.
• Go is popular. This matters because popularity delivers community, tools support, a good job pool, and an easy sell into new teams.

I also see Go projects failing. I've found two underlying reasons that seem to manifest again and again:

1. Expectations of Magic from Go's GC

For most small and mid-sized projects, Go's GC is indeed magical. In many trivial cases, Go's GC keeps memory use quite low while not imposing a performance penalty. Most importantly, Go's GC delivers without imposing stylistic limits on how people code with it.

That said, Go's GC seems to be so good in trivial cases that developers stop thinking about memory entirely. Most developers do not perform suitable load tests at useful duration, so instances of resource exhaustion only manifest in production. This is not a failure of Go's GC, which is doing exactly what is advertised - releasing memory when references to it in code have left scope. But, if you are a lazy developer who likes to write long-running giant programs with lots of state, then Go's GC can't help you!

A typical response to memory issues is to fire up pprof and stare at the pretty diagrams it makes. Using a profiler is supposed to be something good developers do...but pprof isn't intended to debug your design. If you design a program to use a lot of memory, it will! Mostly, I find people using pprof are just wasting time and delaying the inevitable refactoring of overly-complex code.

2. The Irrational Need To Use All Concurrency Features

Programmers like to flex. A prevailing theme with Go developers is the need to demonstrate aptitude to one's peers by writing Go code that employs concurrency features pointlessly. Unfortunately, this means that any sufficiently large Go program will end up resembling a poorly specified, cooperatively scheduled operating system.

This is a variation on a recurring theme in software development: it is the intermediate developer who has the greatest potential for harm. Junior developers don't over-complicate things because they don't know how to. Senior developers understand that making code complex just to beat your chest is not beneficial. It is the intermediate developer who will make sure every Go concurrency feature is put to use, relevant or not!

I highlight concurrency features here apart from any other advanced feature an intermediate developer may misuse because the types of bugs that result from concurrent code are especially awful; concurrency bugs can be difficult to reproduce and concurrency can dramatically amplify resource exhaustion issues. Furthermore, many developers misusing concurrency features fail to understand that in many cases, core libraries are already safely providing for concurrent use and don't require another layer of abstractions in project code.

The good news here is that senior Go developers can continue to make good money "rescuing" troubled Go projects by removing pointless concurrency!

last update 2022-12-05