It's sad - I spent a good 12 years writing Scala every day and it was the ideal language for my brain. Until it wasn't - sbt got too complex for it's own good, everything became "very smart" developers over-using implicit conversions, you couldn't find a project that wasn't an opinion war on cats vs whatever. It collapsed on the weight of it's own smugness.
go and kotlin aren't it, gleam scratches the itch but I can't justify writing code that would impossible to hire for.
Every job I've has has used different languages so I don't really understand the need to find a dev for a specific language. I went from network firmware in C to banking application servers in Scala and it took like 2 weeks to ramp up. Not a big deal. Now I write lower level networking stuff again in Go, which seems like its just worse than e.g. C-with-templates (and occasional classes) style C++ so I don't really understand why people like it.
I think it used to be common to just look for smart people and assume they can run with whatever stack. Wasn't that the point of abstract algorithm questions etc. (basically an IQ test)?
>I think it used to be common to just look for smart people and assume they can run with whatever stack. Wasn't that the point of abstract algorithm questions etc. (basically an IQ test)?
Lots of companies where software isn't the focus see it as a cost center, so they'd prefer to hire lower-IQ specialists instead of higher-IQ generalists, because the latter are more expensive/have more options.
Java is almost always significantly faster than Go because the Go runtime does a poor job of exploiting large memory page, doesn't support text-on-huge-pages, and barely supports profile-guided optimization. With HotSpot you get all of this and more for free. Go is fine but Java is peak.
If it’s rare and the impact is low, then it’s not getting prioritized. It doesn’t matter how much time passes if you decide not to spend time investigating.
I’ve been living in a bubble with my .NET day-job, where debugging/tracing/postmortems are a breeze. Compare with, say, a CORBA or DCOM system, deployed to prod with uber-optimized binaries without any debugging-symbols.
So it’s not that I haven’t worked on large-scale, complex legacy systems - but that I haven’t worked on any large-scale, complex legacy systems written in languages bereft of runtime reflection and verbose error reporting.
—————
It’s also possible that the bug was never found because its impact was so minimal: e.g. 1 crash per year, each causing 3 minutes’ downtime in a noncritical system: that’s something that will never get investigated fully.
Sorry but I don't buy it, any crash can be found and fix by one or more humans, if it was not it's either they're incompetent ( I doubt that ) or they did not beleive it was important enough to fix.
A crash is actually the easiest kind of problem to fix since you have a crash. It means stacktrace, core dump, kernel error etc..
"This Raft library is stable and feature complete. As of 2016, it is the most widely used Raft library in production, serving tens of thousands clusters each day. It powers distributed systems such as etcd, Kubernetes, Docker Swarm, Cloud Foundry Diego, CockroachDB, TiDB, Project Calico, Flannel, Hyperledger and more."
etcd has had numerous liveness and safety bugs, with one happening as recently as December of 2025. Would you consider that a correct implementation?
You may be interested in knowing that the largest managed Kubernetes service in the world (AWS EKS) ripped out etcd for in favor of their homegrown consensus service for large scale EKS clusters: https://aws.amazon.com/blogs/containers/under-the-hood-amazo...
etcd is some of the most amateur code I've ever seen, despite being one of the oldest and presumably most mature "infrastructure" projects written in Go.
goBGP is arguably even worse.
I don't have a third place in mind that's even worth mentioning relative to these two.
No they won't. They are generally faster in throughput than any non-gc application that isn't heavily hand optimized. Their problems are higher memory usage and unpredictable latency, not speed.
Just look at the code. Anywhere you like. There's nothing I can say that will be a substitute for seeing it yourself.
I've had to use it as a library sometimes, and it's also really poorly designed as a library. There's no consistent principle for what is exported and what remains internal, so almost anything non-trivial you do may require you to copy-paste parts of the library code because something happened to not be exported. So you must think, okay sure, but in return for that you get a minimal and stable API. Nope! It also has some of the most API churn of any Go module, already up to /v4 and that's only counting the semver-major breakage they bothered to acknowledge, not the many semver papercuts along the way.
Sorry to pile on, but yeah, I wanted to use etcd during 2021 and 2022, around v3.5, but etcd had serious issues including silent data corruption. If you are curious, ask gemini flash "there were a number of etcd releases years ago where it seems a new wave of developers came in and started breaking everything"
I'd like to know what you base your statement on that the Raft implementations in etcd or CockroachDB are incorrect. Your original paper does not mention those implementations, so where does that claim come from?
Having run a fleet of 100s of etcd clusters for 10000s of rps, and the fact that upstream runs tests similar to antithesis and recently partnered with antithesis [0], and jepsen has tested it long ago as well [1]. Etcd's raft algorithm is fine. Someone even did a TLA+ proof on it in the last couple years[2]. Yes there was a correctness issue a few years ago but otherwise the person you're replying to doesn't know what they're talking about. Also those bugs have nothing to do with the raft implementation, but instead the state machine implemented on top.
"there was a correction issue" is downplaying it. Etcd is truly the worst example of Raft.
Etcd corruption and loss of quorum is extremely common in practice and the GitHub issues sit for years. The design is simple, the performance is modest, yet it still has still never been reliable, despite being marketed as so. I can't speak to whether this is specifically due to their Raft implementation, but I'd argue the entire codebase is over-engineered and questionable.
Their lock, leader election, sessions, and leases are all awful and I'd never recommend anyone to use those. But as a strongly consistent kv store and if you need the watch mechanics, its useful. It has its place and that's mostly being used by kubernetes.
The raft algorithm works and if you implemented a less complex state machine (like using a simpler kv store that doesn't need global event ordering via revisions and watches) it would work. That's what antithesis said they did to test the raft algorithms in the other article linked
doesnt raft have a problem that it assumes no hysteresis? and that in general you can construct a latency graph that deterministically causes a permanent lock in the leadership election phase?
I'm not sure I fully understand your question, but the heartbeat time outs and leader election time outs are static. And yeah if you make votes and heartbeats time out in a way that nobody can be elected, then raft can't make progress.
I've never had this be a problem in reality but AWS has a pretty good backbone. Maybe if you were running it over a pretty unreliable network this would be an issue?
Surely the King is doing it, so that must be the correct way. Look, the King even wears clothes and is totally not naked at all.
That the world runs on Kubernetes is no qualitative statement about the correctness of its Raft implementation. You can say that it's clearly good enough to not matter most of the time, but that is a different statement.
No matter who you look at, they're just cooking with gas like you do, and they can make mistakes in just the same way.
Now; I'm only attacking your argument. I do neither know nor particularly care about the correctness of that implementation itself. There's been better refutations of the claim you replied to in other answers anyway.
I have a bunch of volumes that I'd like to get automatically attached and mounted to nodes on which their respective workload runs (which are automatically scheduled) who automatically fetch and mount their config files and secrets from a HA DB on demand. I also need some internal loadbalancing and integrations with something like certbot for all of my web workloads. Id also like to make sure that I get metrics and logs from every workload in some form.
Thats basically it for starters, what non-technical solution do you propose?
So you propose to just let Ops personnel handle attaching volumes and renewing certificates, that these arent technical problems but just one of team structure?
Please, do describe your solution for the concerns above. So far Ive only seen equivalents of Greenspun's tenth rule
Having maintained multiple etcd clusters for self-managed kubernetes the last few years I disagree with the "problematic" characterization.
Sure it may not be the best fit in a scenario where you want a cluster spanned over the entire globe (thats why GKE uses paxos-based Spanner instead of it) , but even spanned across an entire continent (in europe via glass fiber) it works quite well for me. Its one of the least problematic parts of the stack.
I don't care for Go myself (especially its concurrency model, which is a total dinosaur in a world where we have structured concurrency) so I'm not saying this to support my favourite language, but:
I've been using golangci-lint for years, it's okay but it's slow even with a cache, as soon as you have good amount of lints enabled your ci can suddendly take minutes.
That's not the fault of golangci-lint, it's just the concept of bringing together a bunch of different linters and keeping it fast is impossible. And the configuration is a bit wild too, because every linter obviously has their own decisions on how it wants to be configured.
SQLite gets so much praise here but when you start using it, you realize quickly how bad it is, the type system is by default very limited and dangerous.
It's like comparing old php with a strongly typed language.
It’s curious how many people don’t understand what SQLite is and its intended feature set. They get huffy that it’s not a full client server model with multimaster clustering across 8 data centers on 12 continents plus New Zealand with realtime synchronous replication.
It’s a product that allows you to do sql like things without a database server. If you need to have database server behavior, you’re using the wrong product.
Well, it goes both ways. You'll see articles saying essentially "you don't need Postgres or any other fancy database, SQlite is enough" while ignoring the fact that some use-cases warrant a more conventional DB server.
I agree with you. There are 2 dozen foot-guns to be kept in mind. And discovered a new footgun regarding multi-byte strings and NUL handling today on HN. SQLite became popular because it was the only free and open-source choice 2 decades ago. Now there are other type-safe and robust choices.
It is very simple. Which means fast to setup in dev environment for local testing. Which makes first version very easy. And then people just keep fixing that one.
Still I quite a lot of question the use on servers if you have decided that I need a database.
Not that there isn't more valid use cases like local storage or self-contained information transfer for specific use.
From what I can tell, DuckDB is more focused on huge-scale data analysis than simple data persistence, so I’m not sure if it fits my use cases. Otherwise, it looks good.
It’s not as bad since you can always use a powerful programming language with a good type system that avoids type errors at the SQL level. You can build good abstractions in your programming language.
reply