Only a few of these are actual programming tricks. The problem with sharing them is that they'll typically seem obvious to you, since you know them. It's difficult to know what is actually unknown to other people, and if you share stuff everybody knows you risk coming off as arrogant.
Here's one that I think more people should know: avoid branches. If I can do the same thing without an if statement and even a logical expression, the code typically both becomes easier to understand for people and easier to run for the CPU.
>if you share stuff everybody knows you risk coming off as arrogant
I have always felt like my bar for publishing something (even just to internal wikis/channels) is too high due to being overly self-conscious. I think we should try not to validate that feeling by implying that there is a non-negligible number of readers who will think you have a personality flaw because you wrote down your personal collection of tips in a public place, or that those people deserve consideration in the first place.
There is no such thing as "the things everybody knows". There are just too many things. Even a list of basic tips is probably going to contain one thing I didn't know or perhaps forgot. Write-ups like this are where most of my practical knowledge comes from, not RTFM (which I do).
I'm struggling to comprehend how branches can be avoided (or why one would want to, as they are the cornerstone of programming). I can only think how to obfuscate them, which is rarely useful.
Flow control is not always necessary. Other times it can be minimized. The point is not to never branch, but to avoid unnecessary ones.
It's not applicable to every situation, but one way to do this is some very basic fuzzy logic. You do a little math and then either choose a single branch at the end, or sometimes avoid a branch altogether. https://www.geeksforgeeks.org/artificial-intelligence/fuzzy-...
Another way to avoid some branches is to have specialized routines, maybe with multiple dispatch, rather than more general methods with a bunch of checks within them for slightly different situations.
A classic performance hack for critical sections is loop unrolling.
Most of the time, no. Branchless code is harder to read and easier to get wrong. Besides, compilers know a lot of tricks and already do a lot of this work for us.
Only when a profiler points at a hot loop, and the loop contains a branch on unpredictable data this technique can pay off big.
Yeah, I don't buy the premise that branchless code is intrinsically easier to understand. Maybe OP's point is that adding unnecessary branches makes code harder to read? But that's generally the case for any unnecessary code.
and skipping the if just makes the function shorter and simpler, while also not involving the CPU branch prediction. Another one that doesn't necessarily skip all branching but at least drops one - and more importantly makes the code simpler and easy to verify, is removing the if statement in code like
if (count == 0) {
return;
}
for (int i = 0; i != count; i++) {
puts("hello");
}
Both of your examples are optimized by the compiler (gcc 16.1 -O3).
In the first case, the compiler removes the first if/return
In the second case, if you don't have the first if/return the compiler will add it. That's because it will actually convert your loop into a do/while, with the test in the end, because it is more efficient. But it has to handle the count == 0 special case first, so it will do that early return even if it is not explicitly there.
That's the kind of optimization modern compilers are good at.
There's stuff like the "Command Pattern"/dispatching/subclasses etc that can make this nice, although it's not always a good fit.
Like imagine you have a few different classes of things A,B,C so instead of checking if the thing you're handling is an A,B,C you have like a shared interface across all and can call Thing.do_it or whatever.
Still branching conditionally but it's passing it off to language features instead of code you have to write.
well here is a branching strategy I often see, pseudocode, and often this is a really stupid example as I do not have the time to come up with a good one:
if Val === "A" then
Do funcA()
else if Val === "B" then
and so forth for lots of values, or using a switch statement or similar branching instead of
Object functions = {
"A": funcA() {does what funcA does},
"B": funcB() {does what funcB does}
etc. etc.
Actually writing it I remember now someone who did this, a junior who had to update a validation function for XML invoices based on their root namespaces, which there could be a large number of these, and so she wrote out
I can't remember all the details as this was almost 20 years ago, however while it was true that one branched on the schema, it made much more sense to look up what one was supposed to do based on the rule for branching and then just execute that one action rather than writing a bunch of branching logic.
So to make it more concrete: Once branching rules becomes sufficiently complex prefer query for what you should do rather than branching
on edit: note again, not real code, but should be understandable and translatable into real code to understand what is being said easily enough.
on 2nd edit: this is also just basically one of the things I prefer instead of getting a lot of branching logic. I have never seen any stats on any benefit to this model than just having a bunch of branching statements, but I feel that the benefit is there nonetheless.
What people refer to when they say "branchless code" is something very particular, and it refers to not triggering the CPU's branch prediction. That is, don't make the CPU have to guess which fork in the code you're going to take. This is usually accomplished in one of two ways: either bit twiddling hacks or specialized instructions that do not affect the CPU's branch prediction, such as the 'cmov' family in x86. If you search for `examples of branchless code using conditional moves` using your search engine of choice, you'll find numerous examples.
A trivial example is actually written with a branch in C/C++, but relies on compiler optimizations to kick in. If you compile a ternary operator in C/C++ (and probably rust, C# and other languages) such as in:
int min_branchless(int a, int b) {
return a < b ? a : b; // Often emits cmov with -O2
}
With gcc/clang a -O2, one would expect the compiler to emit the following assembly:
cmp edi, esi
cmovle eax, edi ; select a if a <= b
ret
There's numerical tricks for other operations/comparisons, and compilers know a lot of them. But, I just suggest compiling your code and configuring your compiler to emit the generated assembly with references to the code it was generated from (you should be able to get it to emit source line references in the assembly). You'll likely be surprised at the optimizations applied at -02, and utterly confused by what you find at -03.
edit: Also, it doesn't mean to never branch, but to minimize branching, especially in tight loops. Branch outside loops, not inside, for instance.
e.g. don't do:
for (...) {
if (condition independent of loop variable) {
...
} else {
...
}
}
do:
if (condition independent of loop variable) {
for (...) {
...
}
} else {
for (...) {
...
}
}
Yes, and back when hand-writing vectorized kernels via intrinsics, one learned to do the equivalent of (pseudocode here - picture SSE, AltiVec, NEON, etc.):
where each lane of the conditionmask has either all bits set or all bits clear, depending on the outcome of the conditional test for that lane.
The processor obviously does execute both branches here, so there's going to be wasted work. But since it's just a linear sequence of operations it can often schedule them independently and run them out-of-order and in parallel. And of course, if there's any shared computation between the two branches, the compiler can do common subexpression elimination.
That said, that sort of approach where you go ahead and do both and then blend them was definitely the kind of optimization where you'd want to profile rather than doing it blindly. But it was a pretty common thing to do when hand-vectorizing code. (Thankfully, auto-vectorizers are pretty good at doing this sort of optimization for you these days. It's been a very long time now since I've had to hand-write vector intrinsics.)
Rust is an expression language and so it doesn't have "the ternary operator"† you can use conditionals like if anywhere in your expression anyway.
If you want to tell the Rust compiler that you're certain a branch predictor can't help here [be very sure, most often humans are wrong which is why historically these "I know better than the branch predictor" features get ignored by optimisers] you can core::hint::select_unpredictable(condition, a, b) rather than using a dedicated operator.
† That's not its actual name, some languages have an operator with three operands which does something else, such as fused multiply-add so in a multi-lingual context better to say explicitly you mean the ternary conditional operator.
The latter example, sounds like something trivially done by the compiler. I mean I would sometimes, adhere to it, but only if the loops afterwards become substantially different. If I would just repeat most of the loop body, I would prefer the former.
> The latter example, sounds like something trivially done by the compiler.
Taken by itself? Absolutely. In the middle of 50-deep templatized call stack? Maybe not. Compilers have their limitations (memory, runtime, other cost budgeting algorithms), and the more complex the code, the hard the optimizer has to work. But, yeah. Most of the gains from branchless code is going to come from this like numerical tools, vectorization and loop unrolling, things of that nature. It was an admittedly contrived example.
A common pattern in an old C++ job I had: People writing for loops, coupled with if conditionals, for things that could just be done by chaining functions in the algorithm library.
Don't do a for loop, check for a condition, and break. Use find_if.
There are multiple ways to avoid branches. An early return, a lookup table are two that I use regularly and consider a code smell when the AI uses many if clauses or switches.
A (hopefully interesting) aside about avoiding branches is if you don't need an exact answer but need your code to make a decision based on an approximation over some known range, you can employ a basic fuzzy logic method. Serially add, subtract, or multiply to adjust a value by a handful of weighted inputs then use that value instead of branching repeatedly to choose the right action. You might branch once based on the final value where it would have otherwise been a larger tree of decisions. In fortuitous situations, you may avoid branching altogether.
if you share stuff everybody knows you risk coming off as arrogant.
Or stupid, like all those vloggers posting "ZOMG! Go all in with these secret hidden weird trick iPhone life hacks to level up!" that are just regurgitating what's in the manual.
v = setup()
if v == 1:
side_effect_1()
elif v > 1:
side_effect_1()
side_effect_2(v)
else:
raise Exception()
then we can "refactor"
v = setup()
if v < 1:
raise Exception()
side_effect_1()
if v > 1:
side_effect_2(v)
i know that this might seem "dumb" that the code was ever setup the first way but code can grow into that shape pretty easily. this refactor "removes" the v==1 branch. this new code also follows the "early return" pattern, which improves readability.
No, because the point is that the branch version does not touch "total" when the branch is taken, but the non-branch version will always multiply "total" by something.
I keep my skills in a Home Manager repo and install them into my .claude / .codex / whathaveyou directory through the home manager config. I'll know if they don't work because they are specific instructions on how to git commit, how to merge code, how to author text (without the typical AI tells), or API usage documentation for specific libraries, etc. If they didn't work the agent would do things incorrectly and I'd notice.
And sometimes it doesn't follow the instructions well. I have a skill for that too: it tells the agent, given what it knows about attention and LLM:s in general, to evaluate the instructions and the mistake the LLM made, try to diagnose why it didn't follow the instructions as expected, and come up with an improvement of the skill based on that diagnosis.
> What is happening here is that leading AI labs are charging not only for inference but also for research in model architecture, training data collection and curation, model training cost (which can be tens or even hundreds of millions of dollars), paying their employees and recovering the marketing costs.
Great, but that's only a part of operational costs. A craftsman's revenue may exceed the electricity bill for the power drill, doesn't mean the business is sustainable.
"Otherwise". If the craftsman revenue isn't enough to recover the investment expenses, the business is operating at loss. But that's beside the point, because research investments are not the issue at hand.
As said before: Interference costs are not the only operational costs. Same as electricity costs for the craftsman. Running a power drill is not the the whole expense to consider. The craftsman has to eat, AI company's employees have to eat. The craftsman has to learn about new building standards, the AI company has to train their models because no one wants to use a product stuck in time (that's not "research", just maintenance). If not even interference was recovered in revenue, nobody would even start to argue about sustainability.
I can't debate this further, because HN is rate limiting my account for dissenting opinions in the past.
They are not. They are necessary for the development of future models, which does not influence the availability of the current ones. Plus you have chinese models distilling current SOTA for pennies on the dollar, so as a consumer I never will be worse off in the long (1-2 years) run.
Is this supposed to be some sort of gotcha? Apart from research and marketing, that's operational costs. I mean, every product could be cheaper, if you didn't have to pay for employees and means of production.
> Adding an in-product tip to recommend running /clear when re-visiting old conversations (we shipped a few iterations of this)
I feel like I'm missing something here. Why would I revisit an old conversation only to clear it?
To me it sounds like a prompt-cache miss for a big context absolutely needs to be a per-instance warning and confirmation. Or even better a live status indicating what sending a message will cost you in terms of input tokens.
Instead of just dropping all the context, the system could also run a compaction (summarizing the entire convo) before dropping it. Better to continue with a summary than to lose everything.
I use claude code with an API key and pay per token, and the /cost command is very helpful.
And before people ask, it's because I have a very low usage and it's cheaper to pay per token. I'll have the odd month at $30, then nothing for a few months
It exists on my work enterprise account but not my personal account which is a monthly flat rate. I assume if I exceed my quota and I choose pay as I go then it will become available.
And if you don't want to buy a Mac? A 80 GB NVidia GPU costs $10,000K (equivalent to 30 years of ChatGPT Plus subscription) and will probably be obsolete in 5-7 years anyway. What are my options if I want a decent coding agent at a reasonable price?
My performance when using an RTX 5070 12GiB VRAM, Ryzen 7 9700X 8 cores CPU, 32GiB DDR5 6000MT (2 sticks):
- "qwen2.5:7b": ~128 tokens/second (this model fits 100% in the VRAM).
- "qwen2.5:32b": ~4.6 tokens/second.
- "qwen3:30b-a3b": ~42 tokens/second (this is a MoE model with multiple specialized "brains") (this uses all 12GiB VRAM + 9GiB system RAM, but the GPU usage during tests is only ~25%).
- qwen3.5:35b-a3b: ~17 tokens/second, but it's highly unstable and crashes -> currently not usable for me.
So currently my sweet spot is "qwen3:30b-a3b" - even if the model doesn't completely fit on the GPU it's still fast enough. "qwen3.5" was disappointing so far, but maybe things will change in the future (maybe Ollama needs some special optimizations for the 3.5-series?).
I would therefore deduce that the most important thing is the amount of VRAM and that performance would be similar even when using an older GPU (e.g. an RTX 3060 with as well 12GiB RAM)?
Performance without a GPU, tested by using a Ryzen 9 5950X 16 cores CPU, 128GiB DDR4 3200 MT:
I'm able to run the Unsloth quants on an ancient dual socket Xeon 1U server I keep around for homelab stuff. It has 8 DDR3 channels, which gives me about as much memory bandwidth as two channels of DDR5 :-/ But 16 sockets and cheaper prices. So it has 256gb in it right now. I have to run the minimum size Unsloth quant for the largest open weight models. They definitely feel a bit dazed. This machine can support up to 1.5TB of DDR3, which would allow me to run many of the largest models unquantized, but at 1/4 of the already abysmal speeds I see of ~ 1 Token / s which is only really usable with multiple agents running a kanban style async development process. Nothing interactive. That said, I picked up the hardware at the local surplus for $25 and it's vintage ~2010. Pretty impressive what this enterprise gear can do.
Power consumption? Don't ask. A subscription is cheaper.
That’a the thing, at the end of it all power consumption will matter more for the end-user who doesn’t have money to burn away, because I suspect that power-consumption will, in the majority of cases, exceed the price of the HW itself in a matter of just a few months of intense use, let’s say a year.
Assuming models of a fixed size continue to improve in capability, continued advancement in semiconductors and optimization will reduce power consumption and/or improve performance over time. And used equipment will always approach the scrap price eventually. For me today, on scrap equipment, I get about 4 tokens / watt-hour, which is nominally ~$0.17 US but could run $0.40 after all the taxes and fees and surcharges. $0.10 / token. Ouch.
If I were to try to purpose build a rig for it, I would get an engineering sample Epyc/motherboard/ram combo from Aliexpress with 12 channels of DDR5 and as few cores as allowed me to still use all the memory bandwidth, and I'd run it at the lowest possible power and voltage settings with aggressive ram timings. A system like that can draw 1/3 of what my scrap rig draws, at full load. And has similar memory bandwidth to a high end Mac or GPU allowing it to crank out 5 - 10 Tokens / s on the largest models, which works out to 1/3 of a penny to 2/3 of a penny per token. But either way, Epyc or Mac is going to set you back $10k or more. Hopefully in a few years when they are scrap though...
GPUs are not going obsolete anytime soon. the nvidia p40/p100 launched in 2016, 10 years ago and is popular in the local space. My first set of GPUs were a bunch of P40s from 3 years ago for $150 a piece. They at one point went up all the way to $450, but price is now down to $200 range. I think I have gotten my value from those and I suspect I'll still have them crunching out tokens for at least 3 more years. They still beat 90% of cpu/memory inference combo.
My point being that no one should be buying expensive GPUs when you can pick up a few used ones to get started. But for the sake of discussion let's say you do get a blackwell pro 6000 that's now going for $10,000. I can assure you it will not be $150 10 years from now, with the falling price of dollar, demand for AI inference and hardware shortage, it might cost exactly the same 10 years from now...
the macs outperform it and I figure it's a better general purpose computer than strix halo. if budget is a problem, then a strix halo is a decent alternative.
Well a mac isn't really an alternative to a mac, or is it? ;)
Personally I'm not interested in having a mac as I work with linux. And yes, they outperform them, but only if you ignore the price. When comparing what you get for ~$2k, a Strix Halo is miles ahead.
Yeah I'm pretty happy with the M5 (beside the look). It's most probably the same SixUnited board most Strix Halo devices use (including the ones from HP and Lenovo).
Still lot to learn, but after a while you have something like Qwen3-Coder-Next-Q8_0 running and - at least for me - it works quite well, both as ChatGPT like chat-interface using llama.cpp and as coding agent
I'm not really using them for coding (only played a little bit with minimax2.1), which is probably the most common use case here.
I mainly use them for deep work with texts and deep research. My main criterion is privacy, both for legal reasons (I'm in the EU and can't and don't want to expose customer's data to non-gdpr-compliant services) and wouldn't use US services personally either, e.g. I would never explore health related topics chatgpt or gemini for obvious reasons.
Technically I've set it up in my office with llama.cpp and have exposed that (both chat interface and openai compatible api) with a simple wireguard tunnel behind nginx and http auth. Now I can use it everywhere. It's a small, quiet and pretty fast machine (compiling llama.cpp is around 20 seconds?), I quite like it.
What are my options if I want a decent coding agent at a reasonable price?
I'd even come from another angle.. What are my options if I want a decent coding agent, on the level of what Claude does at any given price? Let's say few tens of thousands of dollars? I've had a limited look at what's available to be run locally and nothing is on par.
Does not exist AFAIK. Even other labs struggle with Claude level performance in real world task. My experience is that no open model is close. You can get RTX 6000 Pro Blackwell (Max-Q is better for power is half). I have heard good things about Qwen3 coder next but I could not get tool calling to be high performance but it’s likely to be pebkac.
If you want to spend big bucks get h200 141 GB but honestly RTX 6000 pro is good enough till you know what you want. Workstation edition is good. It takes care of cooling etc.
Tbh even better is to just get model through cloud. If you want you can rent GPU. Then see if it’s what you want.
The gist of it is no matter the money you spend on hardware, you will not get the same quality you get from claude. Main question is then what can you run that's good enough? I haven't tested all there is available, but everything I did see does not come even close.
reply