When people benchmark MLX related quant models, they really need to publish numbers on benchmarks. You cannot take this as it is what you get of the original models. MLX uses pretty simple quantization methods so at lower bits without QAT, it is just not as good quality as llama.cpp ones.
Kinda yes. The benchmarks become part of the validation set, which means the models get slightly overfit to them if they are used as criteria for stopping the training. But a lot less compared to using them in the training data.
I'd guess everybody uses at least some benchmarks as stopping criteria, which is kinda sensible, but it also does induce some benchmaxxing, and explains partly why the newest models always tend to eke out in benchmarks.
Correct. If just stopping criteria, that is less contaminated. The question gets muddier once you also use it to determine hyperparameters during small-scale runs.
They are using it to evaluate checkpoints during the training, they are probably not using the benchmarks for training the models. It's a common practice for big reinforcement learning runs.
They exist to detect degradation. Datasets are not perfect and if a batch contains too much bad data it can ruin a run, also an opportunity to find bad data and improve the dataset filtering.
They run one step/iteration on an additional chunk of training data, then use the snapshot of the weights after that iteration in a separate validation benchmark while continuing to train on another chunk of data for the next iteration.
They result of the benchmark does not feed back into the training, it simply serves to provide a measurement of progression over time.
You gotta have something to aim at. And, presumably, the benchmark is not part of the training data, it is the test against which the model is tested at each stage; is behavior moving in the right direction?
It’s the difference between “study law until you can pass any random bar exam” and “here are 200 legal questions and we’ll drill them, with me correcting and explaining when you get one wrong, until you can pass exactly these 200”.
Your right that tuning can aim for a benchmark, but it does not leak any information about the answers.
The first implies generalization. It's not a test of generalization.
It's actually close to the second. "Here are 200 software questions, will drill you on *other stuff* until you can pass exactly these 200. If the other stuff isn't improving your scores we will change ratios of it till it does."
The reason it benchmaxes is that *other stuff* ends up looking more and more like SWE Bench without you realizing it.
I sort of understand DSH arch, and their choices, but still baffled by why you would want the agent to change its own agent loop other than the system prompt (in that extension, memory / soul / tools / skills).
The local LLM scene needs a Draw Things equivalent for Mac. Too much fiddle for things that doesn't make sense (Qwen 3.8 27B should be exactly the same speed as Qwen 3.6 27B). It feels like that I am teasing (I am the author of Draw Things) something, because it is.
What you get with Draw Things: 1. Download the app from Mac AppStore; 2. Download the model; 3. Tap "Try recommended settings", have the guarantee that for whatever model it supports, it is the fastest in Mac ecosystem, no need to fiddle.
What you get with local LLM options: 1. Download the inference engine app from the web; 2. Download the model; 3. Configure MTP / DFlash / DSpark whatever; 4. Configure your Pi / OpenCode harness to point to this local LLM inference engine. 5. Configure tools for these harness to be effective. 6. Switching between Ollama, LM Studio, llama.cpp (DwarfStar4), oMLX, MTPLX, to see which one is fastest for your workload. 7. Again switching between different quants of the same model to see which one is less dumb.
To be honest, llama.cpp probably the closest to deliver on "just use it, don't worry about speed" if your focus is about a pure LLM inference engine.
I’ve casually used Draw Things a couple of times (great tool!), and I did not find it beginner friendly. I remember feeling a lot of doubt, confusion, and frustration.
Am I getting these results because I picked the wrong model? Or I need to improve my prompt? Or the tool just can’t do what I’m trying to do? How current are these model recommendations? Have they been superseded by something newer?
As a true beginner to AI at the time, even the sizes and bits were meaningless to me. And I don’t remember having any context as to what I should be attempting to run on my mac.
So I think you need to include steps 4, 5, and 6 of swapping between different models, quants, and prompts. And step 7 is probably wading through the complicated UI, full of jargon that most people don’t know.
Don’t get me wrong, I recognize that it’s a powerful tool, and the steep learning curve exists because it exposes quite a few power-user features. But for someone graduating from commercial AI image generators that take a text prompt and maybe a choice of couple models, it’s not easy.
Yes, I heard you! I think one of the issue Draw Things inherited is the baggage of supporting too much models. Once you settled on a model, then it is just "try recommended settings", and prompt.
The model part is unfortunate, but luckily converging now.
On the LLM side, hopefully it is not an issue too as long as getting aggressive at pruning models.
Only if you do greedy sampling. With probabilisitic sampling (categorical sampling), you will end up with different trajectory just “mathematically equivalent”.
Not OP, but you can influence how deterministic your LLM behaves using the temperature setting. The neural network doesn't directly output tokens, but logits which are then converted to probabilities and then a token is chosen at random, unless the temperature is 0 (i.e. greedy, we just always pick the most probable token without any randomness). All speculative decoding methods have to "commit" to a token though even when they don't know the actual logits of the full size NN yet. The question then is (and I don't know the answer): how do the common inference engines behave when the speculation landed on the most probable token, but the random choice still doesn't land on it? You can imagine that in the interest of performance as long as we stay reasonably inside the probability we just go ahead with the speculation. Not sure if thats implemented like that though.
Edit: I just looked up the math, and actually the idea of speculative decoding is done in a clever way that fully preserves the probability distribution while still maximizing the acceptance rate of draft tokens. So I would have to disagree with OP and say that no, non-greedy sampling doesn't influence the trajectories.
Yes, it doesn’t impact the probability distribution due to verifier. However, remember how you use PRNG and effectively due to the drafter is sampled from a different distribution initially, a separate rejection sampling won’t be able to recover what the “old PRNG” would choose in a “without drafter” case. Hence in my original post, it is about different trajectories you will end up with, not the correctness of each stochastic sampling.
If you assume that the RNG generates true randomness, then the two are identical. Only if you care about the determinism of the RNG (for example you want to use identical seeds and get the exact same generation between the two) it makes a real difference.
Correct. I am trying to explain why even it is "exact", the generated text is different from the with / without DFlash2 runs, and potentially why the DFlash2 run will contain the invalid Python syntax.
If the underlying probability distributions are the same, then DFlash can lead to an invalid Python Syntax iif the autoregressive process could have generated one if the random sampling picked a different token.
If a model can output a “wrong” sequence with a certain probability p, then Dflash can also output the wrong sequence with the same probability. They wouldn't necessarily produce the same output from the same seed, but speculative decoding shouldn't be able to produce anything that the autoregressive model couldn't also produce when using a different seed.
It will contain the same or different syntax with or without it. Also multiple runs without DFlash2 will contain the same or different syntax. And multiple runs with DFlash2 will have the same or different syntax with the same probability. DFlash2 literally has no influence (unless buggy). The difference is purely caused by the randomness.
1. The syntax surface is smaller, allowing less LLM "creativity;
2. The error handling is mechanical, which LLM clearly prefers (LLM is already trigger happy about writing tons of throw / try...catch.. in other languages, doing tons of `if err` is just in it comfort-zone).
One thing similar would be projecting both the head.weight and the final LLM activations into a smaller vector space, since that is basically just cosine similarity ranking step (so that would reduce the head.weight size). But again, it must be tried many times and just not working as well. LLM space is pretty saturated with tricks.
One thing is not obvious to me is how ConvRot can be applicable beyond diffusion models. Especially for LLM decoding, as each ConvRot would be more expensive for a given decoding vector, and it is required now, so you cannot easily get the benefit for prefill only, while maintaining the same decoding performance.
It is a well-known trick, given that the timestep is between 0 to 1, you can slicing them at any resolution (1000, or 10000, give or take), and then keep a look-up table for modulation scale / bias etc for each. It is quite different from quantization and it is indeed lossless.
It is also only applicable to diffusion models as only these operates at per-timestep.
They ship a complete checkpoint for easily management (inference & training) in their own infrastructure. Moving to a LUT would make training on these layers impossible. BTW, these are not useful for lightweight fine-tuning, but might still be useful if you do serious post-training work.
Of course, these are also not an issue for things like FLUX.2 which adopts DiT-Air arch, that doesn't have this wasted space issue.
It may or may not be true. The people who made this modification and the other commenters didn't do anything rigorous to verify what they did. They just eyeball it. They could very well make some other error - this has happened frequently - that developing on prod, not knowing what they are doing, has and hopefully will again solve.
reply