@sousekd

Thanks for the benchmarks here on the new GLM-4.7 comparing with Kimi-K2-Thinking over here: https://huggingface.co/ubergarm/Kimi-K2-Thinking-GGUF/discussions/5

I didn't graph the comparisons which would be easier to read the tea leaves, but seems like Kimi-K2-Thinking as faster TG as context depth gets deeper. Pretty interesting as Kimi is a 1000B model vs the GLM's 358B! Maybe has something to do with MLA style attention on Kimi?

A few thoughts about your comments over there:

Kimi-K2 Thinking Q4_X with max supported context on RTX PRO 6000. It only fills 50% of VRAM (47 GB), while GLM 4.7 IQ5_K takes it all.

Yeah the MLA is much more efficient on VRAM storing long context is my guess. If you don't need all that extra context length, offloading more routed experts into VRAM will likely boost your speed making Kimi even faster at TG relatively. (My modelcard for GLM-4.7 has some newer examples)

This is without --grouped-expert-routing --merge-qkv, which I'm not quite sure what they are about:

I believe the -ger grouped expert routing is not for all models, but does work with GLM and can give a small improvement. Similarly when using a single GPU you can get like 1-2% speed-up also with --merge-qkv as these are small fusions of kernels reducing overhead is my understanding.

It has been tricky to keep up with all of ik's hard work lately haha... Especially if you have 2x GPUs and NCCL installed you can get some great improvments with -sm graph becoming competitive with VLLM in my early impressions at least at single user generations. Doesn't help tho for 1x GPU as that is already optimized.

The significant drop in speed around 82K is interesting.

Yeah wonder what that is about, perhaps some kind of power limiting kicking in or temperature throttling or something happening ? Have you checked out LACT for undervolting your GPUs instead of simple power limiting? Ends up giving roughly same performance at like lower power usage too! (holler if you need a link about LACT or i have some discusson of it in my recent talk here with timestamp listed in text: https://blog.aifoundry.org/p/adventures-in-model-quantization

Cheers and happy new years!

Thank you for your insights, @ubergarm .

Yeah wonder what that is about, perhaps some kind of power limiting kicking in or temperature throttling or something happening ?

That was my first thought too — I do have fairly aggressive throttling configured when RAM temps get too high. So I monitored temps and re-ran the bench three times, and it doesn’t seem to be the culprit.

Have you checked out LACT for undervolting your GPUs instead of simple power limiting? Ends up giving roughly same performance at like lower power usage too! (holler if you need a link about LACT or i have some discusson of it in my recent talk here with timestamp listed in text: https://blog.aifoundry.org/p/adventures-in-model-quantization

Not yet — I’ve been a bit scared to touch undervolting 😅. But it’s on my "TODO (eventually)" list. Thanks for the link — I’ll watch it now!

Cheers and happy new years!

You too! And thank you for your work.

Here is Q8_0 for comparison. It is significantly slower at lower context, but the difference is marginal later. It doesn't fit 192K at -ub 8192 -ctk f16 to 96 GB tho:

Ah, this is tempting me to run GLM 4.7 on the RTX 5090 and see how it hangs. I've been positive with MiniMax M2.1 since it's smaller and more generous on memory. Different fine tuner.

Have you checked out LACT for undervolting your GPUs instead of simple power limiting? Ends up giving roughly same performance at like lower power usage too! (holler if you need a link about LACT or i have some discusson of it in my recent talk here with timestamp listed in text: https://blog.aifoundry.org/p/adventures-in-model-quantization

I have LACT and afaik there's no direct undervolting capability on Linux as these been disabled on nvidia drivers for Blackwell. I was adamant getting it done.

Although, offsetting the core clock +50 and a touch lower power consumption (400w) substantially drops the temps, offsetting the small performance loss.

Works well on fully VRAM MoE models which virtually hits full core speed, whereas dense models will be visibly slower.

Is the temperature on RAM significant on offloaded models? Never checked their temps. Nuts.

Is the temperature on RAM significant on offloaded models? Never checked their temps. Nuts.

Yeah — it can be. Offloaded models put a lot of pressure on the memory subsystem. On my machine, I couldn’t finish a single sweep-bench run until I added a DRAM heatsink. Consumer DIMMs often ship with heatsinks these days, but server RAM typically doesn’t — presumably because it’s designed for chassis airflow with turbine-style fans.

Is the temperature on RAM significant on offloaded models? Never checked their temps. Nuts.

Yeah — it can be. Offloaded models put a lot of pressure on the memory subsystem. On my machine, I couldn’t finish a single sweep-bench run until I added a DRAM heatsink. Consumer DIMMs often ship with heatsinks these days, but server RAM typically doesn’t — presumably because it’s designed for chassis airflow with turbine-style fans.

Thanks for the heads-up. Big blind spot from my side. Might have been lucky since I always have the room cooler and has plenty of airflow kicking earlier.

Never seen any particular signs. What caused your bench not to finish? Locking up?

Mine don't have heatsinks. I suppose I need to stress test them and align myself to potentially get some.

@wonderfuldestruction

It was a few months back, but I remember watching RAM temps climb to ~97°C, everything in red, and me not shutting it down… until the machine did it itself 🙂.

After booting again, one DRAM module didn’t show up. I shuffled the modules around the next day, pretty sure I had just lost a 64 GB stick and I just needed to figure out which one — but the machine eventually recognized all of them. And it actually recovered: I stress-tested it thoroughly afterwards.

That experience forced me to find the “memory frequency throttling” screen in BIOS and put sensible limits there. Once I did, every sweep-bench was crippled by throttling. The heatsinks I installed later fixed all of that.

https://cdna.pcpartpicker.com/static/forever/images/userbuild/510756.1b9ce15bb0a74a45db0b0d8c33e81959.1600.jpg