Desktop computer mainly, sometimes a laptop. Tablets are painful to use IMO

Great news!

You gun down 1 (one) CEO in the street, and Musk starts carrying a 4 year old at chest level everywhere he goes

You can overwrite the model by using the same name instead of creating one with a new name if it bothers you. Either way there is no duplication of the llm model file

What I am talking about is when layers are split across GPUs. I guess this is loading the full model into each GPU to parallelize layers and do batching

Agreed.

Because they were banned.

I didn’t make this claim, no

Because they would just be inactive rather than banned? You’re the one claiming that bans have occurred, which would be a major censorship issue for lemmy…

Can you try setting the num_ctx and num_predict using a Modelfile with ollama? https://github.com/ollama/ollama/blob/main/docs/modelfile.md#parameter

Are you using a tiny model (1.5B-7B parameters)? ollama pulls 4bit quant by default. It looks like vllm does not used quantized models by default so this is likely the difference. Tiny models are impacted more by quantization

I have no problems with changing num_ctx or num_predict

Um… modlog is public. Where’s your evidence?

Models are computed sequentially (the output of each layer is the input into the next layer in the sequence) so more GPUs do not offer any kind of performance benefit

Ummm… did you try /set parameter num_ctx # and /set parameter num_predict #? Are you using a model that actually supports the context length that you desire…?

Notice how all those bot accounts that were so active leading up to the election have completely vanished from the internet now? Yeah.

It’s not. I can run the 2.51bit quant

Tell that to my home rig currently running the 671b model…

Hawley’s statement called DeepSeek “a data-harvesting, low-cost AI model that sparked international concern and sent American technology stocks plummeting.”

data-harvesting

???

It runs offline… using open-source software that provably does not collect or transmit any data…

It is low-cost and out-competes American technology, though, true

this is deepseek-v3. deepseek-r1 is the model that got all the media hype: https://huggingface.co/deepseek-ai/DeepSeek-R1

https://archive.org/details/www.cdc.gov_en_all_novid_2025-01 there’s a zim

Trump certainly sounded like he wanted it to be that way with Ivanka