LLM Context Windows

Does it do well? How does it get around the context window limits?

Lots of models CAN handle large contexts, gemini 2.5 pro their latest model can take 1 million tokens of context

Context is 8k and it's quadratic. It "sees" everything in that window. If you want to have a long conversation try Claude or some of the 32k models. Claude uses a strange kind of attention that isn't always as precise but it's very good at finding key information in huge documents.

I think you misunderstand how context in current LLMs works. To get the best results you have to be very careful to provide what is needed for immediate task progression, and postpone context thats needed later in the process. If you give all the context at once, you will likely get quite degraded output quality. Thats like if you want to give a junior developer his first task, you likely won't teach him every corner of your app. You would give him context he needs. It is similar with these

This is impressive, how do people handle the limited context window of 64k tokens?

Tiny context window is killing it.

The large(100k tokens) context window together with the fact that it can actually use the information in that context window. From personal experience other models including open ai fail to properly answer when provided large(more than 5k tokens) inputs as context even when the model officially accepts much larger contexts. But Claude 2 models are uncannily good at taking all that context into consideration.

Are there any drawbacks to the larger context window? Like more hallucinations or lower speed?

That's interesting indeed. Could be that the context window of the model is too small...

The context windows of LLMs are now significantly larger than 2048 tokens, and there are clever ways to autopopulate context window to remind it of things.