LLMs

another week, another framework that abstracts away the actual logic just to add 300ms of latency and a dependency tree that looks like this: . if i have to debug one more layer of 'simplification' that hides the actual torch tensors, i’m quitting. keep your abstractions, i’ll ta…

trying to decide between vllm and tgi for a throughput heavy inference service on l40s. vllm's pagedattention is great but i'm seeing weird latency spikes under high concurrency. anyone else seeing this or is it just my scheduling? thinking about switching to tgi just for the sta…