Re-ranking

Reranking 150 candidates for every question, at the running case's peak of 6 questions a second, means pushing 900 question-and-chunk pairs through a model every second, inside a p95 budget of 1.5 seconds to the first token. Whether that takes one GPU or six, 20 milliseconds or a second, depends on the kind of model doing the scoring, how many candidates it sees and how long they are.