"Nearest in the coordinate space" and "actually the best answer" aren't always the same chunk. A reranker is the second, separate look that catches the difference.
"Can I return the headphones I bought two weeks ago?"
So what
Vector search finds what's nearby. A reranker's only job is deciding what's actually useful — two different questions, worth two different steps.