I know of the chatbot arena, but that is more related to quality of output afaik. Benchmarking is really difficult with the recent explosion of models, as the permutation grew intractable.