Recommendations to avoid model thrashing?

[TimothySeah]

I have a cluster of triton servers. Each of these loads a different model depending on the request it receives. However, because these models are large, there is a lot of "model thrashing" i.e. we waste time loading/unloading models that are too large to all fit in gpu memory. I was wondering if there is a general/canonical solution to this? For example, is there an easy way to route requests requiring a specific model to pods that already have that model loaded? Thanks.