[CI]: Test container layer caching optimization

[njhill]

We should make sure that the layering/stages are such that the vllm python/docs code resides only in a very thin top layer and that changes to these don't affect lower layers. For example installing all dependencies in a separate command/layer (I think we're already doing that, but we should verify that it's working as intended).

Further we should ensure that the intermediate layers are pushed to the docker repo when building and also that remote layer caching is enabled/working when building.

This should be complimentary to the other base-image reuse work @dougbtv is doing for #23588, and contributes to #24779.

It should help with build time but also startup time on other workers pulling new images.

[njhill]

@afeldman-nm if you comment here I can assign this to you...

[afeldman-nm]

Feel free to assign me!

[afeldman-nm]

Possibly related: #25177

[bbartels]

Hi, yeah I didn't even see this issue, just got annoyed that it wasn't caching layers correctly when i was building the docker image locally and making changes to various directories. Should be a bit better with #25177

[afeldman-nm]

CI infrastructure PR vllm-project/ci-infra#174

See status update here:
vllm-project/ci-infra#174 (comment)