Rewrite Dockerfile into multi-stage build, minor updates to support that change

[rtizzy]

The previous Docker setup is a bit confusing but I tried to infer what was intended.

Changes:

1. Ensures a proper ENTRYPOINT and CMD are so the MCP server starts successfully
2. Utilizes a multi-stage build so the version pushed to the registry is a slim image (~275 MB) that can have various LSPs and tooling layered on top as needed.
3. Includes minor updates to docs and ensures the push workflow leverages the proper target (prod)

The idea here is that someone who wishes to run this MCP inside of Docker can base off the pushed serena image and layer on their necessary dependencies.

Still a bit more work to do here and I'm currently experimenting a bit locally.

[MischaPanch]

Thanks, yes, it would be nice to improve this. We haven't put much attention to the docker topic or to optimizing the image. Pls ping me when you are done on your side, I'll then review.

Pinging @tirthpatel90 who is also working on improving Serena's docker issues

[tirthpatel90]

Awesome work @rtizzy! Just a heads-up, I'm currently working on Dockerfile.maximal in #1339 specifically to bake in all the heavy language dependencies (like R, Julia, OCaml) to accelerate the CI pipelines. Looking forward to seeing both our Docker optimizations in the next release!

[rtizzy]

@tirthpatel90

Thanks. A bit of constructive criticism:

It may make sense to step back and look at the entire CI/CD pipeline instead of just the Dockerfile for a few reasons.

The current multi-OS test suite is running outside Docker to test how it works on that exact OS for someone running outside Docker, which is still the standard setup. At minimum, you will still need the Windows test to exist outside Docker.

I haven't had a chance to dig fully into the tests yet but if all the tests do not require the full tool chain (My bet is they don't), what likely makes sense is something like:

1. Segment the tests by toolchain
2. Parallelize the CI/CD workflows across both OS (if desired) and by toolchain.

That means instead of needing one fat image, you can have a base optimized image (such as the one I'm working on here), base further images off of that if desired, install only the necessary toolchain, and run many of the tests in parallel.

That optimization will work with or without Docker and should result in a significant speedup.

I intended to perform that work after finishing this PR but feel free to pick it up.

[rtizzy]

@MischaPanch

Hope you're well.

Feel free to give this a once over and let me know if there is anything I might have missed.

I did have a question to make sure I'm understanding something correctly:

I'm pretty sure the architecture is that Serena MCP can only have one project activated at a time. I.E A single MCP server can only have one project activated and in use.

Is that correct?

[tirthpatel90]

Hi @rtizzy,

Thanks for the constructive feedback! You make an excellent architectural point. Segmenting the tests by toolchain and running them in parallel matrix jobs is definitely a cleaner, more scalable CI/CD practice compared to maintaining a monolithic image.

I would be happy to pivot and pick up the CI workflow segmentation to work alongside your base image.

@MischaPanch, what are your thoughts on this direction? If you prefer this parallelized approach over the Phase 1 maximal image, I can hold off on #1339 and start drafting the matrix workflows to run the segmented tests using rlzzy's optimized image. Let me know how you'd like to proceed!

[MischaPanch]

Hi @rtizzy . Thanks for the contribution and the discussion!

I'm pretty sure the architecture is that Serena MCP can only have one project activated at a time. I.E A single MCP server can only have one project activated and in use.

Yes, that's correct. In this PR you let the entrypoint use http mode for the MCP server - is that needed? Can't we use stdio and somehow forward the stream? It would be easier for the users

[MischaPanch]

@tirthpatel90 yes, if you are up to it, the parallelized approach would be far superior. Even though we would use more compute (due to multiple container startups) and the complexity of CI would become much larger, it would have the huge benefit of not having to wait 40 minutes each time.

A necessary requirement is that in the parallelized approach we don't have to change the GH workflows each time language support is added (although we should be able to do that when desired). Thus, it should be something like batch1, batch2, ... batchN, last_batch_executing_everything_else. Do you think that's feasible?

[tirthpatel90]

Hi @MischaPanch,

Yes, that is absolutely feasible and a brilliant way to future-proof the CI!

To satisfy the requirement of not modifying the GH workflows for every new language, we can dynamically batch the test discovery. For instance, we can explicitly assign the heaviest toolchains (like C++, Go, Rust, Java) into their own isolated parallel matrix jobs (batch1, batch2, etc.). For the final batch (last_batch_executing_everything_else), we can programmatically run pytest on all remaining tests that were NOT explicitly included in the previous batches.

This ensures that any newly added language tests automatically fall into the "catch-all" bucket without requiring manual YAML updates, keeping the maintenance overhead strictly at zero.

Since we are officially pivoting to this superior architecture, I will close my maximal image PR (#1339) to keep the repo clean. I'll start drafting the matrix workflows to run the segmented batches using rtizzy's leaner image and will open a fresh PR for this soon.

I'll get to work on this!

[rtizzy]

Hi @rtizzy . Thanks for the contribution and the discussion!

I'm pretty sure the architecture is that Serena MCP can only have one project activated at a time. I.E A single MCP server can only have one project activated and in use.

Yes, that's correct. In this PR you let the entrypoint use http mode for the MCP server - is that needed? Can't we use stdio and somehow forward the stream? It would be easier for the users

@MischaPanch

Thanks for the details. Give me a bit and I'll test things out with that approach.

stdio seems to fit the use case of Serena in VSCode much much better and completely circumvents a problem I was reasoning about.