Migration of Simulation Systems to Git ecosystem - 1

[yaswant]

News

Published in: NGMS Newsletter - February 2025 - Issue 7

The NGMS Programme Board, with strong support from Momentum® partners, approved the migration of remaining NGMS codes on Subversion/Trac (Science Repository Service, SRS) like LFRic to Git and GitHub in approximately late 2025, subject to avoiding any knock-on impact on LFRic GPU/HPC benchmarking. This is a really positive step forward for collaborative software development across the Momentum partnership, but it is a big change.

Rationale

The Met Office have been using Subversion and Trac systems as source control management tools for many years, and this provided a solid platform for scientific code development of delivery of high impact products such as the UM. However, Subversion and Trac development have declined to the point of no releases in the last ~4 years, single digit contributors, and their survivability as maintained projects in the late 2020s is under threat. Transitioning to Git and GitHub will allow us to transform our current version control system capabilities to the modern productive tooling standard improving usability of systems and technical maintainability, reducing overheads for external collaboration as we move to NGMS.

Why move to the Git ecosystem

Improved performance, enhanced collaboration and simplified workflows

The shift to Git and GitHub will bring significant advantages to the Met Office's development environment by improving performance, boosting collaboration, and streamlining workflows. Git’s distributed structure offers faster branching and merging, making it more modern and flexible compared to current systems. Its extensive community support and superior scalability will make our version control processes more efficient and responsive. This transition allows us to improve our development process, enhancing both productivity and collaboration across our teams and Momentum® partners.
GitHub, designed as an integrated platform for the Git ecosystem, offers a richer suite of tools than our current Trac-based system. While Trac has served well for issue tracking and basic project management, GitHub brings advanced collaboration features such as Pull Requests, Issues, Project Boards, and Discussions. Additionally, its built-in continuous integration and deployment (CI/CD) capabilities provide a robust system for automated building, testing, and deploying of code and documentation, which aligns perfectly with our future-oriented development needs.

What will this transition include

The transition will affect the science repositories that include our key Met Office Simulation systems. This includes:

1. UM (Unified Model) family: Repositories such as UM, GCOM, SHUMlib, mule, JULES, UKCA, SOCRATES, CASIM, and MOCI.
2. LFRic family: LFRic (core) infrastructure and LFRic applications repositories, with more permissive BSD-3-Caluse licence.

Throughout the transition, we will maintain open communication on change management, ensuring stakeholders stay informed and engaged.

How will we achieve this

The transition to Git and GitHub involves a carefully planned, multi-step approach:

• Stakeholder Engagement: Close collaboration with internal teams, Momentum® partners, and academic stakeholders will ensure that the migration meets the needs of all involved. Ongoing
• Governance and Best Practices: New governance protocols and working practices will be developed, including guidelines on safeguarding operational configurations.
• Training: The Git Migration Team will provide targeted training to equip staff with the skills needed to navigate the new system. Ongoing
• Preparation Steps
  • Repository Forking: Forking of key repositories, such as UM Physics Fork #285 to LFRic Apps.
  • License Review: Revisiting and, where necessary, decoupling licensing agreements.
  • Tool Evaluation: Assessing and implementing migration tools gitlify/milestone/1🔒
  • Pilot Project: Testing core migration assumptions on a smaller project, such as shumlib🔒 or mule🔒
  • Parallel Testing: Running stability checks to ensure smooth workflow transitions
• Final Transition Steps
  • Ticket Migration: Critical tickets will be migrated to GitHub as needed, while others will be resolved or closed.
  • Stakeholder Consultation: Further stakeholder discussions to refine governance, assess risks, and finalize best practices.
  • Cooling Period: A brief, one-month "cooling" phase to ensure readiness.
  • Go-Live: Following a freeze of SRS repositories (read-only access), we will go live on GitHub. Development work will then fully transition to the Git ecosystem.

When will the transition take place

Assuming we have completed the UM Physics fork and a stable version of Fab build system, tentative timeline for this migration is as follows with an aim for March 2026 completion (subject to change).

Risks and Mitigation

Key challenges in this transition include migrating workflows smoothly from FCM to Fab, managing potential SRS outages, and synchronizing the workflow migration with parallel suite operations. A comprehensive mitigation plan will be implemented to minimize disruptions and address any issues that arise.
This move to Git and GitHub represents an exciting step forward for the Met Office, equipping us with a scalable, modern development platform that will support our scientific and operational goals well into the future.

What does this mean for partners?

• Be prepared to migrate away from Subversion and Trac in late 2025 and early 2026, e.g. by planning to reserve time or proactively engage in training if you need it
• We are looking for Git and GitHub champions from partners to help provide feedback, help write documentation, inform strategy, and help users (including delivering training on occasion) at your own locations.
• Please look out for future updates and opportunities to contribute and jointly build a great development platform.

[penguian]

Thanks for this. Can you elaborate on what this will mean for workflows currently based on Cylc 7, FCM and UM-based code? Will they be expected to migrate to Cylc 8 and FAB, and over what time scales? How will partner testing be managed?

[yaswant]

Hi @penguian, Apologies for the delay in reply. Our current expectation is that everyone should switch to Cylc 8 asap. There is a ticket to deprecate Cylc 7 for UM rose-stem in the next release (UM 13.9). It is very likely that we will migrate to Git/GitHub with the current build system as an interim solution, while work will continue to further develop Fab-based build system. I'll post an update on migration timeline soon. We are currently reviewing the working practices; further communication will follow around September.

[MichaelWhitall]

Hi folks, just looking at the latter part of the the proposed timeline:

• October 2025: SRS code freeze (UM, LFRic).
  ...
• January 2026: Go live (on github).

Does this mean there will be a 3 month interval where no model code-development can take place??? (since all branches on FCM/trac will be frozen in October and therefore can't be committed to, but we won't be able to resume UM and LFRic code-development in github until the "go live" in January?)

[yaswant]

@MichaelWhitall Note that the code will already be on GitHub before final transition. So there is an overlap between "final transition"/"cooling period" and opportunity for developer engagement. The transition wont take long, but making sure the working practice is in place, upgrading various suites will. We'll announce when small group of developers can actually access GitHub repositories during the testing phase.

[MichaelWhitall]

Hi that's great thanks Yaswant,
So actually we will be able to move our branches and tickets over to git to carry on working on them some months before the "go live"? When will this become possible?

Cheers!
Mike

[yaswant]

We are gathering requirements to make a decision on development strategy - most likely we will adopt a fork-based approach (tbc). Don't worry, we'll provide guidance on how to migrate your svn branches to git. For general developers, this should be possible at the first release on GitHub. We can provide early access to core developers slightly earlier. I'll post an updated announcement soon.