diff --git a/git-migration/README.md b/git-migration/README.md new file mode 100644 index 0000000..2898ac2 --- /dev/null +++ b/git-migration/README.md @@ -0,0 +1,130 @@ +# Met Office Simulation Systems Git Migration Project + +This project is aimed to migrate the Met Office simulation systems repositories +to Git ecosystem. See MetOffice/simulation-systems/discussions/337 for details. + + +> [!IMPORTANT] Pre-requisites to run the migration script +> - Read access to the Met Office Science Repository [MOSRS](https://code.metoffice.gov.uk/trac/home) or the Met Office internal mirror (`svn://fcm1`). +> - Optional Write access to the [Met Office GitHub organisation](https://github.com/MetOffice) for push to remote (usually, an admin). +> - Tools: [`fcm`](https://metomi.github.io/fcm/doc/user_guide/introduction.html), [`git`](https://git-scm.com), [`gitlify`](https://github.com/MetOffice/gitlify), [`jq`](https://jqlang.org), and [`gh`](https://cli.github.com) available on the system. + +## Checklist + +1. Test `gitlify` translation tool + - [x] Able to convert only svn trunk to Git. + - [x] Able to map svn revisions to Git tag. + - [x] Synchronise trunk updates locally in Git repositories. + - [x] Set-up a (`scron`) job to update code/tags routinely. + +2. Create/update GitHub repositories under [MetOffice](https://github.com/MetOffice) + - [ ] `svn:um/main` (@trunk) → [um](https://github.com/MetOffice/um) (@trunk) + - [ ] `svn:um/aux` (@trunk) → [um_aux](https://github.com/MetOffice/um_aux) (@trunk) + - [ ] `svn:um/meta` (@trunk) → [um_meta](https://github.com/MetOffice/um_meta) (@trunk) + - [ ] `svn:um/doc` (@trunk) → [um_doc](https://github.com/MetOffice/um_doc) (@trunk) + - [ ] Compress/convert large graphics in the repository consulting the authors, if possible. + - [ ] Implement GitHub Action to build/deploy docs: test using [texlive docker image](https://hub.docker.com/r/texlive/texlive/tags?name=2018) + - [ ] `svn:um/mule` (@trunk) → [mule](https://github.com/MetOffice/mule) (@trunk) + - [ ] Contact [metomi](https://github.com/metomi/mule) owner to deprecate their repository. + - [ ] `svn:gcom/main` (@trunk) → [gcom](https://github.com/MetOffice/gcom) (@trunk) + - [ ] `svn:jules/main` (@trunk) → [jules](https://github.com/MetOffice/jules) (@trunk) + - [ ] Check licence agreements, if going public. + - [ ] Plan to migrate JULES documentation here. + - [ ] `svn:socrates/main` (@trunk) → [socrates](https://github.com/MetOffice/socrates) (@trunk) + - [ ] `svn:utils/shumlib` (@trunk) → [shumlib](https://github.com/MetOffice/shumlib) (@trunk) + - [ ] - [ ] Contact [metomi](https://github.com/metomi/shumlib) owner to deprecate their repository. + - [ ] `svn:ukca/main` (@trunk) → [ukca](https://github.com/MetOffice/ukca) (@trunk) + - [ ] `svn:monc/casim` (@trunk) → [casim](https://github.com/MetOffice/casim) (@trunk) + - [ ] `svn:moci/main` (@trunk) → [moci](https://github.com/MetOffice/moci) (@trunk) + - [ ] `svn:lfric/LFRic` (@trunk) → [lfric_core](https://github.com/MetOffice/lfric_core) (@trunk) + - [ ] Repository already exists: ask for admin access to the repository. + - [ ] `svn:lfric_apps/main` (@trunk) → [lfric_apps](https://github.com/MetOffice/lfric_apps) (@trunk) + - [ ] Propose `lfric_core` as a submodule in this repository? + - [ ] Check `config.json` is correct and up-to-date with MOSRS revisions. + +3. General updated for all repositories + - [ ] All _private_ with specific team access only. + - [ ] `README.md` and Repository Description. + - [ ] `LICENCE` + - [ ] `CONTRIBUTING.md` (CLA) + - [ ] Correct Copyright statements, particularly in planned open-source repositories. + +4. Testbed + - [ ] Create additional `main` branches + - [ ] Test Simulation System workflows with GitHub + - [ ] Make sure tags/releases are still functional + - [ ] Update Working practices + - [ ] Allow limited number of developers to test their workflows + - [ ] Add/update Templates + +5. During SRS freeze + - [ ] Merge `trunk` in _future_ default (`main`) and delete `trunk` + +6. Update communication plan + - [ ] Update [Simulation Systems Discussion](https://github.com/MetOffice/simulation-systems/discussions/337) + - [ ] Engage in [Simulation Systems Q&A](https://github.com/MetOffice/simulation-systems/discussions/categories/questions-and-answers) + + +## FAQ + +
+How long it takes to convert svn repo to Git? + +The listing below shows the time taken to convert trunk-only branches and +attach svn tags to the Git repository locally using `ssd_svn2git,sh` script. + + $ tail -n1 *.log + ==> 20250330T025505_casim.log <== + 2025-03-30 02:56:25 Done: casim in 00:01:20 + + ==> 20250330T025625_moci.log <== + 2025-03-30 02:59:42 Done: moci in 00:03:17 + + ==> 20250330T025942_jules.log <== + 2025-03-30 03:14:50 Done: jules in 00:15:08 + + ==> 20250330T031450_socrates.log <== + 2025-03-30 03:17:04 Done: socrates in 00:02:14 + + ==> 20250330T031704_ukca.log <== + 2025-03-30 03:18:06 Done: ukca in 00:01:02 + + ==> 20250330T031806_shumlib.log <== + 2025-03-30 03:19:40 Done: shumlib in 00:01:34 + + ==> 20250330T031940_mule.log <== + 2025-03-30 03:24:19 Done: mule in 00:04:39 + + ==> 20250330T032419_um_aux.log <== + 2025-03-30 03:29:52 Done: um_aux in 00:05:33 + + ==> 20250330T032952_um_doc.log <== + 2025-03-30 03:42:31 Done: um_doc in 00:12:39 + + ==> 20250330T034231_um_meta.log <== + 2025-03-30 03:46:47 Done: um_meta in 00:04:16 + + ==> 20250330T034647_um.log <== + 2025-03-30 05:04:04 Done: um in 01:17:17 + + ==> 20250330T050404_gcom.log <== + 2025-03-30 05:06:17 Done: gcom in 00:02:13 + + ==> 20250330T050617_lfric_apps.log <== + 2025-03-30 05:11:44 Done: lfric_apps in 00:05:27 + + ==> 20250330T051145_lfric_core.log <== + 2025-03-30 05:48:32 Done: lfric_core in 00:36:48 +
+ +
+What happens to a tag if the corresponding branch gets deleted? +A branch is simply a way to track a collection of commits. The tag and commit +would stull exist if the branch is deleted. Ref: https://github.com/orgs/community/discussions/23918. +
+ +
+When will the developers get access to the GitHub repositories +Please refer to the timeline in [Simulation Systems Discussion](https://github.com/MetOffice/simulation-systems/discussions/337). +We will announce the opportunity to participate during the later part of migration testbed. +
diff --git a/git-migration/config.json b/git-migration/config.json new file mode 100644 index 0000000..f3f31d1 --- /dev/null +++ b/git-migration/config.json @@ -0,0 +1,513 @@ +{ + "svn_prefix": "https://code.metoffice.gov.uk/svn", + "gh_org": "MetOffice", + "repo": [ + { + "name": "casim", + "description": "The Cloud and AeroSol Interacting Microphysics (CASIM) component of the Met Office-NERC Cloud (MONC) Model", + "trunk": "monc/casim", + "tags": [ + { "apps2.1": 11730 }, + { "um10.7": 3236 }, + { "um10.8": 3236 }, + { "um10.9": 3813 }, + { "um11.0": 4628 }, + { "um11.1": 5298 }, + { "um11.2": 5666 }, + { "um11.3": 6341 }, + { "um11.4": 6341 }, + { "um11.5": 6341 }, + { "um11.6": 7617 }, + { "um11.7": 7617 }, + { "um11.8": 8361 }, + { "um11.9": 8361 }, + { "um12.0": 8805 }, + { "um12.1": 9277 }, + { "um12.2": 9443 }, + { "um13.0": 9796 }, + { "um13.1": 9796 }, + { "um13.2": 10172 }, + { "um13.3": 10378 }, + { "um13.3.1": 10515 }, + { "um13.4": 10614 }, + { "um13.5": 10951 }, + { "um13.6": 11231 }, + { "um13.7": 11589 }, + { "um13.8": 11730 }, + { "vn0.1": 1111 }, + { "vn0.2": 3236 }, + { "vn0.2.1": 3415 }, + { "vn0.2.2": 3813 }, + { "vn0.3": 4628 }, + { "vn0.3.1": 5298 }, + { "vn0.3.2": 5666 }, + { "vn0.3.3": 6341 }, + { "vn0.3.4": 7617 }, + { "vn0.4": 8361 }, + { "vn0.5": 8805 }, + { "vn0.6": 9277 }, + { "vn1.0": 9796 }, + { "vn1.1": 10172 }, + { "vn1.2": 10378 }, + { "vn1.3": 10614 }, + { "vn1.4": 11231 }, + { "vn1.5": 11589 } + ] + }, + { + "name": "moci", + "description": "Met Office Coupling Infrastructure", + "trunk": "moci/main", + "tags": [ + { "postproc_1.0": 371 }, + { "postproc_1.1": 681 }, + { "postproc_1.2": 1177 }, + { "postproc_2.0": 1388 }, + { "postproc_2.1": 1989 }, + { "postproc_2.2": 2381 }, + { "postproc_2.3": 3154 }, + { "postproc_2.4": 4419 }, + { "postproc_2.5": 5515 }, + { "drivers_0.1": 1488 }, + { "drivers_0.2": 1775 }, + { "drivers_0.2.1": 1966 }, + { "drivers_0.3": 1999 }, + { "drivers_1.0": 2408 }, + { "drivers_1.1": 2578 }, + { "drivers_1.2": 2896 }, + { "drivers_1.3": 3105 }, + { "drivers_1.4": 3443 }, + { "drivers_2.0": 3808 }, + { "drivers_2.0.1": 3926 }, + { "drivers_2.0.2": 5232 }, + { "GO6_fcm_make_configs_1.0": 1352 }, + { "GO6_fcm_make_configs_1.1": 2065 } + ] + }, + { + "name": "jules", + "description": "The Joint UK Land Environment Simulator", + "trunk": "jules/main", + "tags": [ + { "apps2.1": 29791 }, + { "um8.0": 3 }, + { "um8.1": 3 }, + { "um8.2": 7 }, + { "um8.3": 22 }, + { "um8.4": 50 }, + { "um8.5": 64 }, + { "um8.6": 79 }, + { "um9.0": 86 }, + { "um9.1": 98 }, + { "um9.2": 129 }, + { "um10.0": 131 }, + { "um10.1": 692 }, + { "um10.2": 1709 }, + { "um10.3": 2461 }, + { "um10.4": 3197 }, + { "um10.5": 4285 }, + { "um10.6": 5320 }, + { "um10.6.1": 5826 }, + { "um10.7": 6925 }, + { "um10.8": 8484 }, + { "um10.9": 9522 }, + { "um11.0": 10836 }, + { "um11.1": 12251 }, + { "um11.2": 13156 }, + { "um11.3": 14197 }, + { "um11.4": 15100 }, + { "um11.5": 15927 }, + { "um11.6": 16960 }, + { "um11.7": 17881 }, + { "um11.8": 18812 }, + { "um11.9": 19395 }, + { "um12.0": 20512 }, + { "um12.1": 21512 }, + { "um12.2": 22411 }, + { "um13.0": 23518 }, + { "um13.1": 24383 }, + { "um13.2": 25256 }, + { "um13.3": 25896 }, + { "um13.3.1": 26366 }, + { "um13.4": 26897 }, + { "um13.5": 28091 }, + { "um13.6": 28692 }, + { "um13.7": 29181 }, + { "um13.8": 29791 }, + { "vn3.1": 11 }, + { "vn3.2": 27 }, + { "vn3.3": 52 }, + { "vn3.4": 65 }, + { "vn3.4.1": 67 }, + { "vn4.0": 101 }, + { "vn4.1": 131 }, + { "vn4.2": 793 }, + { "vn4.3": 1511 }, + { "vn4.3.1": 1709 }, + { "vn4.3.2": 1978 }, + { "vn4.4": 2461 }, + { "vn4.5": 3197 }, + { "vn4.6": 4285 }, + { "vn4.7": 5320 }, + { "vn4.8": 6925 }, + { "vn4.9": 8484 }, + { "vn5.0": 9522 }, + { "vn5.1": 10836 }, + { "vn5.2": 12251 }, + { "vn5.3": 13249 }, + { "vn5.4": 14197 }, + { "vn5.5": 15100 }, + { "vn5.6": 15927 }, + { "vn5.7": 16960 }, + { "vn5.8": 17881 }, + { "vn5.9": 18812 }, + { "vn6.0": 19395 }, + { "vn6.1": 20512 }, + { "vn6.2": 21512 }, + { "vn6.3": 22411 }, + { "vn7.0": 23518 }, + { "vn7.1": 24383 }, + { "vn7.2": 25256 }, + { "vn7.3": 25896 }, + { "vn7.4": 26897 }, + { "vn7.5": 28091 }, + { "vn7.6": 28692 }, + { "vn7.7": 29181 }, + { "vn7.8": 29791 } + ] + }, + { + "name": "socrates", + "description": "Suite of Community Radiative Transfer codes based on Edwards and Slingo", + "trunk": "socrates/main", + "tags": [ + { "apps2.1": 1745 }, + { "um10.1": 1 }, + { "um10.2": 11 }, + { "um10.3": 36 }, + { "um10.4": 99 }, + { "um10.5": 158 }, + { "um10.6": 206 }, + { "um10.6.1": 206 }, + { "um10.7": 293 }, + { "um10.8": 358 }, + { "um10.9": 409 }, + { "um11.0": 409 }, + { "um11.1": 469 }, + { "um11.2": 506 }, + { "um11.3": 666 }, + { "um11.4": 735 }, + { "um11.5": 810 }, + { "um11.6": 855 }, + { "um11.7": 923 }, + { "um11.8": 958 }, + { "um11.9": 989 }, + { "um12.0": 1045 }, + { "um12.1": 1126 }, + { "um12.2": 1173 }, + { "um13.0": 1226 }, + { "um13.1": 1273 }, + { "um13.2": 1348 }, + { "um13.3": 1414 }, + { "um13.3.1": 1427 }, + { "um13.4": 1475 }, + { "um13.5": 1563 }, + { "um13.6": 1642 }, + { "um13.7": 1673 }, + { "um13.8": 1741 } + ] + }, + { + "name": "ukca", + "description": "United Kingdom Chemistry and Aerosols", + "trunk": "ukca/main", + "tags": [ + { "apps2.1": 5509 }, + { "um13.0": 264 }, + { "um13.1": 611 }, + { "um13.2": 1218 }, + { "um13.3": 1718 }, + { "um13.3.1": 1718 }, + { "um13.4": 2293 }, + { "um13.5": 3314 }, + { "um13.6": 4075 }, + { "um13.7": 4898 }, + { "um13.8": 5509 } + ] + }, + { + "name": "shumlib", + "description": "Shared UM Libraries from the Met Office Unified Model", + "trunk": "utils/shumlib", + "tags": [ + { "apps2.1": 7655 }, + { "um10.8": 874 }, + { "um10.9": 1223 }, + { "um11.0": 1740 }, + { "um11.1": 2192 }, + { "um11.2": 2503 }, + { "um11.3": 2806 }, + { "um11.4": 3267 }, + { "um11.5": 3697 }, + { "um11.6": 4008 }, + { "um11.7": 4008 }, + { "um11.8": 4831 }, + { "um11.9": 4831 }, + { "um12.0": 5320 }, + { "um12.1": 5658 }, + { "um12.2": 5864 }, + { "um13.0": 6240 }, + { "um13.1": 6514 }, + { "um13.2": 6731 }, + { "um13.3": 6823 }, + { "um13.3.1": 7055 }, + { "um13.4": 7198 }, + { "um13.5": 7324 }, + { "um13.6": 7464 }, + { "um13.7": 7655 }, + { "um13.8": 7655 }, + { "2017.06.1": 874 }, + { "2017.10.1": 1223 }, + { "2018.01.1": 1552 }, + { "2018.02.1": 1740 }, + { "2018.06.1": 2192 }, + { "2018.10.1": 2503 }, + { "2019.02.1": 2806 }, + { "2019.10.1": 3697 }, + { "2020.01.1": 4008 }, + { "2020.11.1": 4831 }, + { "2021.03.1": 4831 }, + { "2021.07.1": 5320 }, + { "2021.10.1": 5658 }, + { "2022.02.1": 5864 }, + { "2022.11.1": 6514 }, + { "2023.03.1": 6731 }, + { "2023.06.1": 6903 }, + { "2023.08.1": 7055 }, + { "2023.10.1": 7198 }, + { "2024.03.1": 7373 }, + { "2024.11.1": 7655 } + ] + }, + { + "name": "mule", + "description": "Python packages providing an interface to the various files used and produced by the Met Office Unified Model (UM)", + "trunk": "um/mule", + "tags": [ + { "vn1.0": 13178 }, + { "vn1.1": 16161 }, + { "vn1.2": 19778 }, + { "vn1.3": 26444 }, + { "vn1.4": 34062 }, + { "2017.06.1": 40541 }, + { "2017.08.1": 43028 }, + { "2018.07.1": 57375 }, + { "2019.01.1": 64490 }, + { "2020.01.1": 79553 }, + { "2022.05.1": 109242 }, + { "2022.07.1": 111345 }, + { "2023.08.1": 119685 }, + { "2024.11.1": 126922 } + ] + }, + { + "name": "um_aux", + "description": "Auxiliary files for the Met Office Unified Model (UM)", + "trunk": "um/aux", + "tags": [ + { "vn1.7": 1 }, + { "vn1.8": 6888 }, + { "vn1.9": 12679 }, + { "vn1.10": 17596 }, + { "vn1.11": 24106 }, + { "vn1.12": 30823 }, + { "vn1.13": 39342 }, + { "vn1.14": 45603 }, + { "vn1.15": 48564 }, + { "vn1.16": 54823 }, + { "vn1.17": 76247 }, + { "vn1.18": 91506 }, + { "vn1.19": 99849 }, + { "vn1.20": 106173 }, + { "vn1.21": 110005 }, + { "vn1.22": 118501 }, + { "vn1.23": 122041 }, + { "vn1.24": 126461 } + ] + }, + { + "name": "um_doc", + "description": "Documentation for the Met Office Unified Model (UM)", + "trunk": "um/doc", + "tags": [ + { "vn9.2": 79 }, + { "vn10.0": 459 }, + { "vn10.1": 3100 }, + { "vn10.2": 7954 }, + { "vn10.3": 13494 }, + { "vn10.4": 18679 }, + { "vn10.5": 25338 }, + { "vn10.6": 29875 }, + { "vn10.7": 35549 }, + { "vn10.8": 41467 }, + { "vn10.9": 46193 }, + { "vn11.0": 51341 }, + { "vn11.1": 56608 }, + { "vn11.2": 61783 }, + { "vn11.3": 67096 }, + { "vn11.4": 72161 }, + { "vn11.5": 76967 }, + { "vn11.6": 82355 }, + { "vn11.7": 87303 }, + { "vn11.8": 92977 }, + { "vn11.9": 96628 }, + { "vn12.0": 100826 }, + { "vn12.1": 105064 }, + { "vn12.2": 107567 }, + { "vn13.0": 112184 }, + { "vn13.1": 114606 }, + { "vn13.2": 117036 }, + { "vn13.3": 119231 }, + { "vn13.4": 120793 }, + { "vn13.5": 123441 }, + { "vn13.6": 125228 }, + { "vn13.7": 127063 } + ] + }, + { + "name": "um_meta", + "description": "Metadata for the Met Office Unified Model (UM)", + "trunk": "um/meta", + "tags": [] + }, + { + "name": "um", + "description": "The Met Office Unified Model (UM)", + "trunk": "um/main", + "tags": [ + { "vn9.2": 1 }, + { "vn9.2.1": 32 }, + { "vn9.2.2": 94 }, + { "vn10.0": 233 }, + { "vn10.1": 2765 }, + { "vn10.2": 7660 }, + { "vn10.3": 13189 }, + { "vn10.3.1": 13704 }, + { "vn10.4": 18260 }, + { "vn10.4.1": 20043 }, + { "vn10.5": 24655 }, + { "vn10.6": 29689 }, + { "vn10.6.1": 31236 }, + { "vn10.7": 34978 }, + { "vn10.8": 40732 }, + { "vn10.9": 45850 }, + { "vn11.0": 50973 }, + { "vn11.1": 56263 }, + { "vn11.2": 61261 }, + { "vn11.3": 66700 }, + { "vn11.4": 71790 }, + { "vn11.5": 76551 }, + { "vn11.6": 81992 }, + { "vn11.7": 87191 }, + { "vn11.8": 92349 }, + { "vn11.9": 96576 }, + { "vn12.0": 100515 }, + { "vn12.0.1": 104136 }, + { "vn12.1": 104450 }, + { "vn12.2": 107106 }, + { "vn13.0": 111272 }, + { "vn13.1": 114076 }, + { "vn13.2": 116723 }, + { "vn13.3": 118802 }, + { "vn13.3.1": 119758 }, + { "vn13.4": 120750 }, + { "vn13.5": 123226 }, + { "vn13.6": 124981 }, + { "vn13.7": 127030 }, + { "vn13.8": 128625 } + ] + }, + { + "name": "gcom", + "description": "Communications layer used by the UM and other systems at the Met Office", + "trunk": "gcom/main", + "tags": [ + { "vn4.8": 1 }, + { "vn5.0": 5 }, + { "vn5.1": 41 }, + { "vn5.2": 94 }, + { "vn5.3": 152 }, + { "vn5.4": 212 }, + { "vn5.4.1": 224 }, + { "vn6.0": 257 }, + { "vn6.1": 296 }, + { "vn6.2": 357 }, + { "vn6.3": 408 }, + { "vn6.4": 469 }, + { "vn6.5": 552 }, + { "vn6.6": 622 }, + { "vn6.7": 748 }, + { "vn6.8": 783 }, + { "vn6.9": 822 }, + { "vn7.0": 841 }, + { "vn7.1": 863 }, + { "vn7.2": 927 }, + { "vn7.3": 960 }, + { "vn7.4": 979 }, + { "vn7.5": 1006 }, + { "vn7.6": 1040 }, + { "vn7.7": 1096 }, + { "vn7.7.1": 1106 }, + { "vn7.8": 1147 }, + { "vn7.9": 1166 }, + { "vn7.9.1": 1175 }, + { "vn8.0": 1181 }, + { "vn8.1": 1215 }, + { "vn8.1.1": 1226 }, + { "vn8.2": 1251 }, + { "vn8.3": 1288 }, + { "vn8.4": 1386 } + ] + }, + { + "name": "lfric_apps", + "description": "LFRic-based science applications for Momentum", + "trunk": "lfric_apps/main", + "tags": [ + { "um13.5": 782 }, + { "um13.6": 2409 }, + { "um13.7": 4909 }, + { "vn1.0": 782 }, + { "vn1.1": 2409 }, + { "vn1.2": 4909 }, + { "vn1.2.1": 5299 }, + { "vn2.0": 5732 }, + { "vn2.0.1": 5838 }, + { "vn2.1": 8283 } + ] + }, + { + "name": "lfric_core", + "description": "Software infrastructure that supports the Momentum Atmosphere model", + "trunk": "lfric/LFRic", + "tags": [ + { "um12.0": 30137 }, + { "um12.1": 32661 }, + { "um12.2": 34263 }, + { "um13.0": 37211 }, + { "um13.1": 39455 }, + { "um13.2": 41804 }, + { "um13.3": 43752 }, + { "um13.4": 46556 }, + { "lfricinputs_2022.08.1": 37503 }, + { "apps_migrate": 49954 }, + { "apps1.0": 50045 }, + { "core1.1": 50658 }, + { "core1.2": 51381 }, + { "core1.2.1": 51467 }, + { "core2.0": 51632 }, + { "core2.1": 52315 } + ] + } + ] +} diff --git a/git-migration/ssd_svn2git.sh b/git-migration/ssd_svn2git.sh new file mode 100755 index 0000000..a88830a --- /dev/null +++ b/git-migration/ssd_svn2git.sh @@ -0,0 +1,219 @@ +#!/bin/bash --login +set -euo pipefail + +# ---------------------------------------------------------------------------- +# Script Metadata +# ---------------------------------------------------------------------------- +# This script is designed to migrates Met Office Simulation Systems SVN +# repositories to GitHub. + +# ---------------------------------------------------------------------------- +# Functions +# ---------------------------------------------------------------------------- + +usage() { + cat < +EOF +} + +check_apps() { + local status=0 + echo "--------------------------------------------------------------------" + for app in "$@"; do + if ! command -V "$app" 2>/dev/null; then + echo "** $app is not installed. Please install $app to run this script." + ((status++)) + fi + done + echo "--------------------------------------------------------------------" + if ((status > 0)); then return 127; fi +} + +validate_repo() { + local repo_name=$1 + if ! jq -e ".repo[] | select(.name == \"$repo_name\")" "$CONFIG_FILE" >/dev/null; then + echo "Repository: '$repo_name' not found in $CONFIG_FILE" + exit 1 + fi +} + +process_repo() { + local repo_name=$1 + local svn_url=$2 + local repo_metadata=$3 + SECONDS=0 # start timer + + echo "$(date +'%F %T') Processing: $repo_name" + + if [[ -d $repo_name ]]; then + echo "-- Sync commits from head of trunk" + pushd "$repo_name" >/dev/null + gitlify resync -t + popd >/dev/null + else + echo "-- Convert SVN to Git" + gitlify migrate -d enquiries@metoffice.gov.uk -x -t -y "$svn_url" "$repo_name" + fi + + pushd "$repo_name" >/dev/null + attach_tags "$repo_metadata" + update_github "$repo_name" "$repo_metadata" + deltaT=$(TZ=UTC0 printf '%(%H:%M:%S)T\n' $SECONDS) + echo "$(date +'%F %T') Done: $repo_name in $deltaT" + popd >/dev/null +} + +attach_tags() { + local repo_metadata=$1 + echo "-- Attach tags" + local tags + tags=$(echo "$repo_metadata" | jq -c '.tags[]') + for tag in $tags; do + local tag_name tag_value + tag_name=$(echo "$tag" | jq -r 'keys[]') + tag_value=$(echo "$tag" | jq -r '.[keys[]]') + echo " Tag: $tag_name, Revision: $tag_value" + git tag "$tag_name" -f "$(git svn find-rev "r${tag_value}")" + done +} + +update_github() { + local repo_name=$1 + local repo_metadata=$2 + + if ((UPDATE_GITHUB == 0)); then + return + fi + + echo "-- Create remote GitHub repository, if necessary" + git ls-remote -h "${GH_PREFIX}/${repo_name}" || { + local description + description=$(echo "$repo_metadata" | jq -r '.description') + gh repo create "${GH_ORG}/${repo_name}" --description="$description" --private --source=. + } + + echo "-- Push local branch to GitHub" + git config --get remote.origin.url || git remote add origin "${GH_PREFIX}/${repo_name}.git" + git push --set-upstream origin trunk + + echo "-- Push tags to GitHub" + git push --tags +} + +initialize_environment() { + mkdir -p "${WORKDIR}/logs" + pushd "$WORKDIR" + + git -C gitlify pull 2>/dev/null || gh repo clone MetOffice/gitlify + export PATH="$PATH:${WORKDIR}/gitlify" + + check_apps gitlify jq gh + printf "WORKDIR: %s\nCONFIG_FILE: %s\nUPDATE_GITHUB: %s\n" \ + "$WORKDIR" "$CONFIG_FILE" "$UPDATE_GITHUB" + echo "--------------------------------------------------------------------" +} + +process_repositories() { + local process_ids=() + local max_jobs + max_jobs=${BATCH:-$(nproc)} + TIMESTAMP=$(date +'%Y%m%dT%H%M%S') + while read -r repo; do + local repo_name trunk svn_url + repo_name=$(jq -r '.name' <<<"$repo") + [[ -z "$REPO_NAME" || "$REPO_NAME" == "$repo_name" ]] || continue + trunk=$(jq -r '.trunk' <<<"$repo") + svn_url="${SVN_PREFIX}/${trunk}" + if (( FROM_MIRROR )); then + svn_url="svn://fcm1/${trunk/$repo_name/$repo_name.xm_svn}" + fi + if (( DRY_RUN )); then + printf "%-.52s %s\n" \ + "$svn_url ....................................." \ + "${GH_PREFIX}/${repo_name}" + continue + fi + echo "$(date +'%F %T') Processing: $repo_name" + process_repo "$repo_name" "$svn_url" "$repo" \ + &> "${WORKDIR}/logs/${TIMESTAMP}_${repo_name}.log" & + process_ids+=("$!") + + # Wait for background jobs if max_jobs are running + if (( ${#process_ids[@]} >= max_jobs )); then + for pid in "${process_ids[@]}"; do + wait "$pid" + done + process_ids=() + fi + done < <(jq -c '.repo[]' "$CONFIG_FILE") + + # Wait for any remaining background jobs + for pid in "${process_ids[@]}"; do + wait "$pid" + done +} + +# ---------------------------------------------------------------------------- +# Main Script +# ---------------------------------------------------------------------------- + +OPTS=$(getopt \ + -o hdmnuj:c:r:w: \ + -l help,debug,from-mirror,list-repos,update,parallel:,config:,repository:,workdir: \ + -n "${0##*/}" -- "$@") || { usage; exit 1; } +eval set -- "$OPTS" + +SCRIPT_DIR=$(dirname -- "$(readlink -fe -- "$0")") +export CONFIG_FILE="${SCRIPT_DIR}/config.json" +export WORKDIR="${DATADIR}/git-migration" +UPDATE_GITHUB=0 +FROM_MIRROR=0 +REPO_NAME= +DRY_RUN= + +while true; do + case "$1" in + -h | --help) usage; exit 0 ;; + -d | --debug) PS4='L${LINENO}: '; set -x; shift ;; + -m | --from-mirror) FROM_MIRROR=1; shift ;; + -n | --list-repos) DRY_RUN=1; echo "$(date +'%F %T') Dry run"; shift ;; + -u | --update) UPDATE_GITHUB=1; shift ;; + -j | --parallel) export BATCH="$2"; shift 2 ;; + -c | --config) CONFIG_FILE=$(readlink -fe "$2"); shift 2 ;; + -r | --repository) REPO_NAME="$2"; validate_repo "$REPO_NAME"; shift 2 ;; + -w | --workdir) WORKDIR=$(readlink -fe "$2"); shift 2 ;; + --) shift; break ;; + *) break ;; + esac +done + +initialize_environment +SVN_PREFIX=$(jq -r '.svn_prefix' "$CONFIG_FILE") +GH_ORG=$(jq -r '.gh_org' "$CONFIG_FILE") +GH_PREFIX="git@github.com:${GH_ORG}" + +process_repositories +echo "$(date +'%F %T') Done."