Skip to content

Commit 39b3f3f

Browse files
inaki-amatriadaniel-otero
inaki-amatria
and
daniel-otero
committed
Add our upstream patches
Co-authored-by: Daniel Otero <[email protected]>
1 parent 3b3c591 commit 39b3f3f

21 files changed

+10809
-0
lines changed

‎codee/HowToMergeUpstream.md

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# How to merge `tree-sitter-fortran`
2+
3+
This document explains the process for updating our downstream copy of
4+
`tree-sitter-fortran` repository with the latest changes from the upstream
5+
repository, ensuring Codee remains up-to-date with new features, fixes, and
6+
improvements.
7+
8+
## 0. Prerequisites
9+
10+
Before merging upstream changes, ensure that the official `tree-sitter-fortran`
11+
repository is added as a Git remote and updated. If it is not already
12+
configured, you can set it up with the following commands:
13+
14+
```bash
15+
$ git remote add tree-sitter-fortran [email protected]:stadelmanma/tree-sitter-fortran.git
16+
$ git fetch tree-sitter-fortran master
17+
```
18+
19+
Additionally, confirm that your local working directory is clean, with no
20+
uncommitted changes, to avoid conflicts during the merge process.
21+
22+
## 1. Run the update script
23+
24+
Run the [`codee/merge-upstream.sh`](/codee/merge-upstream.bash) script to check
25+
whether our [`codee/patches`](/codee/patches) are synchronized with our
26+
downstream repository. If the patches are outdated, you must update and merge
27+
them to ensure compatibility before proceeding with the main merge process.
28+
29+
Once the patches are verified and up-to-date, the script transitions to a second
30+
stage. In this stage, the script applies the patches to our downstream project,
31+
incorporating the latest upstream changes. This process involves using `git am`
32+
to apply each patch from the `codee/patches` directory. During this step,
33+
conflicts may arise that you will need to resolve manually.
34+
35+
To identify the source of conflicts, use `git am --show-current-patch=diff`.
36+
This command highlights the specific patch causing the issue, helping you assess
37+
and resolve the conflict efficiently. After addressing the conflict, continue
38+
applying the remaining patches until all patches have been successfully applied.
39+
40+
After all patches are applied, your current branch should resemble the
41+
following:
42+
43+
```bash
44+
$ git log --oneline
45+
92081e66c3b5 Baz
46+
4b5ccc6f50e4 Bar
47+
6858ad4b7960 Foo
48+
# ...
49+
dbdb4564d47c Merge tree-sitter-fortran/master
50+
# ...
51+
```
52+
53+
The commits following `Merge tree-sitter-fortran/master` represent the updated
54+
versions of the patches with conflicts resolved and context synchronized with
55+
the latest upstream changes.
56+
57+
At this point, regenerate the patches to reflect the latest changes by running
58+
the appropriate script. Once regenerated, commit these changes as a fixup commit
59+
to keep the process organized. After updating the patches, you can squash all
60+
the commits created by `git am`, as the relevant changes are now captured within
61+
the patches themselves. There is no need to revisit all the upstream changes;
62+
instead, focus the review on the updates made to the patches in the fixup
63+
commit, ensuring that only the necessary adjustments are highlighted.
64+
65+
## 2. Prepare the changes for review
66+
67+
At this stage, it is time to prepare your branch for review. Take into
68+
consideration that a clean and well-organized commit history will significantly
69+
improve the review process and make it easier to identify meaningful changes.
70+
71+
For example, the commit that updates the conflicting patches can be separated.
72+
This commit often contains updates to the context of the patches, but we are
73+
only interested in those where conflicts actually occurred. You can either
74+
squash these updates into the merge commit or leave them as a separate fixup
75+
commit. If you choose the latter, make sure to clearly label these commits with
76+
a message like "Do not review" so reviewers can easily skip them.
77+
78+
Additionally, ensure that each commit has a clear and concise message, ideally
79+
with references to relevant upstream changes. Once you have organized and
80+
squashed any unnecessary commits, your branch should be ready for review.
81+
Reviewers can then focus solely on the key changes that align with the upstream
82+
updates, without being bogged down by irrelevant or redundant commits.
83+
84+
Furthermore, during the process, you may identify commits that can be merged
85+
into the downstream project prior to the actual upstream merge. These commits
86+
could include improvements that help streamline the review process, such as
87+
migrating away from deprecated APIs that will be removed in the merge. In such
88+
cases, it is best to separate these changes and submit them as a distinct
89+
pull request, to be merged before the main upstream merge. This approach keeps
90+
the integration process clean, ensures clarity, and avoids potential
91+
complications by addressing non-merge-related improvements ahead of time.
92+
93+
To further ease the review process, consider submitting a branch with just the
94+
merge commit. Then, create a separate branch that targets this merge branch,
95+
containing only the fixup commits. This will allow reviewers to focus
96+
exclusively on the necessary adjustments, without having to look through the
97+
upstream commit history.
98+
99+
## 3. Cleanup and pushing the final changes
100+
101+
Once the pull request has been fully reviewed and you have addressed all the
102+
review comments, it is time to finalize the changes and prepare them for
103+
merging. The goal here is to ensure that all necessary changes are
104+
well-organized and that the commit history is clean and concise.
105+
106+
First, take a look at any post-merge commits and decide how to handle them. You
107+
may place them before the merge commit if applicable, or squash them into the
108+
merge commit, making sure to retain the squashed commit's message in the merge
109+
commit's message.
110+
111+
Assuming the merge is already passing tests in the
112+
`feature/UpgradeTreeSitterFortranAuto` branch, the workflow for this step should
113+
look something like this:
114+
115+
```bash
116+
# Branch off feature/UpgradeTreeSitterFortranAuto to update it
117+
$ git switch -c feature/UpdatedUpgradeTreeSitterFortranAuto feature/UpgradeTreeSitterFortranAuto
118+
# Merge changes from origin/codee and solve any potential conflicts with newer
119+
# codee changes
120+
$ git merge origin/codee
121+
# Create a final branch to clean things up
122+
$ git switch -c feature/FinalUpgradeTreeSitterFortranAuto origin/codee
123+
# Cherry-pick commits to place before the merge commit
124+
$ git cherry-pick <commit to place before merge>
125+
# Re-run the intended merge
126+
$ git merge --no-commit <merge commit>
127+
# If merging unrelated histories, you may need to remove deleted files
128+
$ for f in $(git status --porcelain | grep "UD" | cut -d " " -f 2); do git rm $f; done
129+
# Don't worry about conflicts; simply restore the original branch
130+
$ git restore --source feature/UpgradeTreeSitterFortranAuto .
131+
# Stage everything
132+
$ git add .
133+
# Commit to finish the merge, including messages for the remaining post-merge
134+
# commits that were not moved
135+
$ git commit
136+
# Optionally add commits that need to be placed after the merge
137+
$ git cherry-pick <commit to place after merge>
138+
# Optionally check that the new branch matches the updated branch
139+
$ git diff feature/FinalUpgradeTreeSitterFortranAuto feature/UpdatedUpgradeTreeSitterFortranAuto
140+
# Push the final branch
141+
$ git push feature/FinalUpgradeTreeSitterFortranAuto origin/feature/FinalUpgradeTreeSitterFortranAuto
142+
```
143+
144+
Once you have pushed your final branch, open a pull request targeting `codee`
145+
with `feature/FinalUpgradeTreeSitterFortranAuto` and request any necessary
146+
approvals. This step should be a formality as everything has already been
147+
reviewed in the previous pull request.
148+
149+
After the pull request is approved and tests pass, push it to `codee`:
150+
151+
```bash
152+
$ git push origin feature/FinalUpgradeTreeSitterFortranAuto:codee
153+
```

‎codee/merge-upstream.bash

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
################################################################################
6+
# Description
7+
################################################################################
8+
# This script is here to assist the maintenance of the upstream patches that are
9+
# required to support this project.
10+
#
11+
# The script is divided in 2 stages:
12+
# - Stage 1 is to ensure that the current stored patches are exactly the same
13+
# to what is already commited in the repo.
14+
# - Stage 2 is to re-apply the patches over a merge-commit on the target branch
15+
# and ensure that they are still valid, update them if necessary and change
16+
# the stored patches accordingly.
17+
18+
################################################################################
19+
# Functions
20+
################################################################################
21+
function _info() {
22+
>&2 echo "## ${@}"
23+
}
24+
25+
function _error() {
26+
>&2 echo "error: ${@}"
27+
}
28+
29+
function _git() {
30+
>&2 echo "+ git ${@}"
31+
git -C "${REPO_DIR}" "${@}"
32+
}
33+
34+
function cleanup() {
35+
rm -r "${TMP_DIR}"
36+
_git switch "${STARTING_BRANCH}"
37+
_git branch -D "${TESTING_BRANCH}"
38+
}
39+
40+
function format_patch_help() {
41+
echo "rm '${PATCHES_DIR}'/* && git format-patch --no-signature --keep-subject --zero-commit --output-directory '${PATCHES_DIR}' '${INITIAL_REF}'..HEAD"
42+
}
43+
44+
################################################################################
45+
# Globals
46+
################################################################################
47+
# The path to this script parent folder
48+
SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
49+
# The path to the root of the git repo
50+
REPO_DIR="${SCRIPT_DIR}/.."
51+
# The path to the dir with all the upstream patches
52+
PATCHES_DIR="${REPO_DIR}/codee/patches"
53+
# The starting branch in case we want to go back
54+
STARTING_BRANCH="$(_git branch --show-current)"
55+
# The branch that we will use to detect that the first stage was done correctly
56+
TARGET_BRANCH="feature/UpgradeTreeSitterFortranAuto"
57+
# Name of a temporary branch that we can delete later
58+
TESTING_BRANCH="merge-upstream-testing-${RANDOM}"
59+
# The name of the upstream git remote repo
60+
MERGE_WITH="${MERGE_WITH:-tree-sitter-fortran/master}"
61+
# List of git refspecs for the upstream code excluding downstream
62+
DOWNSTREAM_PATHS=(:/ :^codee :^src/tree_sitter :^src/grammar.json :^src/node-types.json :^src/parser.c)
63+
64+
################################################################################
65+
# Script
66+
################################################################################
67+
if [[ $# != 0 ]]; then
68+
>&2 echo "usage: ${0}
69+
70+
Helper script to help merge (and verify our changes in) upstream code.
71+
72+
This script parameters aren't supposed to be modified, so they are coded as
73+
variables inside the script."
74+
exit 1
75+
fi
76+
77+
if [[ "${STARTING_BRANCH}" != "${TARGET_BRANCH}" ]]; then
78+
_info "Stage 1: Validate that the existing patches represent the current state
79+
- First, create a new '${TESTING_BRANCH}' branch to perform the validation
80+
- On that branch, reset the upstream folders to the upstream version
81+
- Apply all the stored patches and check if they match the original branch
82+
- If so, proceed to the second stage
83+
- If not, give instructions on how to fix the issue
84+
"
85+
STAGE2=false
86+
else
87+
_info "Stage 2: Finish the upstream update
88+
- It is assumed that the first stage finished successfully.
89+
- We will merge the '${MERGE_WITH}' branch discarding all our changes
90+
- To then apply all the stored patches one by one
91+
- After solving all conflicts, the merge should be ready to:
92+
- Pass tests, prepare for review and finally recreate the patches
93+
"
94+
STAGE2=true
95+
fi
96+
97+
_info "Do you want to proceed (Enter to continue, Ctrl+C to exit)?"
98+
read
99+
100+
# Always ensure that the working tree and the stage are clean
101+
if [ "$(_git status --porcelain)" != "" ]; then
102+
_error "Your working tree and stage must be clean"
103+
exit 1
104+
fi
105+
106+
# In the first stage, sync up the remote branch
107+
if ! _git describe "${MERGE_WITH}" &>/dev/null; then
108+
_error "there is no upstream to merge with ('${MERGE_WITH}'). You can add it
109+
with:
110+
111+
git remote add 'tree-sitter-fortran' [email protected]:stadelmanma/tree-sitter-fortran.git
112+
git fetch tree-sitter-fortran master"
113+
exit 1
114+
fi
115+
116+
if [ "${STAGE2}" == false ]; then
117+
# The target branch shouldn't exist on the first stage
118+
if git show-ref --quiet --verify "refs/heads/${TARGET_BRANCH}"; then
119+
_error "target branch '${TARGET_BRANCH}' must not exist on stage1"
120+
exit 1
121+
fi
122+
123+
UPSTREAM_TARGET=$(_git merge-base HEAD "${MERGE_WITH}")
124+
echo "# Stage 1: Verify that the patches are up to date (on current base)"
125+
_git switch --create "${TESTING_BRANCH}"
126+
else
127+
UPSTREAM_TARGET="${MERGE_WITH}"
128+
129+
echo "# Stage 2: Apply the already verified patches (on '${UPSTREAM_TARGET}')"
130+
_git merge --strategy=ours --no-commit "${UPSTREAM_TARGET}"
131+
fi
132+
133+
# Copy the patches to a place outside git (in case you're changing them in the process)
134+
TMP_DIR="$(mktemp -d)"
135+
cp "${PATCHES_DIR}"/* "${TMP_DIR}"
136+
137+
# Discard all the upstream changes already committed
138+
_git restore --source="${UPSTREAM_TARGET}" --worktree --staged -- "${DOWNSTREAM_PATHS[@]}"
139+
140+
# Create a commit to then, apply the patches stored
141+
if [ "${STAGE2}" == false ]; then
142+
_git commit -m "[merge-upstream] Reverted upstream chances since '${UPSTREAM_TARGET}'"
143+
else
144+
_git commit -m "Merge tree-sitter-fortran/master (to be finished)"
145+
fi
146+
147+
# Store the reference to the start of the format-patch spec
148+
INITIAL_REF="$(_git rev-parse HEAD)"
149+
150+
# Apply all the patches
151+
if ! _git am --3way -k "${TMP_DIR}/"*; then
152+
_error "Patches are not up to date. You'll need to address the issues
153+
and redo the patches before trying again.
154+
155+
$(format_patch_help)
156+
"
157+
exit 3
158+
fi
159+
160+
if [ "${STAGE2}" == false ]; then
161+
# Check if there are any other changes that weren't in the patches
162+
if ! _git diff --quiet HEAD.."${STARTING_BRANCH}"; then
163+
>&2 echo "There are changes that are still not part of the patches:
164+
165+
git diff HEAD..'${STARTING_BRANCH}'
166+
167+
If the changes are new, just commit them as normal. If the changes are
168+
modifications of previous commits, you can try to ammend them automatically:
169+
170+
git diff HEAD..'${STARTING_BRANCH}' | git apply --index
171+
git absorb --base '${INITIAL_REF}'
172+
173+
Review carefully the changes you need to do, and then recreate the patches.
174+
175+
$(format_patch_help)
176+
"
177+
exit 2
178+
fi
179+
180+
cleanup
181+
# Create the target branch for the second stage
182+
_git switch --create "${TARGET_BRANCH}"
183+
184+
echo "
185+
# Stage 1 completed succesfully. You are ready to jump to stage 2:
186+
$0"
187+
else
188+
echo "
189+
# Stage 2 completed succesfully. You're now on your own. Don't forget to:
190+
- Ensure that the patches compile successfully.
191+
- Fix the failing tests.
192+
- Update the final patches to the new version and open a PR to review them.
193+
$(format_patch_help)
194+
- Squash all the patches in the final merge commit!
195+
196+
Good luck!"
197+
fi

‎codee/patches/0001-Add-TypeScript-source-annotation.patch

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
2+
From: Diego Alonso <[email protected]>
3+
Date: Wed, 19 Mar 2025 13:42:50 +0100
4+
Subject: Add TypeScript source annotation
5+
6+
---
7+
grammar.js | 6 ++++++
8+
1 file changed, 6 insertions(+)
9+
10+
diff --git a/grammar.js b/grammar.js
11+
index bcae961..6c99d98 100644
12+
--- a/grammar.js
13+
+++ b/grammar.js
14+
@@ -14,6 +14,12 @@
15+
// since the parser doesn't support the ^ regex token, a using a seq
16+
// might work as long as the label is optional.
17+
//
18+
+
19+
+// This annotation tells typescript where to find the DSL. Useful for LSP
20+
+// navigation and hover documentation
21+
+/// <reference types="tree-sitter-cli/dsl" />
22+
+// @ts-check
23+
+
24+
const PREC = {
25+
ASSIGNMENT: -10,
26+
DEFAULT: 0,

0 commit comments

Comments
 (0)