Changes missing after merge with upstream

[bastianbeischer]

Describe the issue

With this git commit graph:

(base) -> A (weblate)
       -> B (upstream)

where the relevant po file in the repository was modified in both A and B (but in a way which doesn't result in a conflict), modifications in A were done in weblate, modifications in B were done directly in the upstream repository.

weblate merged the remote repository (I have setup the weblate component to use merge style 'merge' and not rebase) and created the following graph:

(base) -> A (weblate old)  >
                             -> M (weblate new)
       -> B (upstream)     >

where M is a merge commit with parents A and B. So far so good.

However in the actual po file in commit M the changes in commit A are gone. The files in the commit M are identical to those in commit B (the git tree is the same). It looks like weblate was using some weird merge strategy which lead to changes in A being ignored?

When manually recreating the merge by creating a new branch on B and merging A into it, I get a different commit M2 which does not have this problem. Here changes from both A and B are present after the merge.

Any ideas how I could debug this further? I have already looked at the git reflog of the weblate internal repository (on the server), but it didn't provide any insights, except that apparently the merge was made with the 'ort' strategy, which seems to be reasonable...

I already tried

• I've read and searched the documentation.
• I've searched for similar filed issues in this repository.

Steps to reproduce the behavior

Might need specific commits in the repository in question, so didn't try to build a reproducer

Expected behavior

No response

Screenshots

No response

Exception traceback

No response

How do you run Weblate?

Docker container

Weblate versions

5.4.2

Weblate deploy checks

No response

Additional context

No response

[bastianbeischer]

Relevant part of git reflog:

7a2bc2d HEAD@{2024-03-18 11:04:27 +0000}: merge weblate-merge-tmp: Fast-forward
b13a2f1 HEAD@{2024-03-18 11:04:27 +0000}: checkout: moving from weblate-merge-tmp to dev
7a2bc2d HEAD@{2024-03-18 11:04:27 +0000}: merge dev: Merge made by the 'ort' strategy.
a1fe442 HEAD@{2024-03-18 11:04:27 +0000}: checkout: moving from dev to weblate-merge-tmp
b13a2f1 HEAD@{2024-03-18 11:04:27 +0000}: checkout: moving from dev to dev

but if I manually checkout a1fe442 (commit B above) and then merge b13a2f1 (commit A above) I get a commit which is different to 7a2bc2d (in terms of git tree content, obviously).

Here is the output of git cat-file -p 7a2bc2d (the merge commit created by weblate):

tree 1db974ca8e48c98bc34a55834e3aeb83cf9adf8a
parent a1fe442a35678bf12d86168b3d05aa71724b7b6d
parent b13a2f1967b9c05f5a93c972a2c297750026010b
author Weblate <[email protected]> 1710759867 +0000
committer Weblate <[email protected]> 1710759867 +0000

Merge branch 'origin/dev' into Weblate.

and here is the result of git cat-file -p my_own_merge_commit:

tree 8cb0f12293cc93bc547dabc124e39970b35db16d
parent a1fe442a35678bf12d86168b3d05aa71724b7b6d
parent b13a2f1967b9c05f5a93c972a2c297750026010b
author Bastian Beranek <[email protected]> 1711453999 +0100
committer Bastian Beranek <[email protected]> 1711453999 +0100

Merge commit 'b13a2f1967b9c05f5a93c972a2c297750026010b' into wip

Note that both parent commits are the same but tree is not the same. And sure enough git diff-tree -p 1db974ca8e48c98bc34a55834e3aeb83cf9adf8a 8cb0f12293cc93bc547dabc124e39970b35db16d results in the diff which was introduced by commit A

[nijel]

This is what Weblate does on merge:

weblate/weblate/vcs/git.py

Lines 207 to 232 in b03e248

	self.delete_branch(tmp)
	# We don't do simple git merge origin/branch as that leads
	# to different merge order than expected and most GUI tools
	# then show confusing diff (not changes done by Weblate, but
	# changes merged into Weblate)
	remote = self.get_remote_branch_name()
	# Create local branch for upstream
	self.execute(["branch", tmp, remote])
	# Checkout upstream branch
	self.execute(["checkout", tmp])
	# Merge current Weblate changes, this can lead to conflict
	cmd = [
	"merge",
	"--message",
	message or f"Merge branch {remote!r} into Weblate",
	]
	if no_ff:
	cmd.append("--no-ff")
	cmd.extend(self.get_gpg_sign_args())
	cmd.append(self.branch)
	self.execute(cmd)
	# Checkout branch with Weblate changes
	self.execute(["checkout", self.branch])
	# Merge temporary branch (this is fast forward so does not create
	# merge commit)
	self.execute(["merge", tmp])

The magic there is only to change the ordering of the parents. Without that, the resulting merge requests on GitHub/GitLab/etc. are showing weird diffs.

If the missing changes are in gettext PO files, it might be https://github.com/WeblateOrg/weblate/blob/main/weblate/examples/git-merge-gettext-po merge driver Weblate uses for these.

[bastianbeischer]

I will try to reproduce this with the custom merge driver

[bastianbeischer]

Indeed, with the custom merge driver I can reproduce the faulty merge

[bastianbeischer]

@nijel found the problem and opened a PR. It's a pretty severe bug if you ask me.

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

[bastianbeischer]

@nijel Thanks. Do you think this warrants a 5.4.3 release? In my opinion it does, because the effect of the bug is that it silently discards changes done in weblate and causes loss of data.

[nijel]

@bastianbeischer I didn't think about that so far, but there are probably enough fixes to make a 5.4.3.