Git LFS upload timeout caused by Git Replace objects
A journey of fixing backup mechanism of my Minecraft server
What’s happened ?
As you can see, the above image describes the overview of my Minecraft server backup mechanism. Thanks to the free 10GB repository size and Unlimited bandwidth of LFS object transmission from GitLab SaaS, the whole backup process was scheduled to run once per day. However, I couldn’t see a new LFS commit in my backup repository at one day.
Troubleshoot process
I ran into my MC server, and checked both the MC and the backup’s container were running normally. Then, I found it had tried to push the new backup commit so many times but all failed. Below shows the first problem I met:
fatal: the requested upstream branch 'origin/main' does not exist
hint:
hint: If you are planning on basing your work on an upstream
hint: branch that already exists at the remote, you may need to
hint: run "git fetch" to retrieve it.
hint:
hint: If you are planning to push out a new local branch that
hint: will track its remote counterpart, you may want to use
hint: "git push -u" to set the upstream config as you push.
hint: Disable this message with "git config advice.setUpstreamFailure false"This is just a very simple log that shows git doesn’t know the upstream has main branch.
Although this may seem simple, I still carefully checked to see if git remote was set up correctly. Then I discovered that git remote had disappeared.
There are only two possible reason to cause this.
- Something that can remove git remote config was executed.
- Git remote’s URL was an empty string so that git won’t set up correctly.
Let's first discuss the first situation.
git-filter-repo
When we remove some files in a git repository folder and commit the change, the file looks disappeared but actually it still existed in the .git folder. That means even you push the deletion commit to a remote, the file will still be stored on the remote’s repository. Once the file become very large, the remote repository will also become very large.
So in my backup system, I try to erase all previous backup file (LFS) every time I do next backup. This behavior ensures my repository can always remain the same size as the latest backup file’s.
How I did this ? The shell scripts looks like below:
# Remove the previous backup file from the git repository, using git-filter-repo.
# This should be done before any un-staged changes are made.
git filter-repo --path-glob '*.tgz' --invert-paths --force --prune-empty never
# Cleanup the git-lfs files. (in .git folder)
git for-each-ref --format="delete %(refname)" refs/original | git update-ref --stdin
git reflog expire --expire=now --all
git gc --prune=now
git lfs pruneThese commands use a tool called git-filter-repo, which replaces the original `git filter-branch` and provides more capabilities and performances.
Using these command can help us re-write all history commits and remove the specific object from the WHOLE git history. Hence the file won’t stay in the .git folder. In other words, there’s no way to recover the files you just deleted.
These operation happened in the `backup-pool` folder to ensure that old backup files will not continue to occupy space in the GitLab repository after the new version is pushed. (between step 2 and 3).
However, the git-filter-repo command will also remove the git origin config for some security concerns. This may be the reason why git can not find the remote branch if I forgot to add it back after the cleanup process finished.
Docker secret
Because I didn’t put any of my GitLab ssh keys in the VM which runs Minecraft, I need to use the GitLab https repository URL with Token to be the git remote URL. So I need to protect this Token to prevent leaking. My solution is using Docker secret, a more safe place to put things like token, keys, password, certifications, etc. However, it is a part of Docker Swarm’s feature. So we need to initialize a docker swarm node then add the secret.
When I go into my Minecraft server, I noticed that all docker swarm nodes were gone, so was the token stored in the docker secret.
Once the token not existed in the docker secret, the backup runtime will not understand where to push. This would be another reason.
Later I found that all of the above reasons indeed happened, and I also missed the step of git fetch. After fixing these things, I did grab the remote branch again. However, new problems then emerged.
Git LFS upload timeout
I discovered that my backup process would get stuck at the final push stage, due to the process timing out, being rejected by the remote. I checked the logs and found that both the number and size of the files being pushed were very large. This puzzled me because, as I mentioned earlier, I clear out the old backup files each time, completely removing them from the git history, so why would there be so many files to be pushed?
I checked the local git log, and it shows:
git log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%
│ s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)' --allDid you notice? This record shows what appears to be the state of two branches, with the red line representing my local main branch, but for some reason, there's a green branch on the right, named "replaced."
So this would cause git to push the entire green branch as well when pushing, leading to the total file size exceeding the load.
Actually, the "replaced" label in Git log isn't a branch name, but rather an indication that a commit has been replaced in the repository, typically using the git replace command.
The git replace command allows us to replace an object in Git with another object, without changing the references. It's often used to make temporary modifications to a commit for debugging or to rewrite history without actually modifying the refs.
Here’s some ways to check the refs:
- Show refs
git replace --list- Remove ref
git replace -d <object-hash>So finally, I use the following command to remove ALL refs:
git replace --list | xargs -r git replace -dvery easy, right ?
Conclusion
This experience has given me a deeper understanding of git, and also helped me to strengthen my world backup mechanism. I hope that if someone encounters similar problems in the future, they can use these methods to solve them.
If you want to refer to the full details of my Minecraft world and backup settings, you can join my Minecraft world at the URL mc.xcc.tw, or you can directly go to the repo of my world's configuration files.
