General
Git never deletes anything: what a commit actually stores
The Machine Underneath DEV Community
2 views
You commit a file with an API key in it. You notice within a minute, remove the file, commit the removal, and push. The file is gone from your working tree — and the key is still readable with two commands. git gc does not remove it either.
That is not a bug, and not a mistake on your part. It follows from what a commit records: not your change, but a complete snapshot, in which every object is named after its own contents.
The two commands
git log --all --diff-filter=D --name-only # every file ever deleted
git cat-file -p <commit>:<path> # read any of them back
git gc does not help here, and that is consistent rather than surprising: the commit that introduced the file is still reachable, so the blob it points to is still reachable, and gc only removes what nothing points to.
A commit is a snapshot, not a diff
Three files, one commit. Change one byte in one of them, commit again. Count the objects: eight — two commits, two trees, and four blobs. Not "one diff".
The tree of the second commit lists a new hash for the file you touched and the same hashes as before for the two you did not. That is what "complete snapshot with the unchanged parts reused" means, and it is why git show has to compute the diff at display time: the repository never stored one.
A hundred identical files are one object
Write the same line into a hundred files and commit. Distinct blob objects: one.
Deduplication is not an optimisation Git performs. The object's name is a hash of its content, so identical content cannot be two objects — there is nowhere for the second one to live.
Then the repository would be enormous
This is the objection I expected to be the hard one, and it has a measured answer.
Fifty one commits on a two-hundred-thousand-line file, each commit appending a line:
loose objects, before packing
22032 KiB
after git gc
847.96 KiB
ratio
26x smaller
how the 51 versions are stored
2 in full, 49 as deltas of about 94 bytes each
Nothing was thrown away and not one object was rewritten. Packing keeps a base and a chain of deltas — which is the whole point: "not deleted" and "not stored twice" are different statements.
Running the same script on a different machine (git 2.43.0) gave 24276 KiB loose, 468 KiB packed, one object in full and fifty as deltas. The ratio depends on your git version and on how the file changes; the shape does not.
Two things I had wrong before measuring
Both are the kind of claim that gets repeated:
The reflog window is thirty days, not ninety, for anything your branches can no longer reach — which is always the case after a reset, an amend or a rebase. Ninety days applies to entries still reachable from a branch.
A fresh clone does contain the deleted object. It simply does not check the file out. "Not in the working tree" is not "not in the repository".
The mechanism, animated
Fourteen minutes, no face, no music, every number measured:
The script behind every number above
Run it and post what you get. If a number comes out different, include your git version — I would rather fix the article than defend it.
#!/usr/bin/env bash
# What a commit actually stores — every number below comes out of git, not out of memory.
#
# bash git-objects.sh
#
# Three questions, in order:
# 1. is a commit a diff or a snapshot?
# 2. what happens to a hundred identical files?
# 3. if nothing is ever deleted, why is the repository not enormous?
set -e
D=$(mktemp -d); cd "$D"
git init -q; git config user.email a@b; git config user.name t
git config commit.gpgsign false
echo "== 1. A commit points at a tree, not at a diff =========================="
printf 'alpha\n' > a.txt; printf 'beta\n' > b.txt; printf 'gamma\n' > c.txt
git add -A; git commit -qm one
git cat-file -p HEAD | head -3
echo
echo "-- the tree is a list of names and content hashes"
git cat-file -p 'HEAD^{tree}'
echo
echo "-- change one byte in a.txt and commit again"
printf 'alphaX\n' > a.txt; git add -A; git commit -qm two
git cat-file -p 'HEAD^{tree}'
echo " a.txt has a new hash; b.txt and c.txt keep theirs, so the second commit"
echo " is a complete snapshot in which the unchanged parts are reused"
echo
echo "-- object count: 2 commits + 2 trees + 4 blobs = 8"
git count-objects -v | head -2
echo
echo "== 2. A hundred identical files ========================================="
D2=$(mktemp -d); cd "$D2"
git init -q; git config user.email a@b; git config user.name t
git config commit.gpgsign false
for i in $(seq 1 100); do printf 'same content\n' > "f$i.txt"; done
git add -A; git commit -qm many
echo "-- distinct blob objects for 100 identical files:"
git cat-file --batch-all-objects --batch-check | awk '$2=="blob"' | wc -l
echo " one, not a hundred: the object name is a function of the content,"
echo " so identical content is one object by construction"
echo
echo "== 3. Fifty one commits on a large file, loose vs packed ================"
D3=$(mktemp -d); cd "$D3"
git init -q; git config user.email a@b; git config user.name t
git config commit.gpgsign false
seq 1 200000 | sed 's/$/ line/' > big.txt
git add -A; git commit -qm base
for i in $(seq 1 50); do
printf '%s changed\n' "$i" >> big.txt
git add -A; git commit -qm "edit $i"
done
echo "-- loose objects, before packing:"
git count-objects -v | grep -E '^(count|size):'
git gc -q --aggressive 2>/dev/null || git gc -q
echo "-- after packing:"
git count-objects -v | grep -E '^(in-pack|size-pack):'
echo
echo "-- how the pack stores those 51 versions of big.txt:"
# verify-pack columns: sha1 type size size-in-pack offset [depth base-sha1]
# a delta row therefore has 7 fields, a fully stored object has 5
git verify-pack -v .git/objects/pack/*.idx 2>/dev/null \
| awk '$2=="blob"{print (NF>=7 ? "delta" : "stored in full")}' | sort | uniq -c || true
echo
echo " nothing was thrown away and no object was rewritten — packing stores"
echo " one base and a chain of deltas, which is why 'not deleted' and"
echo " 'not stored twice' are different statements"
One line for your own repository, if you read no further:
git log --all --diff-filter=D --name-only
Every file that ever left your project is on that list, and every one of them is still readable.
Read original: https://dev.to/machineunderneath/git-never-deletes-anything-what-a-commit-actually-stores-3nil
← Previous
How to Setup Cloudflare Workers for Serverless Compute
Next →
ChatGPT for Financial Services Targets Research, Modeling and Client Materials
Related
Vercel Sandbox now provides 64 GB of storage
General
5
Vercel Blog
Why is nobody talking about Conductor?
General
5
Reddit r/programming
Finally found a free Service Status site I like
General
2
Reddit r/programming
1.1.1.1 now supports post-quantum DNSSEC, all 2,420 bytes of it
General
4
Cloudflare Blog
Comments0
No comments yet — be the first