Joey Hess [Mon, 10 Feb 2025 19:24:28 +0000 (15:24 -0400)]
OsPath conversion
While some RawFilePath and FilePath remain, this converts most of
git-annex to using OsPath.
(When built without the OsPath build flag, is falls back to using
type OsPath = RawFilePath.)
The goals are
1) improved performance by using OsPath end-to-end when possible
2) potentially avoiding memory use problems caused by pinned strict
ByteString, since OsPath uses ShortByteString
3) eventually eliminating the filepath-bytestring dependency so I don't
need to keep maintaining that library
(this doesn't get all the way, but close)
4) generally improved type safety, since OsPath is a newtype, while
FilePath and RawFilePath are just type aliaes.
This is the result of a type checker driven process. I started by
converting from System.Directory to System.Directory.OsPath, and from
System.FilePath to System.OsPath. Then I fixed all the compile errors,
which took 3 weeks of work.
Unfortunately, there are several test suite failures at this point.
Also, it only has been built on linux, on windows and OSX there are
probably ifdefs whose code still needs to be converted.
Note that there is a parallel line of commits, starting with 05bdce328d890cbac68a8627aaae262078a8290a
which is the incremental progress as I worked on this. It will be merged
with this commit. In some cases, commits in that line explain in more
details the reasons for some specific changes.
Joey Hess [Wed, 22 Jan 2025 17:22:51 +0000 (13:22 -0400)]
simplify replaceFile using relatedTemplate
Now that truncateFilePath and relatedTemplate have both been optimised,
may as well use them in replaceFile, rather than the custom hack it
used.
Removed the windows-specific ifdef as well, because on Windows long
filepaths no longer really a problem, since ghc and git-annex use UNC
converted paths.
replaceFile no longer checks fileNameLengthLimit. That took a syscall,
and since we have an existing file, we know filenames of its length are
supported by the filesystem. Assuming that the withOtherTmp directory is
on the same filesystem as the file replaceFile is being called on, which
I believe it is.
Joey Hess [Wed, 22 Jan 2025 16:34:54 +0000 (12:34 -0400)]
optimize truncateFilePath
Often the filepath will be all ascii, or mostly so, and this
optimisation makes a file that has an ascii suffix of sufficient length
be roundtrip converted between String and ByteString only once, rather
than once per character.
Joey Hess [Tue, 21 Jan 2025 21:00:37 +0000 (17:00 -0400)]
use openTempFile from file-io
And follow-on changes.
Note that relatedTemplate was changed to operate on a RawFilePath, and
so when it counts the length, it is now the number of bytes, not the
number of code points. This will just make it truncate shorter strings
in some cases, the truncation is still unicode aware.
When not building with the OsPath flag, toOsPath . fromRawFilePath and
fromRawFilePath . fromOsPath do extra conversions back and forth between
String and ByteString. That overhead could be avoided, but that's the
non-optimised build mode, so didn't bother.
Joey Hess [Mon, 20 Jan 2025 22:03:26 +0000 (18:03 -0400)]
RawFilePath conversion of System.Directory
By using System.Directory.OsPath, which takes and returns OsString,
which is a ShortByteString. So, things like dirContents currently have the
overhead of copying that to a ByteString, but that should be less than
the overhead of using Strings which often in turn were converted to
RawFilePaths.
Added Utility.OsString and the OsString build flag. That flag is turned
on in the stack.yaml, and will be turned on automatically by cabal when
built with new enough libraries. The stack.yaml change is a bit ugly,
and that could be reverted for now if it causes any problems.
Note that Utility.OsString.toOsString on windows is avoiding only a
check of encoding that is documented as being unlikely to fail. I don't
think it can fail in git-annex; if it could, git-annex didn't contain
such an encoding check before, so at worst that should be a wash.
Joey Hess [Mon, 13 Jan 2025 18:22:49 +0000 (14:22 -0400)]
pre-init config and hook
Added annex.pre-init-command git config and pre-init-annex hook that is run
before git-annex repository initialization.
This can block initialization. Or it can preform pre-initialization
configuration or tweaking.
I left stdio connected while it's running, so it could also be used for
interactive prompting conceivably, although that would want to use /dev/tty
anyway probably in order to not pollute the stdout of a command when
automatic initialization is done.
Joey Hess [Fri, 10 Jan 2025 18:50:49 +0000 (14:50 -0400)]
added hooks corresponding to annex.*-command
* Added freezecontent-annex and thawcontent-annex hooks that
correspond to the git configs annex.freezecontent and
annex.thawcontent.
* Added secure-erase-annex hook that corresponds to the git config
annex.secure-erase-command.
* Added commitmessage-annex hook that corresponds to the git config
annex.commitmessage-command.
* Added http-headers-annex hook that corresponds to the git config
annex.http-headers-command.
that correspond to the post-update-annex and pre-commit-annex hooks.
The use case for these is eg, setting up a git repository that is run in a
container, where the easiest way to provide a script is by putting it in
.git/hooks/, rather than copying it into the container in a way that puts
it in PATH.
This is all the ones that make sense to add for annex.*-config git configs.
annex.youtube-dl-command is not a hook, it's telling git-annex what command
to run. So is annex.shared-sop-command. So omitted those.
May later also want to add hooks corresponding to
`remote.<name>.annex-cost-command` etc.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
Joey Hess [Fri, 10 Jan 2025 17:27:51 +0000 (13:27 -0400)]
configs annex.post-update-command and annex.pre-commit-command
Added git configs annex.post-update-command and annex.pre-commit-command
that correspond to the git-annex hook scripts post-update-annex and
pre-commit-annex.
Note that the hook files take precience over the git config, since the git
config can includ global config which should be overridden by local config.
These new git configs are probably not super useful. Especially the
pre-commit-annex hook is there to install scripts to instead of the
pre-commit hook, since git-annex installs that hook itself. So why would
someone want to use a git config for that? Only reason I can think of would
be in a global git config. Or possibly because it's easier to set a git
config than write a hook script, on an OS like Windows.
The real reason I'm adding these is as groundwork for making other
annex.*-command git configs also be available as hook scripts. I want
to avoid having some things available as only git hooks and others as
both gitconfigs and git hooks. (It seems that some annex.*-command configs
don't translate to git hooks though.)
In the man page, moved documentation of the hooks to be next to the
documentation of the git configs. This is to avoid repitition.
Joey Hess [Tue, 7 Jan 2025 20:37:39 +0000 (16:37 -0400)]
windows permissions fix
Windows: Fix permission denied error when dropping files that have the
readonly attribute set.
Files coming from a special remote may have had write permission removed
from them. The directory special remote does that. And there are
probably others. So rather than fixing it on the special remote side,
made moveAnnex, on Windows, add back the write bit. This apparently
removes the readonly attribute. See Remote.Directory.removeDirGeneric
which already did the same on windows to allow removing files from the
directory special remote.
The reason that cleanObjectLoc also calls allowWrite is to handle
situations where files have already gotten into git-annex repositories on
Windows with the write bit set. Eg, an older git-annex put them there.
Or perhaps the git-annex repository was populated on some other OS.
projects / git-annex.git / log