Saint's Log

Validating Cross-platform File Transfers

I often need to copy files from my MacBook to my Windows desktop. For some files, I especially want to ensure that they were copied correctly (no data loss or any changes to the files). Therefore, I wrote a Python script that takes a directory and enumerates all the files under that directory, computes their SHA 256 hashes, and writes these hashes to a file. As a final step, I compute the SHA 256 hash of the file containing the list of file/digest pairs. Under this setup, I just need to run the script on each machine after copying a directory and verify that it generates the same final SHA 256 hash. This guarantees that the probability of an error in the file transfer is extremely tiny. One of the issues I ran into was this Python error:

$ python /c/repos/scratchpad/scripts/python/write_sha256.py . --concat-hashes hashes.txt
[2026-07-31 12:51:12] Processing: .\._somefile.dat
Traceback (most recent call last):
  File "C:\repos\scratchpad\scripts\python\write_sha256.py", line 79, in <module>
    process_concat(args.directory, args.concat_hashes, force=args.force)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\scratchpad\scripts\python\write_sha256.py", line 54, in process_concat
    concat_file.write(f"{sha256sum}  {relpath}\n")
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python314\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u202f' in position 107: character maps to <undefined>

This had to be the Windows code page keeping things exciting! I explicitly set the encoding to UTF-8 to prevent such issues. For the final hashes to be identical, the list of files in the source directory needs to match the list of files in the destination directory. However, this wasn’t the case when copying files from my MacBook Pro to my Windows machine. I had never stopped to figure out what all these files with a leading period and underscore were whenever I did such a copy: they are AppleDouble files, which are used to store extended attributes/metadata (see description at AppleSingle and AppleDouble formats – Wikipedia). I don’t care about such files when validating my file copies so I made a change to ignore them by default.

The last issue was the line endings of the file. On windows, how would I detect the line ending of a file? – Stack Overflow wasn’t much help but since I’m using the Git Bash environment, the file command suggested in linux – How to find out line-endings in a text file? – Stack Overflow was perfect. For the hashes to be identical, the files on both platforms should have the same line endings. I updated the script to write the text files with LF line terminators.

$ file hashes-macos.txt
hashes-macos.txt: Unicode text, UTF-8 text

$ file hashes.txt
hashes.txt: Unicode text, UTF-8 text, with CRLF line terminators

I already had a change in place to iterate over the files in alphabetical order (which is a stable ordering). With this implementation, I no longer need to tar/compress files or verify hashes for individual files.