- Afsctool Hfs Compression For Mac Computers
- Afsctool Hfs Compression For Mac Os
- Afsctool Hfs Compression For Mac Pro
In Mac OS X 10.6, Apple introduced file compression (AppleFSCompression, internally) in HFS+. Compression is most often used for files installed as part of Mac OS X; user files are typically not compressed (but certainly can be!). Reading and writing compressed files is transparent as far as Apple's file system APIs. HFS+ Compression with Outlook 2016 for Mac Our tests of HFS+ compression saved 9GB of disk space from 20GB of Outlook data – a whopping 47% improvement. Here’s the results of some folder HFS+ compression. The three Outlook 2016 message folders get shrunk nicely by HFS+ compression. I came across the tool afsctool (brew install afsctool), which can, via command-line, apply HFS+ compression to a folder and all files in it. But I believe it does not activates compression of future files moved to the folder. Moving files manually and running a command line instruction repeatedly is inconvenient.
| Click here to return to the '10.6: Compress files with HFS+ compression' hint |
Maybe the compression criteria has to do with whether it will be faster or not. If you can only get minimal compression, you won't really save time on the disk read after decompressing the file. At least, that is the whole reason suggested for compression in the Ars Technica article.
Ugh, so it looks like the file size reported in the Finder is the uncompressed size. Aside from the whole KB vs. KiB change, this just creates confusion. When I see the file size in the Finder, I expect it to mean how much space it takes up on the disk (the compressed size).
When copying or reading a file, where it's uncompressed automatically to be presented as if it was the uncompressed data on the disk, sure I need to see the uncompressed size.
afsctool
(source is included)
Here's the usage information for it: The -c flag applies in place HFS+ compression to a file or recursively to a folder, with the option to specify the compression level (default is 9), change the default maximum size (in bytes) of files to process (default is 20971520, which is 20 MiB), and specify the minimum savings for a file to be HFS+ compressed. If the -k flag is given then compressed files will be checked after compression against the uncompressed files, and if any corruption has occurred then the file will be reverted back to its uncompressed form. The -l flag lists HFS+ compressed files in a folder or if the -c flag is given, it lists the files that fail to compress. The -d flag decompresses HFS+ compressed files (similar to afscexpand, but sometimes more reliable). The -a flag flattens a HFS+ compressed file to the data fork of the destination file, decompressing the original compressed file afterwards if the -d flag is given. The -x flag does the reverse of -a, unflattening a file with HFS+ compressed data in the data fork to the destination file, then decompressed the destination file is the -d flag is given. The -v flag increases verbosity.
Here's some example output: A warning about using this utility; do NOT try to compress the System folder. Any files that were not compressed in the System folder were left uncompressed for a reason! If you do try to compress your System folder further, then you will very likely end up having to reinstall Mac OS X.
http://n8gray.org/code/backup-bouncer
Do you approve? Do you have this tool hosted somewhere so I can check for updates?
Thanks,
-n8
http://files.me.com/brkirch/ijt4f7
Your tool is great; the source contains a lot of magic numbers buried directly. I guess it took a lot of times for you to reverse-engineer the format!
If you could write a blog post about the format of the HFS compressed file, it would be very helpful for the rest of us..
By the way, do you think there is a private system call which does the compression? Can it be found inside the source code of xnu?
ditto uses the private framework Bom to compress files. However, looking into it further I found that the Bom framework makes a call to another private framework: AppleFSCompression. Unfortunately the syntax for the functions in AppleFSCompression is far from obvious, especially since it is a private framework which means that there are no included headers for it (and it also means that it will probably remain closed source). It didn't in the end matter however, because I found that zlib is used for the actual compression and that library is well documented. So I decided to simply figure out how the HFS+ compressed files are constructed and added that to my program.Here's how the HFS+ compression is applied:
1. Check the file to ensure it does not have a resource fork or com.apple.decmpfs extended attribute.
2. Construct the headers, calculate the number of 64 KiB blocks needed based on the source file size.
3. Compress the 64 KiB blocks using zlib (Apple uses compression level 5, but other compression levels also work); if there is only one block then append it to the com.apple.decmpfs extended attribute if the compressed data is 3786 bytes or less in size, otherwise the compressed data is put into the resource fork of the file. If the resource fork is used to store the compressed data, then no block is allowed to be larger after compression (if a block is larger after compression then compression for the entire file will fail). After the compressed blocks are created then their locations and sizes are written to the resource fork data header.
4. Add the com.apple.decmpfs extended attribute to the file, then the resource fork if one is needed.
5. Truncate the data fork length to zero and use chflags to set the HFS+ compression flag.
This produces compressed files that are identical to the ones produced by
ditto, provided compression level 5 was used for the zlib functions. Download afsctool here: http://files.me.com/brkirch/ijt4f7
ditto uses). Just a quick q - everything I try to compress via afsctool seems to give me an 'Unable to compress file,' which isn't particularly informative…Â Any clue as to what could be causing everything to fail out? And it's not the obvious - I am on 10.6, HFS+… Ulysses manual.
The latest version of afsctool will only fail to compress a file if one of the following is true:
-the file has a resource fork
-the file has a zero length data fork
-the file is not a regular file (if it is for example, a symbolic link)
-the file will become larger after compression is applied
-the file is greater than 2 GiB in size
-there are insufficient privileges to change the file
-user specified conditions aren't met
I have made a website for my software:
http://web.me.com/brkirch
Future updates to afsctool will be posted there.
> (in fact, contrary to what some posts on this site have suggested, command line utilities like strings will see the same file regardless if it is compressed or not)
Not entirely true. While 'cat' will uncompress the file first, 'more' and 'less' will show a raw binary dump of the compressed file. That's a quick and dirty way to tell if a file is compressed, at least until Apple fixes it. Your utility is far more useful obviously.
Afsctool Hfs Compression For Mac Computers
stat -f %f [file] returns a number that has the 32 flag set (the UF_COMPRESSED flag) then the file is HFS+ compressed. I can't say I can see how more or less can be used to identify compressed files though, as the data being printed to the screen is the uncompressed data, not the compressed (just tried it, you may have somehow gotten a different result but I tried it on some compressed files and it is definitely showing the uncompressed data). Less is a surprisingly complex utility but it still uses a standard POSIX function to open a regular file in this source file, search for open(qopen_filename, OPEN_READ) within the file to find it.
man 3 fdopen 2 open will confirm that both of these system calls are standard POSIX calls, so they don't really know anything about HFS vs. any other file system.
(Also, 'more' is simply the compatibility mode of 'less.' They're the same file, hard linked.)
Nope, I just compressed an rtf file, then used 'more' and 'less', the file appeared normally, as if nothing was special about it, no raw compressed data but the original, readable text.
Wow, I just saved several gigabytes by compressing a few big apps (just using 'sudo ditto --hfsCompression'), I can't tell the difference except for the free space I gained!
For example Photoshop still takes the same amount of time to launch after a cold reboot (even slightly quicker, if anything), does not complain about damaged files, I was afraid it might give me some DRM-like nastiness with moved/changed files, but no, everything just works like before, only taking a few hundreds of MB less!
Next up is the Application support folder :-)
Only one strange thing: when compressing some application folders, I got messages like
ditto: /Applications/Utilities/Gecomprimeerd/ResDataBridge 1.0/.BC.n6FLVl: Permission denied
ditto: /Applications/Games/StellaOSX2.2/Roms/.BC.YRr5fp: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.j2H1FL: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.r0fvwu: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.kCj925: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.vm8yx2: Permission denied
I just listed a few, but I got similar messages for the Photoshop folder and several others.
When I go look for these mysterious files (.BC.xxxxxx), they simply do not seem to exist. 'ls -a' on the original folder does not show them. The compressed applications work just fine afterwards.
What are these 'files'? Is it just a bug in ditto? Is it some internal file system stuff that is not supposed to show up in the higher abstraction levels? It this were Windows I would immediately suspect a virus, but on a Mac that's obviously impossible..
Now all we need is an Automator script or Applescript that will compress all files moved into a folder. Microsoft edge el capitan.
I'm pretty certain you always need to have sudo to run the ditto --hfsCompress command.
That attitude probably creates more security holes than all technical errors put together.
Security is not an absolute matter, nor does it have the same requirements everywhere. Insisting on following rigid little rules like this creates annoyances and inconviences that people then respond to by circumventing the security mechanism entierly, e.g., telling anyone who sits down at the machine their password so they can run secure processes, chowing all the files to their own user, etc.
For instance on my laptop I add myself to the wheel group and let the wheel group use sudo without a password. Would I do this on a corporate machine or webserver? No! But does that make it a security risk on my laptop? No. The point of security on my laptop is to protect my private documents and files. If they already have access to my user account the game is over.
Awesome tip, thanks!
Now, all we need is some enterprising developer to make a nice user friendly app to compress and get info on compressed files.
I am a bit nervous using the terminal to modify/convert files in the event I cause a problem with a typo.. ;)
I just discovered one - and it's free as a special via MacHeist.com right now.
It's called Squeeze.
Afsctool Hfs Compression For Mac Os
MacHeist.com has just begun to give away an app for free that basically provides a nice GUI for this tip - I think the give-away is only for one week.
Go to MacHeist.com to download Squeeze, the author's website is http://www.latenitesoft.com/squeeze/
Has anyone found a way to make Finder accurately report the compressed file sizes?
How do I know how much space I actually have on my drive when Squeeze says I have saved 22.9 GB and Finder says I have only 2.9 GB free (the same amount I had before Squeeze squeezed)?
Path Finder also shows only 2.9 GB free.
This HFS+ compression thread is the first clue to a problem I have. After a boot failure the local apple store did an archive and install. The OS was set up for multiple users with one admin. The admin account data is fine but all the other users data is invisible with a .BC file prefix on all the user data files. The current OS is 10.5.8. Could they have used 10.6 initially and compressed the users files? All system files are OK. Is there a way to restore the 190 GB of now invisible and unreadable files? In terminal mode they all show up with a ls -l command but the filenames are coded. The only solution the apple store offers is a wipe and install.
/Developer folder. To get around this limitation and save some space in /Developer paste the following shell script into a file called dev-compress.sh .. .. then prep and execute.. After a few months I have seen no adverse effects on using XCode 3 or XCode 4 with these folders compressed. Related to HFS+ Compression is the behavior of Retrospect 6.1 backup.
Apparently running Retrospect on a Snow Leopard system causes all HFS+ Compressed files to become uncompressed (and manifests itself as a multitude of execution errors described in the the log as changes of the modification date/time to the current date/time).
See 'https://discussions.apple.com/thread/2674794' and 'http://www.techcare.com/blogs/jk-buzz/102-jk-buzz-172010.html'.
I've found a nice utility that performs this as a background process:
http://latenitesoft.com/clusters/
So far it has worked well for me, including compressing all Adobe products, saving approximately 50% space, which makes quite a difference now that I've moved to a considerably smaller SSD drive.
There is a new GUI tool for HFS+ compression in appstore called MoreSpace Folder Compression.
http://itunes.apple.com/app/morespace-folder-compression/id521635253?mt=12
Simply but usable.
Disk Utility User Guide
Disk Utility on Mac supports several file system formats:
Apple File System (APFS): The file system used by macOS 10.13 or later.
Mac OS Extended: The file system used by macOS 10.12 or earlier.
MS-DOS (FAT) and ExFAT: File systems that are compatible with Windows.
Afsctool Hfs Compression For Mac Pro
Apple File System (APFS)
Apple File System (APFS), the default file system for Mac computers using macOS 10.13 or later, features strong encryption, space sharing, snapshots, fast directory sizing, and improved file system fundamentals. While APFS is optimized for the Flash/SSD storage used in recent Mac computers, it can also be used with older systems with traditional hard disk drives (HDD) and external, direct-attached storage. macOS 10.13 or later supports APFS for both bootable and data volumes.
APFS allocates disk space within a container (partition) on demand. When a single APFS container has multiple volumes, the container’s free space is shared and is automatically allocated to any of the individual volumes as needed. If desired, you can specify reserve and quota sizes for each volume. Each volume uses only part of the overall container, so the available space is the total size of the container, minus the size of all the volumes in the container.
Choose one of the following APFS formats for Mac computers using macOS 10.13 or later.
APFS: Uses the APFS format. Choose this option if you don’t need an encrypted or case-sensitive format.
APFS (Encrypted): Uses the APFS format and encrypts the volume.
APFS (Case-sensitive): Uses the APFS format and is case-sensitive to file and folder names. For example, folders named “Homework” and “HOMEWORK” are two different folders.
APFS (Case-sensitive, Encrypted): Uses the APFS format, is case-sensitive to file and folder names, and encrypts the volume. For example, folders named “Homework” and “HOMEWORK” are two different folders.
You can easily add or delete volumes in APFS containers. Each volume within an APFS container can have its own APFS format—APFS, APFS (Encrypted), APFS (Case-sensitive), or APFS (Case-sensitive, Encrypted).
Mac OS Extended
Choose one of the following Mac OS Extended file system formats for compatibility with Mac computers using macOS 10.12 or earlier.
Mac OS Extended (Journaled): Uses the Mac format (Journaled HFS Plus) to protect the integrity of the hierarchical file system. Choose this option if you don’t need an encrypted or case-sensitive format.
Mac OS Extended (Journaled, Encrypted): Uses the Mac format, requires a password, and encrypts the partition.
Mac OS Extended (Case-sensitive, Journaled): Uses the Mac format and is case-sensitive to folder names. For example, folders named “Homework” and “HOMEWORK” are two different folders.
Mac OS Extended (Case-sensitive, Journaled, Encrypted): Uses the Mac format, is case-sensitive to folder names, requires a password, and encrypts the partition.
Windows-compatible formats
Choose one of the following Windows-compatible file system formats if you are formatting a disk to use with Windows.
MS-DOS (FAT): Use for Windows volumes that are 32 GB or less.
ExFAT: Use for Windows volumes that are over 32 GB.