iCloud Drive: Large Folder Problem and Solution

I’ve had a love/hate relationship with cloud-based file synchronization services over the years. None seem to offer the golden set of reliability, security, privacy and tight OS integration.

Needless to say, I was excited when Apple finally released iCloud drive with OS X Yosemite (10.10) and iOS 8.1. According to Apple’s documentation, data stored in iCloud drive is encrypted in transit and at rest. Apple has not explicitly said if they scan your files while on their servers, but Apple’s statements to date make me think that they do not. I would call Apple’s security good, their privacy “probably good” until proven otherwise and their OS integration excellent. While some folks have had issues with iCloud reliability, it’s worked well for me over the past year, so I’m willing to give it a try as shared file storage space.

So, with that background I decided to move off of Bit Torrent Sync (which is very good, but not well integrated into OS X or iOS) and onto iCloud Drive right after I installed Yosemite on my iMac and MacBook Pro. Yes, I know, I was asking for early adopter problems and unfortunately I was not disappointed.

My initial attempt was a simple move of files into iCloud Drive using the Finder. This worked fine for a few small (< 2GB) folders. However, when I moved over a large (42 GB) folder with some animation asset collections (> 10,000 files per folder). I ran into problems.

For reference, the iCloud drive folder is hidden. You can find it in your “Library” folder at “~/Library/Mobile\ Documents/com~apple~CloudDocs”. This directory is what is mapped to the “iCloud Drive” folder in the Finder.

After much digging around, here’s what appears to happen:

  1. Upon moving files into the iCloud Folder, meta-data for each file is recorded locally for purposes of tracking file changes.
  2. The local meta-data is bundled up, apparently on a per-folder basis and transmitted up to Apple’s servers in batches as files are written to the iCloud drive folder.
  3. When a large enough group of files changes all at once (or nearly so) as happens with a large finder copy, the bundled up meta-data groupings get very large.
  4. When more than 100 files are updated all at once, the resulting meta-data bundle exceeds the size of a single database commit on Apple’s servers and the sync process registers an error and stops working.

How did I find this out? There’s a ASL-based log of all of the iCloud Drive sync processing that can be found in the directory /var/log/com.apple.clouddocs.asl. You can view it either using the Console application or using the new “brctl” command, like so:

brctl log -w

The “log” verb specifies log file viewing and “-w” invokes a “tail -F” style of continuous playback of the current log file. The Console app can do the same thing by picking the directory listed above and the most recent log file in the directory.

The tell-tail error message is as follows:

documentContent/731CDF0F-DD1C-4D7B-A627-5EED6E245B9A:(com.apple.CloudDocs:__defaultOwner__) = <CKError 0x7f8d08d919a0: "Limit Exceeded" (27/2023); server message = "Database commit size exceeds limit">

It’s the last part of that error, the “server message” that’s important. The “documentContent” code is unique to my files. You will have a different one.

To work around this problem, I did the following:

  1. I tar’d up the animation asset folders I had lying about in my source folder. I did that and the total number of files in the overall folder dropped from over 100,000 to just over 12,000.
  2. I reset the iCloud Drive meta-data store, by killing off the “bird” daemon (killall bird), removing the meta-data store which is found in ~/Library/Application\ Support/CloudDocs/ (cd ~/Library/Application\ Support ; rm -rf CloudDocs), and immediately restarting the Mac.
  3. I then waited until the iCloud Drive sync process rebuilt the meta-data and finished syncing successfully as evidenced by activity stopping in the log file (see above) without any new errors appearing. This can take several hours.
  4. Finally, I copied over the remainder of the folder slowly, leveraging a simple infinite loop in bash and the bandwidth usage control in rsync, like this:
while true; do rsync -av --delete --max-delete=10 --bwlimit=4 --exclude-from=/tmp/exclude source-dir/ iCloud; sleep 5; done

The combination of the infinite shell loop, the delete limit and bandwidth limit on rsync causes the copy to move slowly enough to avoid a mass file change, even when deleting files. This takes several hours to finish, but works without error.

The contents of /tmp/exclude were picked to avoid duplicating Apple specific file meta-data and the meta-data directories used by Bit Torrent Sync which I had been using previously. The contents of /tmp/exclude are:

.DS*
.Sync*

If I were starting to move to iCloud from scratch, I would use the shell loop and rsync shown above right from the start to avoid problems.

I hope this helps some OS X Yosemite users escape this problem and avoid several days of detective work to find and work around this issue.

NOTE: As always, when moving important files to new drives, make sure you have multiple backup copies! It’s really easy to accidentally delete things and it’s absolutely required to have a backup to compare to once you believe things have worked! I am able to say this worked because I compared the results on two Macs to a known good backup and every file matched, byte for byte.