Category Archives: OS X Server

iCloud Drive: Large Folder Problem and Solution

I’ve had a love/hate relationship with cloud-based file synchronization services over the years. None seem to offer the golden set of reliability, security, privacy and tight OS integration.

Needless to say, I was excited when Apple finally released iCloud drive with OS X Yosemite (10.10) and iOS 8.1. According to Apple’s documentation, data stored in iCloud drive is encrypted in transit and at rest. Apple has not explicitly said if they scan your files while on their servers, but Apple’s statements to date make me think that they do not. I would call Apple’s security good, their privacy “probably good” until proven otherwise and their OS integration excellent. While some folks have had issues with iCloud reliability, it’s worked well for me over the past year, so I’m willing to give it a try as shared file storage space.

So, with that background I decided to move off of Bit Torrent Sync (which is very good, but not well integrated into OS X or iOS) and onto iCloud Drive right after I installed Yosemite on my iMac and MacBook Pro. Yes, I know, I was asking for early adopter problems and unfortunately I was not disappointed.

My initial attempt was a simple move of files into iCloud Drive using the Finder. This worked fine for a few small (< 2GB) folders. However, when I moved over a large (42 GB) folder with some animation asset collections (> 10,000 files per folder). I ran into problems.

For reference, the iCloud drive folder is hidden. You can find it in your “Library” folder at “~/Library/Mobile\ Documents/com~apple~CloudDocs”. This directory is what is mapped to the “iCloud Drive” folder in the Finder.

After much digging around, here’s what appears to happen:

  1. Upon moving files into the iCloud Folder, meta-data for each file is recorded locally for purposes of tracking file changes.
  2. The local meta-data is bundled up, apparently on a per-folder basis and transmitted up to Apple’s servers in batches as files are written to the iCloud drive folder.
  3. When a large enough group of files changes all at once (or nearly so) as happens with a large finder copy, the bundled up meta-data groupings get very large.
  4. When more than 100 files are updated all at once, the resulting meta-data bundle exceeds the size of a single database commit on Apple’s servers and the sync process registers an error and stops working.

How did I find this out? There’s a ASL-based log of all of the iCloud Drive sync processing that can be found in the directory /var/log/com.apple.clouddocs.asl. You can view it either using the Console application or using the new “brctl” command, like so:

brctl log -w

The “log” verb specifies log file viewing and “-w” invokes a “tail -F” style of continuous playback of the current log file. The Console app can do the same thing by picking the directory listed above and the most recent log file in the directory.

The tell-tail error message is as follows:

documentContent/731CDF0F-DD1C-4D7B-A627-5EED6E245B9A:(com.apple.CloudDocs:__defaultOwner__) = <CKError 0x7f8d08d919a0: "Limit Exceeded" (27/2023); server message = "Database commit size exceeds limit">

It’s the last part of that error, the “server message” that’s important. The “documentContent” code is unique to my files. You will have a different one.

To work around this problem, I did the following:

  1. I tar’d up the animation asset folders I had lying about in my source folder. I did that and the total number of files in the overall folder dropped from over 100,000 to just over 12,000.
  2. I reset the iCloud Drive meta-data store, by killing off the “bird” daemon (killall bird), removing the meta-data store which is found in ~/Library/Application\ Support/CloudDocs/ (cd ~/Library/Application\ Support ; rm -rf CloudDocs), and immediately restarting the Mac.
  3. I then waited until the iCloud Drive sync process rebuilt the meta-data and finished syncing successfully as evidenced by activity stopping in the log file (see above) without any new errors appearing. This can take several hours.
  4. Finally, I copied over the remainder of the folder slowly, leveraging a simple infinite loop in bash and the bandwidth usage control in rsync, like this:
while true; do rsync -av --delete --max-delete=10 --bwlimit=4 --exclude-from=/tmp/exclude source-dir/ iCloud; sleep 5; done

The combination of the infinite shell loop, the delete limit and bandwidth limit on rsync causes the copy to move slowly enough to avoid a mass file change, even when deleting files. This takes several hours to finish, but works without error.

The contents of /tmp/exclude were picked to avoid duplicating Apple specific file meta-data and the meta-data directories used by Bit Torrent Sync which I had been using previously. The contents of /tmp/exclude are:

.DS*
.Sync*

If I were starting to move to iCloud from scratch, I would use the shell loop and rsync shown above right from the start to avoid problems.

I hope this helps some OS X Yosemite users escape this problem and avoid several days of detective work to find and work around this issue.

NOTE: As always, when moving important files to new drives, make sure you have multiple backup copies! It’s really easy to accidentally delete things and it’s absolutely required to have a backup to compare to once you believe things have worked! I am able to say this worked because I compared the results on two Macs to a known good backup and every file matched, byte for byte.

Replacing a harddisk in a Mac Mini Server

Behind the scenes, my household network services are provided by a pair of Apple Mac Mini Servers, one for internal services (DNS, DHCP, file sharing, backups) and one for external services (DNS, mail, web hosting, ownCloud). I also have an older Mac Mini attached to my home theatre. I like the Mac Mini’s a lot. They are inexpensive as servers go, draw little power, and are quiet, cheap and reliable.

Apple Mac Mini Server

Apple Mac Mini Server

Sadly, the internal server’s boot disk died a couple of weeks ago. It failed slowly over a couple of hours and thankfully did not corrupt the server’s time machine based backups on its way out. I have prepared for this eventuality by keeping a cold spare  in the form of an older Mac Mini that sits on the shelf, ready to be put into service if one of the active Mac Minis has a problem. So, after an attempt to resuscitate the dying hard disk, I gave up on it and restored the most recent backup of the server onto the cold spare.

The restore process is pretty simple. First, I choose to boot the cold spare from a USB key that I had prepared with a bootable installation image of OS X Mavericks. I used the included script that’s buried in the Maverick install image to create the bootable USB key. The directions for using it can be found from Apple or with a bit more detail from MacWorld. You will need to have either saved, or downloaded the current Mavericks installer before you follow those instructions.

Once the USB key and cold spare are ready to boot, I booted with the “option” key held down and picked the prepared USB key from the list of available boot devices. When restoring a server with more than one disk, you need to do a little bit of extra work to restore the second disk before you start the restoration of the boot disk as detailed in this Apple Knowledge Base article. After restoration, the cold spare booted up without problems. I corrected it’s network settings (it has a USB ethernet connection, long story for another post…), checked that services were running, “inherited” the current time machine backups and my internal services were back in operation.

Next up, I ordered up an SSD to replace the failed disk. SSDs have come a long way in terms of reliability and are well worth the added expense given the marked speed improvement they offer. I chose a Samsung 750GB EVO 840 drive based on the recommendation of the good folks at Mac Mini Colo. I bought the drive from OWC and followed their installation video. The install is indeed “challenging” and took me about 90 minutes.

Mac Mini Disassembled

My Mac Mini disassembled

A word of warning, I ended up doing the disk installation twice due to pulling the wrong drive the first time. If you can, before you remove the dead disk, look at the system with disk utility which will indicate which drive bay (upper or lower) a disk is located in. That would have saved me some time.

The server with the SSD has been running for a few days now without problem and I plan to swap it back into production service, using the same time machine restoration process, this weekend.

Mavericks syslog server configuration

For you OS X server admins out there, the syntax to enable syslogd to act as a syslog server has changed under Mavericks and OS X Server v3.

As before, convert the plist from binary into xml using plutil:

sudo plutil -convert xml1 /System/Library/LaunchDaemons/com.apple.syslogd.plist

The open the plist in your favorite editor and look for <key>Sockets</key>. The \<dict\> block after this key holds the information that used be in the dict block following <key\>NetworkListener</key>, except now the information goes in a sub-block called BSDSystemLogger. The default BSDSystemLogger block looks like this:

<key>BSDSystemLogger</key>
<dict>
        <key>SockPathMode</key>
        <integer>438</integer>
        <key>SockPathName</key>
        <string>/var/run/syslog</string>
        <key>SockType</key>
        <string>dgram</string>
</dict>

This creates a UNIX socket in /var/run. To enable remote logging, we want a network socket on the syslog port. To get that, simply change the above block to this:

<key>BSDSystemLogger</key>
<dict>
        <key>SockType</key>
        <string>dgram</string>
        <key>SockServiceName</key>
        <string>syslog</string>
</dict>

Save the plist. You can re-convert it to binary if you like or just leave it in XML. Unload and reload the plist and syslog will start and accept remote syslog data.