Category Archives: iOS

iCloud Drive: Large Folder Problem and Solution

I’ve had a love/hate relationship with cloud-based file synchronization services over the years. None seem to offer the golden set of reliability, security, privacy and tight OS integration.

Needless to say, I was excited when Apple finally released iCloud drive with OS X Yosemite (10.10) and iOS 8.1. According to Apple’s documentation, data stored in iCloud drive is encrypted in transit and at rest. Apple has not explicitly said if they scan your files while on their servers, but Apple’s statements to date make me think that they do not. I would call Apple’s security good, their privacy “probably good” until proven otherwise and their OS integration excellent. While some folks have had issues with iCloud reliability, it’s worked well for me over the past year, so I’m willing to give it a try as shared file storage space.

So, with that background I decided to move off of Bit Torrent Sync (which is very good, but not well integrated into OS X or iOS) and onto iCloud Drive right after I installed Yosemite on my iMac and MacBook Pro. Yes, I know, I was asking for early adopter problems and unfortunately I was not disappointed.

My initial attempt was a simple move of files into iCloud Drive using the Finder. This worked fine for a few small (< 2GB) folders. However, when I moved over a large (42 GB) folder with some animation asset collections (> 10,000 files per folder). I ran into problems.

For reference, the iCloud drive folder is hidden. You can find it in your “Library” folder at “~/Library/Mobile\ Documents/com~apple~CloudDocs”. This directory is what is mapped to the “iCloud Drive” folder in the Finder.

After much digging around, here’s what appears to happen:

  1. Upon moving files into the iCloud Folder, meta-data for each file is recorded locally for purposes of tracking file changes.
  2. The local meta-data is bundled up, apparently on a per-folder basis and transmitted up to Apple’s servers in batches as files are written to the iCloud drive folder.
  3. When a large enough group of files changes all at once (or nearly so) as happens with a large finder copy, the bundled up meta-data groupings get very large.
  4. When more than 100 files are updated all at once, the resulting meta-data bundle exceeds the size of a single database commit on Apple’s servers and the sync process registers an error and stops working.

How did I find this out? There’s a ASL-based log of all of the iCloud Drive sync processing that can be found in the directory /var/log/com.apple.clouddocs.asl. You can view it either using the Console application or using the new “brctl” command, like so:

brctl log -w

The “log” verb specifies log file viewing and “-w” invokes a “tail -F” style of continuous playback of the current log file. The Console app can do the same thing by picking the directory listed above and the most recent log file in the directory.

The tell-tail error message is as follows:

documentContent/731CDF0F-DD1C-4D7B-A627-5EED6E245B9A:(com.apple.CloudDocs:__defaultOwner__) = <CKError 0x7f8d08d919a0: "Limit Exceeded" (27/2023); server message = "Database commit size exceeds limit">

It’s the last part of that error, the “server message” that’s important. The “documentContent” code is unique to my files. You will have a different one.

To work around this problem, I did the following:

  1. I tar’d up the animation asset folders I had lying about in my source folder. I did that and the total number of files in the overall folder dropped from over 100,000 to just over 12,000.
  2. I reset the iCloud Drive meta-data store, by killing off the “bird” daemon (killall bird), removing the meta-data store which is found in ~/Library/Application\ Support/CloudDocs/ (cd ~/Library/Application\ Support ; rm -rf CloudDocs), and immediately restarting the Mac.
  3. I then waited until the iCloud Drive sync process rebuilt the meta-data and finished syncing successfully as evidenced by activity stopping in the log file (see above) without any new errors appearing. This can take several hours.
  4. Finally, I copied over the remainder of the folder slowly, leveraging a simple infinite loop in bash and the bandwidth usage control in rsync, like this:
while true; do rsync -av --delete --max-delete=10 --bwlimit=4 --exclude-from=/tmp/exclude source-dir/ iCloud; sleep 5; done

The combination of the infinite shell loop, the delete limit and bandwidth limit on rsync causes the copy to move slowly enough to avoid a mass file change, even when deleting files. This takes several hours to finish, but works without error.

The contents of /tmp/exclude were picked to avoid duplicating Apple specific file meta-data and the meta-data directories used by Bit Torrent Sync which I had been using previously. The contents of /tmp/exclude are:

.DS*
.Sync*

If I were starting to move to iCloud from scratch, I would use the shell loop and rsync shown above right from the start to avoid problems.

I hope this helps some OS X Yosemite users escape this problem and avoid several days of detective work to find and work around this issue.

NOTE: As always, when moving important files to new drives, make sure you have multiple backup copies! It’s really easy to accidentally delete things and it’s absolutely required to have a backup to compare to once you believe things have worked! I am able to say this worked because I compared the results on two Macs to a known good backup and every file matched, byte for byte.

Computable – iPython for the iPad

Now there’s a full featured iPython implementation for the iPad called Computable. It includes NumPy, SciPy, matplotlib and Pandas. It supports the creation of iPython notebooks that you can then share. This is a handy tool when a typical scientific calculator or spreadsheet just won’t cut it. Recommended.
icon
NB: Sadly, due to Apple’s restrictions on apps and executable code (including scripts) the app is not able to download or sync iPython notebooks. I understand and agree with the restriction from a security standpoint, but it does limit the usefulness of a tool like Computable.

Coin: Tempting but problematic

Wouldn’t it be great to carry just one credit card that could replace all the cards in your wallet?

Well this new digital credit card device called coin might make that possible. The video on their site shows you how this works. In brief, you swipe your card through a reader attached to your iPhone, the data is transferred to the device by bluetooth low energy and then the device can set its magnetic strip to mimic any card you’ve loaded into it. Yes, that’s right, you select a “card” on the coin display and then swipe the coin device just like a credit card.

While this sounds great, it looks to me like it has numerous problems. Here are the first few that come to mind.

The first one is acceptance. While asking the cashier of a shop in San Francisco to swipe a device instead of a card might work, I’m doubtful an hourly worker at restaurant here in the Mid-West is going to go for it. It does’t look like a real credit card and won’t pass the typical visual inspection a cashier does on a card. There’s no signature. The expiry and security code are on the display. I would imagine getting rejected more often than not.

The second one is that I believe that use of this device is beyond the limits the credit card companies put on how the card data can be used and stored. The Coin folks say in their FAQ that they are pursuing PCI DSS Certification, which will be critical. Failing to be certified to hold and transmit credit card data could result in a directive from the credit card companies to shops to not accept transactions from this device. Also, it might influence what the credit card companies are willing to do in the event of fraud.

The brings up the third issue, fraud. While I already feel a little uncomfortable giving my card to wait staff who walk off to ring up by bill, giving them this device gives them access to several cards. Coin’s answer to this concern is a proximity based lockout that engages when the device is away from your phone. Could the device be hacked and the data easily downloaded? Can a waiter switch cards, either on purpose or on accident? How about other uses? Can you swipe hotel room key cards and store them in the device? That could cause some interesting problems…

And yet… It sure is cool. I would use one if the problems with it were less.

Handy feature

I updated my trusty iPhone 4 to iOS 5. While everyone is talking about siri and all the stuff siri says, the feature I’m find most helpful is wifi sync.

Wifi Sync Logo

Once wifi sync is enabled on an iOS device, it shows up in iTunes all the time. So, if you want to put a new app on your iPhone or re-arrange the apps on the screen from iTunes, you can. And what’s more, you can initiate a sync from iTunes and update you iPhone while it’s still in your pocket. Nice.