Tag Archives: Mac

Mac OS X Lion Mail: two awesome features

I just found out about two great features of OS X Lion’s mail client, aptly called Mail: server-side, multi-color message flagging and IMAP access to Yahoo mail.

Mail servers: POP vs. IMAP

As you may know, there are two main ways, or protocols, to access your mail on a server: POP and IMAP. POP (“Post Office Protocol”) is a simple, older mechanism that allows a mail client (Apple Mail, Thunderbird, Outlook, etc.) to ask a mail server whether you have new mail, and download messages to your computer or connected device. You can also tell the server to keep a copy of the messages you download, or delete them. So, the basic premise behind POP (as I see it) is that the mail server is just a way station between the machine sending mail to you, and the mail client on your PC.

POP makes sense in a world in which Internet connectivity is slow and storage space on the server is expensive. In other words, not in today’s hyperconnected, always-online world. The basic premise behind IMAP, on the other hand, is that your mail resides on the server; your mail client can access it at any time, of course delete it if you wish, and perhaps even cache it locally for performance reasons. However, the “official” repository of your mail is on the server.

A huge advantage is the fact that it becomes very easy and safe to access your mail from many different machines—your PC at work, your laptop at home, your smartphone, etc. The mail client on each device gets to read mail on the server, but there is one central repository. With POP, you have to make sure that you instruct your client to “leave mail on the server”: if you don’t, the day will come when you are out of the office, need to read an email your boss sent you the day before, and realize that you already downloaded that email from your work PC and deleted it from the server.

IMAP has all kinds of other advantages, such as folders. The feature I want to focus on today, however, is flagging.

Custom server-side flags

The IMAP mail protocol supports flagging a message as important, and of course Mail supports this. The neat thing is that flagging is a server-side attribute: it is stored on the IMAP server, not on the client. Thus, if you flag a message on one machine (say, your work PC), then later retrieve the same message on another machine (e.g. your laptop, or your smartphone), the flag is there.

However, the IMAP protocol also supports “keywords”—additional status indications whose interpretation and visual rendering is left to the individual mail client, but that are also stored on the server. Older versions of the Mozilla Thunderbird mail client used IMAP keywords to store “labels”, which allowed you to mark a message as “Important”, “To do” or similar; messages decorated with different labels were also displayed different (non-black) color. More recent versions have “tags”, which work similarly but can also be customized by the user.

Well, it turns out that, when you flag a message in Lion Mail, you can choose a flag color other than the default red, and the color is stored on the server. In the Lion Mail client, notice that there is a pull-down menu (a triangle pointing downward) next to the Flag button: that’s where you choose the color of the flag you want. If you access your mail from a different mail client, you don’t see the color, but you do see the flag: that is, things degrade gracefully. Yay!

IMAP Access to Yahoo mail

Well, the title says it all. Yahoo offers free POP access to its mail server, in addition to access via its Web-based interface; if you want IMAP access, you have to sign up for premium service. The only exception is the iPhone and the iPad: Apple must have a special deal with them, and iOS devices do get access to Yahoo’s IMAP server.

My wife has had a Yahoo account for as long as I’ve known her, and she is very attached to it. Despite my repeated attempts to get her to switch to Gmail, she remains faithful to Yahoo. Thus, up until OS X Lion, she accessed Yahoo via their POP server on her laptop, and via the special IMAP server on her iPhone and our family iPad. This is an unholy combination; although we always tried to be careful, between laptop upgrades and mail client updates, I have always been worried about inadvertently unchecking the “leave mail on server” option, thereby making my wife’s email inaccessible from her iPhone and iPad—nowadays her primary mail devices by far.

Furthermore, my wife tends to have a rather large Inbox. Actually, this is not surprising if you are used to POP: a mail client can offer to store messages in a folder on your PC, but the server knows nothing about this. So, if you want to have cross-device access to your mail, and still keep old messages for a while, well, the only way is to leave them on the POP server, so that they show up in your Inbox everywhere. Indeed, iOS does a good job in this scenario: you only get to see 50 or so messages at a time, but if you need access to older messages, you can usually get to them quickly via search.

The problem with this approach is that, if iDevices are your primary mail machines (via IMAP), the rare time you access your mail via POP, your client has to download a lot of messages. For us, this got so out of control that my wife simply stopped using her laptop for email. This was a problem if she needed to print an email: the only practical way for her to do so was to forward the message to me, and I’d print it for her.

Well, no more: as of OS X Lion, Apple Mail has full access to Yahoo’s IMAP server. My wife is ecstatic: not only does Mail look very much like the iPad mail client, she also sees exactly the same messages with no trouble at all, and no time spent downloading old messages!

If you are a Yahoo mail user, this is a big win! A hint: if you have already configured access to Yahoo via POP, the easiest thing to do is to (archive your messages and) delete that account, then use Apple’s configuration wizard to add Yahoo back. Go to System Preferences (notMail’s preferences!), choose “Mail, Contacts and Calendars”, and click on Yahoo. Enjoy!

Mac Chrome fonts back to normal!

Yes! Version 16.0.891.1 dev (dev channel) is out, and thank goodness it restores subpixel rendering. Thank you Google!

Chrome 16 dev: degraded font rendering on the Mac

Since this morning, Chrome’s font rendering on the Mac has changed. It is now horrible! I am running ver. 16.0.891.0 dev (this is the “dev channel”, i.e. in-development version); here’s a snippet from Daring Fireball as rendered by today’s Chrome:

And here’s the same snippet, as rendered by Safari 5.1:

Another example, this time from a Wikipedia page I reference below: today’s Chrome…

…and Safari:

Until last night, I could not spot any difference in the rendering (or, if there were differences, they were not so jarring to my eye). Now, there are two key differences.

At the “macro” level, fonts as rendered by Chrome look thinner. Subjectively, they are not as legible. They also look slightly washed out.

At the “micro” level, if you magnify the Chrome and Safari snippets a few times, you will see that Safari uses subpixel rendering: loosely speaking, this means that the “edges” of character shapes (glyphs) are actually drawn in different colors, not just different tones of grays. On the other hand, Chrome seems to have inexplicably switched to purely grayscale rendering of fonts. [Note: the difference is even starker in the original TIFF screen captures, but you should be able to notice in the WordPress-friendly JPEG pictures posted above.]

The “micro” view explains the “macro” perception. Subpixel rendering is used precisely as a way to increase the perceived resolution, and—if used tastefully, as it is on Mac OS X—to achieve greater typographic fidelity. For whatever reason, today’s Chrome dev build seems to have switched that off.

Please, Google, bring subpixel-rendered fonts back, or I’ll have to switch to Safari. You already took away the ability to move between bookmark menus; please don’t take away my nice fonts!

Google bookmark bar menus – again

The latest (15.0.849.1) dev channel version of Chrome presents bookmark bar menus differently from prior versions: specifically, the menu now looks like a “standard” Mac pop-up (or menubar) menu. Previous versions used a smaller font, which is convenient if, like me, you have lots of items in your menus.

Chrome menu problem

A Chrome bookmark bar menu (15.0.849.1 dev)

As you can see, the last three entries after the separator have ampersands (&) that are supposed to define accelerators (i.e. keyboard shortcuts). They obviously shouldn’t be shown in the menu at all–the Mac UI convention is to indicate keyboard shortcuts explicitly, right-justified (i.e. after “Open All Bookmarks”, you would see “Clover-O”, where “Clover” is the funny symbol etched on the Command key). So, it may be the case that this visual change is actually a bug, not a feature.

I really hope that this is the case, because version 15.0.849.1 dev also eliminates one of my favorite Chrome features–the fact that, if you click open one bookmark bar menu, then move the mouse to another one, the corresponding menu opens up automatically. This is the default behavior on Windows, and it used to work the same way on the Mac. Alas, no more: now, you get the exact same behavior as in Safari; you click on one bookmark bar menu, the corresponding menu pops up, then you move your mouse to another menu and nothing happens–the first menu remains open. Actually, you have to click twice on another bookmark bar menu for the actual pop-up to appear.

I hope this behavior will be reverted. I don’t particularly like the visual style in version 15.0.849.1 dev, but I could live with it; however, I would really miss the ability to “hover” from one bookmark bar menu to another. Ye Chrome deities, are you listening? ;-)

Mac OS X Lion file versions, part 2

My previous post dealt with how versioning and non-versioning apps interact under the new Mac OS X Lion release. I now turn to some low-level sleuthing. The motivation is, in a way, similar: I am interested in how command-line tools such as rsync may deal with versioning. Again, the bottom line is that these tools will do exactly what they did under Snow Leopard and earlier releases, but that it is likely going to be hard to get them to back up and sync file versions. I need to do further testing on this point; feel free to provide any details you may have.

As for the previous posts, useful background reading can be found at Ars Technica and Krypted.com.

Low-level details and storage space

Here is what I found. First, the state of any versioning-aware app (that is, whether or not a given file has been edited since a version was explicitly saved) is stored in ~/Library/Saved Application State, but the actual versions of files are in a separate directory structure off the root of the current volume, specifically /.DocumentRevisions-V100. That directory looks like this on my system:

d--x--x--x   7 root  wheel   238 Jul 18 22:39 .
drwxr-xr-x  36 root  wheel  1292 Jul 28 23:24 ..
drwx------   5 root  wheel   170 Jul 28 23:27 .cs
drw-------   2 root  wheel    68 Jul 18 22:39 ChunkTemp
d--x--x--x   3 root  wheel   102 Jul 18 22:39 PerUID
drwx------   4 root  wheel   136 Jul 28 23:27 db-V1
drwx--x--x   2 root  wheel    68 Jul 18 22:39 staging

If you drill down the PerUID directory (there is a further subdirectory for each user ID, or UID, and further subdirectories off of that), you will see every single version of every single versioned file. The actual file names are replaced with hexadecimal hash codes, but extensions are preserved. You can actually open these files directly, e.g. using open file_name.ext from the command line. You may think this is very inefficient: imagine a very large Pages or TextEdit file, with versions differing only by a few characters. Does Lion actually save an entire copy of each version? This seems wasteful, and it is also not what version control systems typically do: rather, they save the “deltas,” or “patches” needed to go from one revision to the next. It turns out that Lion is smarter than this, but the details are a bit opaque.

First, as the above Ars Technica article explains, Lion keeps track of “file chunks”, and only actually saves those chunks that have changed; these are stored in binary blobs off the .cs directory. Actually figuring out which chunks have changed is pretty challenging, and again Ars Technica provides hints and links illustrating the principles and heuristics at work. The whole system is pretty sophisticated, but then again, it has to be: versions are stored in your internal hard disk (indeed, your normal working “volume”); if you are editing a movie, you would not want to keep around 10 almost identical copies of a multi-gigabyte file, would you?

Now, you may rememeber that Time Machine does something similar to keep the size of backups under control. Again, Ars Technica comes to our aid. Instead of saving a copy of your entire hard disk each time it runs, Time Machine creates hard links to files and directories that have not changed since the last backup, and only copies files that have been modified. However, all this happens at the file level. Lion’s file versioning instead operates below the file level—it tracks and saves file chunks, which requires quite a bit more cleverness.

What’s intriguing is that the above does not explain why, when you look at files off the PerUID directory, you see what appear to be full copies of each version of any given file. For example, here’s what I get by issuing ls -l /.DocumentRevisions-V100/PerUID/505/3/com.apple.documentVersions:

total 0
-r--r--r--@ 1 marciano  staff   512 Aug  6 14:00 655BB90C-85A2-4F64-A9D0-9C469DABD56E.rtf
-r--r--r--@ 1 marciano  staff   324 Aug  5 17:29 A8DD2DAF-26F1-4CCA-8C9E-1811A379939A.rtf
-r--r--r--@ 1 marciano  staff   324 Aug  5 17:29 D8BFF539-F540-4EDA-A1A2-14E74C2CE5CA.rtf
-r--r--r--@ 1 marciano  staff  5057 Aug  6 14:02 DDA0C14D-DCC6-484F-BDB4-5F99A3923EE3.rtf

Again, you can open each of these files, and you will see the corresponding revision. The last one (dated August 6 at 14:02) is apparently 5057 bytes long. This matches what you see if you issue ls -l in the directory where I keep the file itself:

total 16
-rw-r--r--  1 marciano  staff  5057 Aug  6 14:02 external.rtf

However, notice the first lines in the two displays. According to the ls manual entry, this displays the number of 512-byte blocks actually used in the directory being listed. I assume that Lion saves files in 4096-byte bloks, so a 5057-byte file requires two such blocks—or exactly 16 512-byte blocks. But note that the versions directory requires zero 512-byte blocks, despite the fact that it seemingly contains 4 non-empty files! So, what gives?

Here’s what I think is going on: if any of you readers happen to know, please use the comments section :-) As you can see in this Wikipedia entry, each file in a modern file system is assocated with a so-called “inode.” This, in turn, contains (among other things) a reference to the actual physical location on the hard disk where the file content is stored, called the “inode pointer structure”. When the OS needs to write a file to disk, it figures out where to place it, then creates an inode structure to keep track of it. However, in principle, the OS could also create an inode pointer structure pointing to physical locations where other files are stored. So, in particular, Lion could be actually writing to disk the file chunks it tracks for a particular versioned file, then create an inode whose pointer structure points to these file chunks, and finally associating a file name with that inode. If this is what is going on, then files thus created would not take up any actual space, because they would merely be pointing to file chunks saved elsewhere, whose storage size is already accounted for. Indeed, it may well be that even the “original” file (external.rtf in the above example) has an inode pointer structure pointing to tracked file chunks.

Whatever the mechanism, it is a fact that Lion’s versioning does not take up unnecessary space. To check this, I created a roughly 16MB rtf file containing only text lines. I then added one more line to it. In the above directory off /.DocumentRevisions-V100/PerUID, I see both files, each 16MB in size. At this point, the Finder reported 131,297,953,160 bytes used. Then, I ran TextEdit and, using the Versions UI, deleted the second version. The finder now reported 131,298,051,464 bytes used; there is a slight increase in hard disk usage, most likely due to virtual memory, intermediate files, or whatever—but the key thing is that hard disk usage did not go down by roughly 16MB. Then, I deleted the original version, thus keeping only the file in the regular, visible user folder: the Finder reported 131,298,071,944 bytes used. Finally, I closed TextEdit and deleted the file in my own folder; Finder finally reported 131,281,700,232 bytes used, or about 16MB less. Bingo!

The bottom line is that you can use file versioning and still retain full control on disk usage: Lion works hard not to keep redundant information on disk, and you can always decide to delete old, unused versions, thereby saving space. However, there are some implications for Dropbox (and by implication rsync):

  • first, if you keep your documents in the Dropbox folder, only the “current” revision is backed up: versions live outside the Dropbox folder, and simply do not get picked up.
  • second, if you think that you can address this issue by aliasing the DocumentRevisions-V100 directory, think again. Yes, it may work, but it’s a bad idea. Most likely, Dropbox would not be able to figure out that files off the PerUID directory occupy no space, and would instead create full copies of these files—a waste of useful (and paid-for) storage space. And that’s assuming this works at all—I haven’t tried and do not plan to!

One thing: there is some (apparent) space used in the .cs/ChunkStorage directory. Maybe this gets cleaned up occasionally—I don’t know. There is one file, .../ChunkStorage/0/0/0/1, which contains chunks; it grew much bigger after the above exercise, even though in the end I deleted the file.

A caveat, and a trick.

In the course of my investigations, I deleted the entire ./DocumentRevisions-V100 directory and its contents. Bad idea! I thought that these would be regenerated upon first saving a version of a document, but this is not the case. Actually, versioning apps will simply refuse to save! The aforelinked site krypted.com suggests recreating the DocumentRevisions-V100 directory structure; this does enable saving and versioning, but not file chunk tracking. In fact, none of the files under db-v1, for instance, was recreated upon saving in a versioned app, although version files off of PerUID were created. What worked for me was using Disk Utility to repair the disk (not just the permissions: the entire disk). You need to run Disk Utility from either the recovery partition (hold CMD+R down while you restart) or your installation DVD, if you created one using the procedure described elsewhere on the internet.

Finally, a little trick: sudo -s gives you a root shell, without the need to actually enable the root user. Cool, and very useful if you, for instance, need to navigate a directory structure that is not accessible to regular users, such as /.DocumentRevisions-V100.

Mac OS X Lion file versions, part 1

I’ve been playing around with the new Mac OS X Lion “file versioning” facility. Ars Technica as usual provides an insightful and detailed explanation of this new feature; Krypted.com provides further low-level details (but, see below).

File versioning is, essentially, a consumer-level and user-friendly version control system. Coupled with Lion’s new autosaving functionality, it aims to change and hopefully simplify the way users think about documents. For example, as you type your Pages or TextEdit document, you don’t need to worry about explicitly saving your work: Lion does this for you every once in a while, when you quit the app, or when some other event occurs that requires the system to preserve your work so far. For instance, you can email the document you are working on without first saving it, because Lion does it for you (so what you are sending is exactly what you have written so far). The bottom line is that, ideally, the user should never worry about losing her work. Saving is now a logically distinct operation: it means that the document is currently in a state that the user wants to preserve for possible future reference, even though she may want to continue editing it. Hence, the semantic change in the corresponding entry in the File menu: “Save a version.” If you use a version control system, consider this: “saving” a file just preserves the changes you’ve made, whereas “commiting” a file to a repository makes sure that, should further edits introduce bugs, you can always roll back to working code. In Lion, “saving-as-preserving” happens automatically, whereas “saving-as-committing” requires explicit user intervention, as it should.

Both autosaving and versioning are application-level, not system-level features: that is, they must be supported by each individual applications. This is similar to other Lion features, such as full-screen mode. As you will see, I think this is a very good thing, because—in my opinion—these features do not necessarily make sense for all applications, or in all circumstances. On the other hand, this raises an important question: how do these features interact with traditional file access, be it command-line or Finder-based, or simply via non-versioning apps? If you use Dropbox, this question is relevant: what exactly gets synchronized? Indeed, Dropbox is essentially a fancy front-end to the rsync utility, which I use directly for my work backups (in addition to Dropbox); again, how do versioning and autosaving interact with it?

Autosaving does not present any problems in this respect: once a file is named, further changes are saved just as if the user was a compulsive CMD-S hitter (that is, me). So, rsync and friends work just fine; there are some subtleties related to accessing a file with both a versioning and a non-versioning app, but nothing that interferes with usage: see below for details. The situation for file versioning is a bit more complicated. To investigate possible adverse effects, I had to explore the implementation of this feature. There is some information on the Internet (again, see the aforelinked articles), but a lot of details are missing, and I should warn you that I myself did not go very far—although I did uncover a few interesting tidbits. In case you are wondering, the publicly accessible Apple developer documentation has essentially zero implementation information about file versioning. For all practical purposes, Apple wants you to treat this feature as a black box.

First, the bottom line: file-level access is not hindered in any way by file versioning either. You can continue using Dropbox, rsync, or more generally any non-versioning app, and you will get pretty much the same functionality in Lion as you did in Snow Leopard. This is excellent news. On the other hand, changes made to a file using any non-versioning app will not be versioned by the operating system: again, versioning is an application-level feature, not something that is enforced by the OS. This is probably good, although it does mean that, as far as I can tell, there is no easy way to, for instance, back up versions of a file using rsync (except in a rather inefficient way). Now for the details. This post describes how accessing files with both versioning and non-versioning apps might work from a user, or app perspective; in the next, I turn to some low-level details I was able to uncover, and that I haven’t seen explained elsewhere (but, feel free to provide references if you find them).

Interacting with non-versioning apps: UI considerations

Again, the basic thing to keep in mind is that versioning is application-specific (although possibly, different apps may access the same file and keep a consistent versioning). At the same time, there is no problem accessing files from non-versioning apps. It is best to describe some concrete use cases. The TextEdit (TE) app saves RTF files, which can be opened and edited in any text editor; I used Sublime Text 2 (ST2) as an example.

Here’s a typical, fully-versioned workflow:

  1. Create a file in TE
  2. Add some text, then “save” or “save a version” in TE
  3. Now add more text in TE, and quit without saving
  4. Reopen the file with TE

In this case, the window title shows “Edited” next to the document name. Instead, consider the following:

  1. Create a file in TE
  2. Add some text, then “save” or “save a version” in TE
  3. Quit TE
  4. Now open ST2 and add more text, then save and quit
  5. Reopen the file with TE

The newly added text will show up in TE; however, the window title will not show “Edited” next to the document name. Also, the cursor will be where it was when you quit TE. Importantly, no new versions are created. In other words, versioning must be implemented along the following lines. First, upon quitting in step 3, TE saves the state of the document under consideration: that is, it records that no editing occurred after the last explicit save. Second, the next time you run TE in Step 5, it reloads the document and the configuration file where its status is saved. The latter configuration file indicates that the user did not edit the document after explicitly saving it, so, as far as TE is concerned, the document is exactly as it was when the user quit TE in step 3, even though it is not.

Again, the way to make sense of this behavior is to think that it is the application (i.e., TE), not the OS, that provides the versioning feature. If you modify a file outside TE, clearly TE knows nothing about it: you shouldn’t expect TE to keep track of such changes. On the other hand, there is one downside with this approach. Consider this sequence of events:

  1. Create a file in TE
  2. Add some text, then “save” or “save a version” in TE
  3. Add some more text in TE, so the file is now marked as “Edited”
  4. Quit TE
  5. Now open ST2 and add more text, or delete existing text, then save and quit
  6. Reopen the file with TE

Upon reopening the file in TE, the document will be marked “Edited.” However, the state of the document will of course reflect the edits done in ST2: the document state at the time you quit TE in step 4 will be lost. In the previous scenario, this was not the case, because we did an explicit save before quitting TE; thus, any change we did in ST2 could be undone by rolling back to the previously saved version. However, this option is no longer available to us in the present scenario. Again, this cannot be helped: TE knows nothing of the modifications in step 5, so when it opens the file in step 6, it just knows that some changes were made since the last save—not which changes.

The bottom line is that, if you know you will be editing a file using some non-versioning app, as well as a native, versioning app, it is a good idea to explicitly save before quitting. This is, after all, what you have been doing all along—save your documents before quitting, or sharing. The only difference is that Lion-native apps feature autosaving, and hence don’t force you to save upon quitting: you have to remind yourself to do so.

OS X Lion and Vodafone Mobile Broadband

My family and I usually spend some time in Europe during the summer; there are several great conferences to attend, and of course we take the opportunity to visit our parents and relatives. My unlocked Captivate, with a Vodafone Italy prepaid voice+data plan, solved my smartphone connectivity problems. However, I also rely on a USB 3G modem to connect with my MacBook Pro when I’m away from wifi networks. Specifically, I use a Vodafone Italy “internet key” and a nice prepaid plan (5 euros per month, and 1 euro each day I actually use the key, for up to 300MB of traffic per day, which is perfect for email and work).

After upgrading to OS X Lion, I remembered that the “internet key” (a Vodafone-branded Huawei UMTS modem) used to require a PowerPC-era driver. This is bad news: OS X Lion drops support for Rosetta, the PowerPC emulator that allowed users of previous OS X releases to run software and drivers written for older-generation Macs. Thus, no Rosetta, no PowerPC drivers, hence no internet key :-(

Well, it turns out that Vodafone is rewriting drivers and software for modern, Intel-based macs! You can get a preview version here; hat tip to Macfixit, a South African web site where I first found a reference to the aforelinked Vodafone page. [Update: the Vodafone preview page now also comes up first if you do a Google search for "vodafone mobile broadband lion preview"].

Incidentally, getting the software obviously poses a chicken-and-egg problem. I solved it by downloading it using my Captivate. Here, though, I had to work around a small hurdle: the default Captivate / Android browser would not download the .zip file with the driver, saying that it was “not meant for my platform” (or something like that). So, I tried using the Dolphin browser instead, and it worked: the file was downloaded, and I simply copied it to my Mac via USB, and installed it. Success!

Everything works like a charm so far. Looking forward to more travel blogging…