Tag Archives: Mac

Mac OS X Lion Mail: two awesome features

I just found out about two great features of OS X Lion’s mail client, aptly called Mail: server-side, multi-color message flagging and IMAP access to Yahoo mail.

Mail servers: POP vs. IMAP

As you may know, there are two main ways, or protocols, to access your mail on a server: POP and IMAP. POP (“Post Office Protocol”) is a simple, older mechanism that allows a mail client (Apple Mail, Thunderbird, Outlook, etc.) to ask a mail server whether you have new mail, and download messages to your computer or connected device. You can also tell the server to keep a copy of the messages you download, or delete them. So, the basic premise behind POP (as I see it) is that the mail server is just a way station between the machine sending mail to you, and the mail client on your PC.

POP makes sense in a world in which Internet connectivity is slow and storage space on the server is expensive. In other words, not in today’s hyperconnected, always-online world. The basic premise behind IMAP, on the other hand, is that your mail resides on the server; your mail client can access it at any time, of course delete it if you wish, and perhaps even cache it locally for performance reasons. However, the “official” repository of your mail is on the server.

A huge advantage is the fact that it becomes very easy and safe to access your mail from many different machines—your PC at work, your laptop at home, your smartphone, etc. The mail client on each device gets to read mail on the server, but there is one central repository. With POP, you have to make sure that you instruct your client to “leave mail on the server”: if you don’t, the day will come when you are out of the office, need to read an email your boss sent you the day before, and realize that you already downloaded that email from your work PC and deleted it from the server.

IMAP has all kinds of other advantages, such as folders. The feature I want to focus on today, however, is flagging.

Custom server-side flags

The IMAP mail protocol supports flagging a message as important, and of course Mail supports this. The neat thing is that flagging is a server-side attribute: it is stored on the IMAP server, not on the client. Thus, if you flag a message on one machine (say, your work PC), then later retrieve the same message on another machine (e.g. your laptop, or your smartphone), the flag is there.

However, the IMAP protocol also supports “keywords”—additional status indications whose interpretation and visual rendering is left to the individual mail client, but that are also stored on the server. Older versions of the Mozilla Thunderbird mail client used IMAP keywords to store “labels”, which allowed you to mark a message as “Important”, “To do” or similar; messages decorated with different labels were also displayed different (non-black) color. More recent versions have “tags”, which work similarly but can also be customized by the user.

Well, it turns out that, when you flag a message in Lion Mail, you can choose a flag color other than the default red, and the color is stored on the server. In the Lion Mail client, notice that there is a pull-down menu (a triangle pointing downward) next to the Flag button: that’s where you choose the color of the flag you want. If you access your mail from a different mail client, you don’t see the color, but you do see the flag: that is, things degrade gracefully. Yay!

IMAP Access to Yahoo mail

Well, the title says it all. Yahoo offers free POP access to its mail server, in addition to access via its Web-based interface; if you want IMAP access, you have to sign up for premium service. The only exception is the iPhone and the iPad: Apple must have a special deal with them, and iOS devices do get access to Yahoo’s IMAP server.

My wife has had a Yahoo account for as long as I’ve known her, and she is very attached to it. Despite my repeated attempts to get her to switch to Gmail, she remains faithful to Yahoo. Thus, up until OS X Lion, she accessed Yahoo via their POP server on her laptop, and via the special IMAP server on her iPhone and our family iPad. This is an unholy combination; although we always tried to be careful, between laptop upgrades and mail client updates, I have always been worried about inadvertently unchecking the “leave mail on server” option, thereby making my wife’s email inaccessible from her iPhone and iPad—nowadays her primary mail devices by far.

Furthermore, my wife tends to have a rather large Inbox. Actually, this is not surprising if you are used to POP: a mail client can offer to store messages in a folder on your PC, but the server knows nothing about this. So, if you want to have cross-device access to your mail, and still keep old messages for a while, well, the only way is to leave them on the POP server, so that they show up in your Inbox everywhere. Indeed, iOS does a good job in this scenario: you only get to see 50 or so messages at a time, but if you need access to older messages, you can usually get to them quickly via search.

The problem with this approach is that, if iDevices are your primary mail machines (via IMAP), the rare time you access your mail via POP, your client has to download a lot of messages. For us, this got so out of control that my wife simply stopped using her laptop for email. This was a problem if she needed to print an email: the only practical way for her to do so was to forward the message to me, and I’d print it for her.

Well, no more: as of OS X Lion, Apple Mail has full access to Yahoo’s IMAP server. My wife is ecstatic: not only does Mail look very much like the iPad mail client, she also sees exactly the same messages with no trouble at all, and no time spent downloading old messages!

If you are a Yahoo mail user, this is a big win! A hint: if you have already configured access to Yahoo via POP, the easiest thing to do is to (archive your messages and) delete that account, then use Apple’s configuration wizard to add Yahoo back. Go to System Preferences (notMail’s preferences!), choose “Mail, Contacts and Calendars”, and click on Yahoo. Enjoy!

Mac Chrome fonts back to normal!

Yes! Version 16.0.891.1 dev (dev channel) is out, and thank goodness it restores subpixel rendering. Thank you Google!

Chrome 16 dev: degraded font rendering on the Mac

Since this morning, Chrome’s font rendering on the Mac has changed. It is now horrible! I am running ver. 16.0.891.0 dev (this is the “dev channel”, i.e. in-development version); here’s a snippet from Daring Fireball as rendered by today’s Chrome:

And here’s the same snippet, as rendered by Safari 5.1:

Another example, this time from a Wikipedia page I reference below: today’s Chrome…

…and Safari:

Until last night, I could not spot any difference in the rendering (or, if there were differences, they were not so jarring to my eye). Now, there are two key differences.

At the “macro” level, fonts as rendered by Chrome look thinner. Subjectively, they are not as legible. They also look slightly washed out.

At the “micro” level, if you magnify the Chrome and Safari snippets a few times, you will see that Safari uses subpixel rendering: loosely speaking, this means that the “edges” of character shapes (glyphs) are actually drawn in different colors, not just different tones of grays. On the other hand, Chrome seems to have inexplicably switched to purely grayscale rendering of fonts. [Note: the difference is even starker in the original TIFF screen captures, but you should be able to notice in the WordPress-friendly JPEG pictures posted above.]

The “micro” view explains the “macro” perception. Subpixel rendering is used precisely as a way to increase the perceived resolution, and—if used tastefully, as it is on Mac OS X—to achieve greater typographic fidelity. For whatever reason, today’s Chrome dev build seems to have switched that off.

Please, Google, bring subpixel-rendered fonts back, or I’ll have to switch to Safari. You already took away the ability to move between bookmark menus; please don’t take away my nice fonts!

Google bookmark bar menus – again

The latest (15.0.849.1) dev channel version of Chrome presents bookmark bar menus differently from prior versions: specifically, the menu now looks like a “standard” Mac pop-up (or menubar) menu. Previous versions used a smaller font, which is convenient if, like me, you have lots of items in your menus.

Chrome menu problem

A Chrome bookmark bar menu (15.0.849.1 dev)

As you can see, the last three entries after the separator have ampersands (&) that are supposed to define accelerators (i.e. keyboard shortcuts). They obviously shouldn’t be shown in the menu at all–the Mac UI convention is to indicate keyboard shortcuts explicitly, right-justified (i.e. after “Open All Bookmarks”, you would see “Clover-O”, where “Clover” is the funny symbol etched on the Command key). So, it may be the case that this visual change is actually a bug, not a feature.

I really hope that this is the case, because version 15.0.849.1 dev also eliminates one of my favorite Chrome features–the fact that, if you click open one bookmark bar menu, then move the mouse to another one, the corresponding menu opens up automatically. This is the default behavior on Windows, and it used to work the same way on the Mac. Alas, no more: now, you get the exact same behavior as in Safari; you click on one bookmark bar menu, the corresponding menu pops up, then you move your mouse to another menu and nothing happens–the first menu remains open. Actually, you have to click twice on another bookmark bar menu for the actual pop-up to appear.

I hope this behavior will be reverted. I don’t particularly like the visual style in version 15.0.849.1 dev, but I could live with it; however, I would really miss the ability to “hover” from one bookmark bar menu to another. Ye Chrome deities, are you listening? ;-)

Mac OS X Lion file versions, part 2

My previous post dealt with how versioning and non-versioning apps interact under the new Mac OS X Lion release. I now turn to some low-level sleuthing. The motivation is, in a way, similar: I am interested in how command-line tools such as rsync may deal with versioning. Again, the bottom line is that these tools will do exactly what they did under Snow Leopard and earlier releases, but that it is likely going to be hard to get them to back up and sync file versions. I need to do further testing on this point; feel free to provide any details you may have.

As for the previous posts, useful background reading can be found at Ars Technica and Krypted.com.

Low-level details and storage space

Here is what I found. First, the state of any versioning-aware app (that is, whether or not a given file has been edited since a version was explicitly saved) is stored in ~/Library/Saved Application State, but the actual versions of files are in a separate directory structure off the root of the current volume, specifically /.DocumentRevisions-V100. That directory looks like this on my system:

d--x--x--x   7 root  wheel   238 Jul 18 22:39 .
drwxr-xr-x  36 root  wheel  1292 Jul 28 23:24 ..
drwx------   5 root  wheel   170 Jul 28 23:27 .cs
drw-------   2 root  wheel    68 Jul 18 22:39 ChunkTemp
d--x--x--x   3 root  wheel   102 Jul 18 22:39 PerUID
drwx------   4 root  wheel   136 Jul 28 23:27 db-V1
drwx--x--x   2 root  wheel    68 Jul 18 22:39 staging

If you drill down the PerUID directory (there is a further subdirectory for each user ID, or UID, and further subdirectories off of that), you will see every single version of every single versioned file. The actual file names are replaced with hexadecimal hash codes, but extensions are preserved. You can actually open these files directly, e.g. using open file_name.ext from the command line. You may think this is very inefficient: imagine a very large Pages or TextEdit file, with versions differing only by a few characters. Does Lion actually save an entire copy of each version? This seems wasteful, and it is also not what version control systems typically do: rather, they save the “deltas,” or “patches” needed to go from one revision to the next. It turns out that Lion is smarter than this, but the details are a bit opaque.

First, as the above Ars Technica article explains, Lion keeps track of “file chunks”, and only actually saves those chunks that have changed; these are stored in binary blobs off the .cs directory. Actually figuring out which chunks have changed is pretty challenging, and again Ars Technica provides hints and links illustrating the principles and heuristics at work. The whole system is pretty sophisticated, but then again, it has to be: versions are stored in your internal hard disk (indeed, your normal working “volume”); if you are editing a movie, you would not want to keep around 10 almost identical copies of a multi-gigabyte file, would you?

Now, you may rememeber that Time Machine does something similar to keep the size of backups under control. Again, Ars Technica comes to our aid. Instead of saving a copy of your entire hard disk each time it runs, Time Machine creates hard links to files and directories that have not changed since the last backup, and only copies files that have been modified. However, all this happens at the file level. Lion’s file versioning instead operates below the file level—it tracks and saves file chunks, which requires quite a bit more cleverness.

What’s intriguing is that the above does not explain why, when you look at files off the PerUID directory, you see what appear to be full copies of each version of any given file. For example, here’s what I get by issuing ls -l /.DocumentRevisions-V100/PerUID/505/3/com.apple.documentVersions:

total 0
-r--r--r--@ 1 marciano  staff   512 Aug  6 14:00 655BB90C-85A2-4F64-A9D0-9C469DABD56E.rtf
-r--r--r--@ 1 marciano  staff   324 Aug  5 17:29 A8DD2DAF-26F1-4CCA-8C9E-1811A379939A.rtf
-r--r--r--@ 1 marciano  staff   324 Aug  5 17:29 D8BFF539-F540-4EDA-A1A2-14E74C2CE5CA.rtf
-r--r--r--@ 1 marciano  staff  5057 Aug  6 14:02 DDA0C14D-DCC6-484F-BDB4-5F99A3923EE3.rtf

Again, you can open each of these files, and you will see the corresponding revision. The last one (dated August 6 at 14:02) is apparently 5057 bytes long. This matches what you see if you issue ls -l in the directory where I keep the file itself:

total 16
-rw-r--r--  1 marciano  staff  5057 Aug  6 14:02 external.rtf

However, notice the first lines in the two displays. According to the ls manual entry, this displays the number of 512-byte blocks actually used in the directory being listed. I assume that Lion saves files in 4096-byte bloks, so a 5057-byte file requires two such blocks—or exactly 16 512-byte blocks. But note that the versions directory requires zero 512-byte blocks, despite the fact that it seemingly contains 4 non-empty files! So, what gives?

Here’s what I think is going on: if any of you readers happen to know, please use the comments section :-) As you can see in this Wikipedia entry, each file in a modern file system is assocated with a so-called “inode.” This, in turn, contains (among other things) a reference to the actual physical location on the hard disk where the file content is stored, called the “inode pointer structure”. When the OS needs to write a file to disk, it figures out where to place it, then creates an inode structure to keep track of it. However, in principle, the OS could also create an inode pointer structure pointing to physical locations where other files are stored. So, in particular, Lion could be actually writing to disk the file chunks it tracks for a particular versioned file, then create an inode whose pointer structure points to these file chunks, and finally associating a file name with that inode. If this is what is going on, then files thus created would not take up any actual space, because they would merely be pointing to file chunks saved elsewhere, whose storage size is already accounted for. Indeed, it may well be that even the “original” file (external.rtf in the above example) has an inode pointer structure pointing to tracked file chunks.

Whatever the mechanism, it is a fact that Lion’s versioning does not take up unnecessary space. To check this, I created a roughly 16MB rtf file containing only text lines. I then added one more line to it. In the above directory off /.DocumentRevisions-V100/PerUID, I see both files, each 16MB in size. At this point, the Finder reported 131,297,953,160 bytes used. Then, I ran TextEdit and, using the Versions UI, deleted the second version. The finder now reported 131,298,051,464 bytes used; there is a slight increase in hard disk usage, most likely due to virtual memory, intermediate files, or whatever—but the key thing is that hard disk usage did not go down by roughly 16MB. Then, I deleted the original version, thus keeping only the file in the regular, visible user folder: the Finder reported 131,298,071,944 bytes used. Finally, I closed TextEdit and deleted the file in my own folder; Finder finally reported 131,281,700,232 bytes used, or about 16MB less. Bingo!

The bottom line is that you can use file versioning and still retain full control on disk usage: Lion works hard not to keep redundant information on disk, and you can always decide to delete old, unused versions, thereby saving space. However, there are some implications for Dropbox (and by implication rsync):

  • first, if you keep your documents in the Dropbox folder, only the “current” revision is backed up: versions live outside the Dropbox folder, and simply do not get picked up.
  • second, if you think that you can address this issue by aliasing the DocumentRevisions-V100 directory, think again. Yes, it may work, but it’s a bad idea. Most likely, Dropbox would not be able to figure out that files off the PerUID directory occupy no space, and would instead create full copies of these files—a waste of useful (and paid-for) storage space. And that’s assuming this works at all—I haven’t tried and do not plan to!

One thing: there is some (apparent) space used in the .cs/ChunkStorage directory. Maybe this gets cleaned up occasionally—I don’t know. There is one file, .../ChunkStorage/0/0/0/1, which contains chunks; it grew much bigger after the above exercise, even though in the end I deleted the file.

A caveat, and a trick.

In the course of my investigations, I deleted the entire ./DocumentRevisions-V100 directory and its contents. Bad idea! I thought that these would be regenerated upon first saving a version of a document, but this is not the case. Actually, versioning apps will simply refuse to save! The aforelinked site krypted.com suggests recreating the DocumentRevisions-V100 directory structure; this does enable saving and versioning, but not file chunk tracking. In fact, none of the files under db-v1, for instance, was recreated upon saving in a versioned app, although version files off of PerUID were created. What worked for me was using Disk Utility to repair the disk (not just the permissions: the entire disk). You need to run Disk Utility from either the recovery partition (hold CMD+R down while you restart) or your installation DVD, if you created one using the procedure described elsewhere on the internet.

Finally, a little trick: sudo -s gives you a root shell, without the need to actually enable the root user. Cool, and very useful if you, for instance, need to navigate a directory structure that is not accessible to regular users, such as /.DocumentRevisions-V100.

Mac OS X Lion file versions, part 1

I’ve been playing around with the new Mac OS X Lion “file versioning” facility. Ars Technica as usual provides an insightful and detailed explanation of this new feature; Krypted.com provides further low-level details (but, see below).

File versioning is, essentially, a consumer-level and user-friendly version control system. Coupled with Lion’s new autosaving functionality, it aims to change and hopefully simplify the way users think about documents. For example, as you type your Pages or TextEdit document, you don’t need to worry about explicitly saving your work: Lion does this for you every once in a while, when you quit the app, or when some other event occurs that requires the system to preserve your work so far. For instance, you can email the document you are working on without first saving it, because Lion does it for you (so what you are sending is exactly what you have written so far). The bottom line is that, ideally, the user should never worry about losing her work. Saving is now a logically distinct operation: it means that the document is currently in a state that the user wants to preserve for possible future reference, even though she may want to continue editing it. Hence, the semantic change in the corresponding entry in the File menu: “Save a version.” If you use a version control system, consider this: “saving” a file just preserves the changes you’ve made, whereas “commiting” a file to a repository makes sure that, should further edits introduce bugs, you can always roll back to working code. In Lion, “saving-as-preserving” happens automatically, whereas “saving-as-committing” requires explicit user intervention, as it should.

Both autosaving and versioning are application-level, not system-level features: that is, they must be supported by each individual applications. This is similar to other Lion features, such as full-screen mode. As you will see, I think this is a very good thing, because—in my opinion—these features do not necessarily make sense for all applications, or in all circumstances. On the other hand, this raises an important question: how do these features interact with traditional file access, be it command-line or Finder-based, or simply via non-versioning apps? If you use Dropbox, this question is relevant: what exactly gets synchronized? Indeed, Dropbox is essentially a fancy front-end to the rsync utility, which I use directly for my work backups (in addition to Dropbox); again, how do versioning and autosaving interact with it?

Autosaving does not present any problems in this respect: once a file is named, further changes are saved just as if the user was a compulsive CMD-S hitter (that is, me). So, rsync and friends work just fine; there are some subtleties related to accessing a file with both a versioning and a non-versioning app, but nothing that interferes with usage: see below for details. The situation for file versioning is a bit more complicated. To investigate possible adverse effects, I had to explore the implementation of this feature. There is some information on the Internet (again, see the aforelinked articles), but a lot of details are missing, and I should warn you that I myself did not go very far—although I did uncover a few interesting tidbits. In case you are wondering, the publicly accessible Apple developer documentation has essentially zero implementation information about file versioning. For all practical purposes, Apple wants you to treat this feature as a black box.

First, the bottom line: file-level access is not hindered in any way by file versioning either. You can continue using Dropbox, rsync, or more generally any non-versioning app, and you will get pretty much the same functionality in Lion as you did in Snow Leopard. This is excellent news. On the other hand, changes made to a file using any non-versioning app will not be versioned by the operating system: again, versioning is an application-level feature, not something that is enforced by the OS. This is probably good, although it does mean that, as far as I can tell, there is no easy way to, for instance, back up versions of a file using rsync (except in a rather inefficient way). Now for the details. This post describes how accessing files with both versioning and non-versioning apps might work from a user, or app perspective; in the next, I turn to some low-level details I was able to uncover, and that I haven’t seen explained elsewhere (but, feel free to provide references if you find them).

Interacting with non-versioning apps: UI considerations

Again, the basic thing to keep in mind is that versioning is application-specific (although possibly, different apps may access the same file and keep a consistent versioning). At the same time, there is no problem accessing files from non-versioning apps. It is best to describe some concrete use cases. The TextEdit (TE) app saves RTF files, which can be opened and edited in any text editor; I used Sublime Text 2 (ST2) as an example.

Here’s a typical, fully-versioned workflow:

  1. Create a file in TE
  2. Add some text, then “save” or “save a version” in TE
  3. Now add more text in TE, and quit without saving
  4. Reopen the file with TE

In this case, the window title shows “Edited” next to the document name. Instead, consider the following:

  1. Create a file in TE
  2. Add some text, then “save” or “save a version” in TE
  3. Quit TE
  4. Now open ST2 and add more text, then save and quit
  5. Reopen the file with TE

The newly added text will show up in TE; however, the window title will not show “Edited” next to the document name. Also, the cursor will be where it was when you quit TE. Importantly, no new versions are created. In other words, versioning must be implemented along the following lines. First, upon quitting in step 3, TE saves the state of the document under consideration: that is, it records that no editing occurred after the last explicit save. Second, the next time you run TE in Step 5, it reloads the document and the configuration file where its status is saved. The latter configuration file indicates that the user did not edit the document after explicitly saving it, so, as far as TE is concerned, the document is exactly as it was when the user quit TE in step 3, even though it is not.

Again, the way to make sense of this behavior is to think that it is the application (i.e., TE), not the OS, that provides the versioning feature. If you modify a file outside TE, clearly TE knows nothing about it: you shouldn’t expect TE to keep track of such changes. On the other hand, there is one downside with this approach. Consider this sequence of events:

  1. Create a file in TE
  2. Add some text, then “save” or “save a version” in TE
  3. Add some more text in TE, so the file is now marked as “Edited”
  4. Quit TE
  5. Now open ST2 and add more text, or delete existing text, then save and quit
  6. Reopen the file with TE

Upon reopening the file in TE, the document will be marked “Edited.” However, the state of the document will of course reflect the edits done in ST2: the document state at the time you quit TE in step 4 will be lost. In the previous scenario, this was not the case, because we did an explicit save before quitting TE; thus, any change we did in ST2 could be undone by rolling back to the previously saved version. However, this option is no longer available to us in the present scenario. Again, this cannot be helped: TE knows nothing of the modifications in step 5, so when it opens the file in step 6, it just knows that some changes were made since the last save—not which changes.

The bottom line is that, if you know you will be editing a file using some non-versioning app, as well as a native, versioning app, it is a good idea to explicitly save before quitting. This is, after all, what you have been doing all along—save your documents before quitting, or sharing. The only difference is that Lion-native apps feature autosaving, and hence don’t force you to save upon quitting: you have to remind yourself to do so.

OS X Lion and Vodafone Mobile Broadband

My family and I usually spend some time in Europe during the summer; there are several great conferences to attend, and of course we take the opportunity to visit our parents and relatives. My unlocked Captivate, with a Vodafone Italy prepaid voice+data plan, solved my smartphone connectivity problems. However, I also rely on a USB 3G modem to connect with my MacBook Pro when I’m away from wifi networks. Specifically, I use a Vodafone Italy “internet key” and a nice prepaid plan (5 euros per month, and 1 euro each day I actually use the key, for up to 300MB of traffic per day, which is perfect for email and work).

After upgrading to OS X Lion, I remembered that the “internet key” (a Vodafone-branded Huawei UMTS modem) used to require a PowerPC-era driver. This is bad news: OS X Lion drops support for Rosetta, the PowerPC emulator that allowed users of previous OS X releases to run software and drivers written for older-generation Macs. Thus, no Rosetta, no PowerPC drivers, hence no internet key :-(

Well, it turns out that Vodafone is rewriting drivers and software for modern, Intel-based macs! You can get a preview version here; hat tip to Macfixit, a South African web site where I first found a reference to the aforelinked Vodafone page. [Update: the Vodafone preview page now also comes up first if you do a Google search for "vodafone mobile broadband lion preview"].

Incidentally, getting the software obviously poses a chicken-and-egg problem. I solved it by downloading it using my Captivate. Here, though, I had to work around a small hurdle: the default Captivate / Android browser would not download the .zip file with the driver, saying that it was “not meant for my platform” (or something like that). So, I tried using the Dolphin browser instead, and it worked: the file was downloaded, and I simply copied it to my Mac via USB, and installed it. Success!

Everything works like a charm so far. Looking forward to more travel blogging…

Thoughts on Mac OS X Lion and natural scrolling

Ok, so I was not a super-early adopter: I waited until July 22nd to buy the latest OS X release from the Mac App Store. (Incidentally, a few hours later I received an email from my university’s IT department saying that we will soon be able to get it for free thanks to a licensing deal with Apple… oh well…).

In a nutshell, so far, so good. First of all, I have yet to find a major compatibility issue, which is great–no downtime. Second, the UI tweaks are cool, including the disappearing space bars (which make a lot of sense: when was the last time you actually used a space bar, given that you can scroll with your touchpad?) Third, I had a hard time understanding exactly how Mission Control was supposed to work when Jobs introduced it, and I was a bit worried about leaving the comfort of Spaces and Expose; but, after using it for less than 24 hours, it’s already growing on me. After all, it’s just Spaces and Expose combined, with support for full-screen apps to boot. By the way, I think the new full-screen mode would be great for LaTeX editing on small screens: you maximize the editing and display surfaces, and have a convenient way to switch back and forth.

Now for the main point of this post. Lion introduces what Apple calls “natural” scrolling. In previous OS X releases up to and including Snow Leopard (as well as on Windows and Linux, on laptops with a decent trackpad), to scroll a page down in your Web browser, word processor, etc., you swipe two fingers down. This may seem natural enough: after all, if you use the scroll bar, you pull the bar down to scroll down, right? But think about this: how do you scroll down on a touchscreen phone (iPhone or Android) or tablet? It’s the other way round: to scroll down, you swipe your fingers up. This makes a lot of sense: if you are reading line-by-line down a long sheet of paper, you want to move the sheet up as you read (think of Hermione Granger reading a scroll in the forbidden section of the Hogwarts library…). Well, this scrolling behavior makes just as much sense on a laptop with a touchpad, and OS X Lion makes it the default. I am quickly getting used to it, because, well, I am already familiar with it: that’s exactly how everything on my Captivate scrolls!

The above discussion was about scrolling a document up or down. But how about left and right? Switching desktops, or to different full-screen apps, also works the “natural” way on Lion: you swipe right to get the desktop (or app) to the left of your current desktop (or app). The animation makes it clear that what you are doing by swiping right is, metaphorically, shifting a very wide sheet of paper where your desktops are “drawn” to the right. The Launchpad app, which is essentially the OS X version of the iOS (and Android) home screen, works the same way.

The question is now, which swiping motion is more natural for moving back and forward in a Web browser? I can see a clear argument for also using “natural” scrolling: think of the pages you visit as if they were drawn on a wide sheet of paper; then, you should swipe right to move the entire sheet right, and hence see the page that was on the left of the current one–that is, the last page you visited before the current one. But, Safari (and indeed Chrome, and Firefox, too, I believe) have long supported the opposite convention: to go back, you swipe left. Why does this make sense? True, the back and forward arrows have always pointed to the left and right respectively (at least, in left-to-right locales). So, maybe the swiping motion is supposed to mimic the direction of the arrow. But, if the back arrow points to the left, doesn’t that mean that you really should think of the last page as being to the left of the current one, so that what you want to do is move the current one to the right to make way for the previous one?

Mission Control is weird in that it supports both metaphors. In the Trackpad preferences, I set “Swipe between pages” to use three fingers, and “Swipe between full-screen apps” to use four; the defaults are two and three fingers respectively, but Chrome won’t recognize two-finger swipes to go back/forward (Safari will). As a result, if I swipe with four fingers in Mission Control, then the metaphor is the “natural” one: I swipe right, and get the desktop to the left of the current one. But, if I swipe with three fingers, it’s the other way round. (Using the default setting, three-finger swipes would give you the “natural” behavior, and two-finger swipes would give you the back/forward behavior).

Anyway, if the above was confusing, don’t worry: you can actually disable “natural” vertical scrolling, and any or all swiping motions for that matter. Still, thinking about these issues highlights the extent to which the iPhone first, and other touchscreen devices later, have changed the way we interact with other computing devices, and still do. Which, after all, was the meaning of the slogan Apple used to announce the OS X Lion release: Back to the Mac.

Instant Preview: Some Raw Data

Attached file: latexTest-line.py. Remove .doc extension after downloading!

This is a follow-up to my earlier post on Instant Preview (IP) with LaTeX. After thinking about different IP solutions, I decided to run a little experiment. How can I measure the “instantness” of IP on the Mac, when compiling to either DVI or PDF output?

My first attempt involved a combination of a text editor, a script, and suitable DVI and PDF previewers. As text editor, I used TextWrangler, because it can be easily controlled via AppleScript, the Mac OS X native scripting language (actually, I should say "scripting infrastructure," as will be clear momentarily). In case you were wondering: yes, I tried using my beloved TextMate, and no, it could not be made to work because it does not expose an AppleScript interface. I then wrote a simple AppleScript that routinely asked TextWrangler if new text had been entered in the main document window. If so, the script asked TextWrangler to save the file, then launched the latex compiler to generate either a DVI or a PDF file, and finally asked the viewer app to refresh the document being displayed. In other words, I implemented a (very) poor man’s version of Flashmode.

As PDF viewer, the best choice for this experiment turned out to be TeXshop; it is scriptable and fast. Skim is also workable, but seems a bit slower, and requires setting "unsupported" defaults in order to refresh the current page in the background, without either asking for user confirmation or displaying an annoying (and time-consuming) progress bar. Finally, Apple’s on Preview app is not scriptable.

I was worried about DVI files. There is no native DVI previewer on the Mac, except for the venerable (and slow, not to mention for-pay) MacDviX. Skim will display DVI files, but it actually converts them to PDF files, which is obviously a problem if the whole point of the exercise is to measure DVI vs. PDF rendering speed. However, it turns out that Texlive, the TeX distribution upon which MacTeX is based, ships with the ancient but trusty xdvi. While xdvi runs under the X Windows system, and is a bit offensive to modern aesthetic sensibilities, it gets the job done–fast. Furthermore, getting xdvi to reload the current file can be achieved by sending it the SIGUSR1 Unix signal.

Long story short, my little AppleScript somehow worked, and clearly indicated that DVI is perceptibly faster than PDF as an output format for IP. While both xdvi and TeXshop lagged behind my typing, TeXshop lagged more (pretty much the same as with Flashmode, which is of course what one should expect). The question was how to quantify this difference. After discarding more "creative" solutions (I seriously considered filming myself while typing to a beat set by metronome!) I figured out that the best way to get some hard numbers was to simulate typing via a script. Now, AppleScript is Apple’s native scripting solution, but it just isn’t as powerful as, say, Python for anything even slightly more elaborate that sending a few method calls to a running application. So, Python it was.

One minor issue I had to deal with was how to control TeXshop from Python. The naive, slow way is to use os.system() or subprocess.Popen to launch the osascript command-line utility, passing the appropriate AppleScript instructions as argument. The smarter, much faster way is to use py-appscript. The only catch is that, to install the latter, you must have the GCC compiler installed, and that comes with the whole XCode environment, which is fairly big. Anyway, if you do have a development environment up and running, installing py-appscript is a simple matter. Then, you just import appscript in your Python code, and can send AppleScript instructions using nicely Pythonic syntax. And, again, things are quite a bit faster. As far as controlling xdvi is concerned, sending Unix signals from Python is a simple matter.

The Python script I used is attached to this post, in case you are interested (please remove the .doc extension: it is needed to keep WordPress happy). As described below, I ran a series of tests, each requiring slight modifications to the code; see the comments in the file for details. The basic idea is simple: the Python script "types" a line of text in the "middle" of a file, one character at a time; more precisely, for each integer n it creates a file containing

  1. a LaTeX preamble, the title and author of the "paper," and an initial fixed line of text;
  2. the first n characters of the line being "typed"; and
  3. in some tests, additional "body text," ranging from a few paragraphs to about 21 pages.

Each time a new file is created, the Python file compiles it to either DVI or PDF, then refreshes the viewer–just like my earlier Applescript did, except that, again, typing is simulated. The advantage is that I can easily time the execution of the script, using Python’s time module.

For each output format, I measured the time required to compile, but not display, the "typed" line (from the first to the last character), and then the time required to compile and display it. Furthermore, I experimented with different amounts of text after the "typed" line, so as to simulate IP in a fragment of text vs. in a medium-sized file. The tests were conducted on my Early 2008 iMac (a 3.06 GHz machine, codenamed iMac8,1 according to Apple’s convention). All measurements are taken from a single run of the script; I repeated the experiment several times, but found no significant difference. Without further ado, the results are shown in the following table.

Instant Preview: DVI vs PDF

To interpret the picture, “dvi/pdf, no body” means that there is no further text after the simulated “typed-in” line; “dvi/pdf, 1x body” means that there are a few paragraphs of subsequent text, consisting of a theorem declaration, a displayed equation, and some gibberish text; finally, “dvi/pdf, full body” means that the paragraphs just described are copied 50 times, so that the document is 22 pages long overall.

The results: I think there are four clear conclusions to be drawn from this little experiment. First, DVI files are measurably faster to render than PDF files; the time to display DVI output is negligible relative to the time it takes to actually generate it from LaTeX input. The same is certainly not true for PDF output. Now, this may in part be due to the fact that sending a SIGUSR1 signal to xdvi is faster than going through the Applescript infrastructure; I doubt that this is the most significant source of discrepancy, but in any event there is no other way to control TeXshop as far as I know (and for the time being).

Second, generating PDF files also takes longer. This exacerbates the differences in rendering times. For instance, without any additional body text, just generating the DVI files corresponding to the simulated typed line takes 11.26 seconds; when the output format is set to PDF, 13.98 seconds are needed—that is, 24% longer. If we include rendering, the time goes from 11.44 seconds to 15.13 seconds–an increase of 32%. If we include only a single copy of the body text, then just compiling takes 11.27 seconds for DVI and 15.18 for PDF, i.e. 35% longer; if we include rendering, the total time required is 11.65 seconds for DVI and 18.05 for PDF, i.e. 55% longer!

Third, when working with a medium-sized or large file, recompiling only the current page makes a difference. This can be seen by comparing the results for the "1x body" and "full body" experiments.

Fourth, and finally, PDF display is too slow to keep up with regular typing speed (as opposed to "hunt and peck" typing). To get an idea, consider the no-body test; the simulated typed-in line had 77 characters, and it took 15.13 seconds to "type" it all (i.e. compile and render the corresponding sequence of files). This translates to 77/15.13=5.1 characters per second. With 1x body text, "typing speed" goes down to 77/18.05=4.26 characters per second. You almost surely type faster than that. Try this test for instance (it’s fun!). I am no touch typist, but I got 7.48 characters per minute. If you are like me, or faster (which is likely), a naive implementation of IP that does what my simple Python script does on a machine no faster than my 3.06GHz iMac will lag behind you as you type. It won’t be quite so "instant." This is exactly my experience with Flashmode. In the case of DVI output, with 1x body text the Python script produces 77/11.65 = 6.6 characters per minute, which is not as fast as I type (according to the above-linked test) but not far behind either.

This test is only a starting point; it suggests where one can try to optimize processing. As already noted, focusing on the current page, rather than recompiling the entire file, makes a difference. But my Python script can be modified to experiment, for instance, with the format file trick described in my previous post. Furthermore, to output PDF files it is necessary to include, or embed, all required fonts; the pdftex manual suggests that this may be avoided, if the fonts are available system-wide. Perhaps that, too, saves time. In any case, more experimentation, followed by more serious hacking, is required.

LaTeX and Instant Preview

Despite my love for LaTeX typesetting, even I am forced to admit that the edit-compile-preview cycle lacks the immediacy of other inferior but WYSIWYG solutions. This is not always a problem, but it sometimes is. "Instant Preview" (IP for short) is a potential (if not yet fully realized), almost complete, best-of-both-worlds solution.

What is Instant Preview?

A picture is worth a thousand words; but, in this case, a Flash animation is worth even more. Take a look at this page, which demonstrates IP as implemented in a TeX distribution for Windows called BaKoMa TeX (I hope I got the capitalization right). Basically, you enter text in an edit window, and the output file is automatically regenerated and redisplayed on each keystroke, thereby giving the illusion of actually working "in the PDF file" (or, for BaKoMa TeX, in the DVI file, TeX’s older output format). IP was most famously implemented in the TeXtures TeX system for traditional Mac OS; Whizzytex is a well-known Emacs-based solution that achieves the same effect.

On modern Mac OS X there are two solutions (that I know of). One is the brand-new Latexian text editor, which features a built-in IP mode, but is fairly slow. The other is Flashmode, a solution by Claus Gerhardt based on the well-known TeXshop integrated editor and previewer. I decided to experiment with Flashmode this week.

Flashmode: installation and use

First of all, if you have the MacTeX LaTeX distribution, you most likely already have TeXshop. Then, you need to obtain Flashmode itself from Claus Gerhardt’s page; download the TeXhelpers package, which also contains other goodies. The Flashmode software comes with an exhaustive manual; however, let me point out a few caveats that caused me minor headaches and wasted time.

  1. After opening the TeXhelpers DMG (disk image) file, you must copy both the Flashmode and the Local Switcher apps to your Applications folder; you will need them both
  2. For some reason, neither Flashmode nor Local Switcher are indexed by Spotlight. So, if you are like me and launch most software (that you don’t already have on your Dock) using Spotlight, you’re out of luck: you need to open the Applications folder, or install scripts that launch Flashmode from inside TeXshop. Of course, you can also put Flashmode in your dock.
  3. You must create links to your current TeX distribution in your home directory: this is what Local Switcher is for. Fortunately, this is to be done only when you first install Flashmode, and every time you update or change your TeX distro. Note: the Flashmode manual states that you no longer need to create these symlinks, but my experience is that you do need them. [Update:Claus has just released a new version, numbered 6.1, that addresses this issue.]
  4. Finally, every TeX file you work on must have a format directive as its first line: that is, the very first line must be something like %&pdflatex, by itself (if the file is to be compiled using pdflatex; otherwise, e.g. %&pdftex or %&xelatex).

After installation, Flashmode is relatively painless to use. You open a tex file in TeXshop, perhaps typeset it once, then launch Flashmode and press the "Run Flashmode" button while the tex source file is the topmost window in TeXshop, right below the Flashmode window. Once again, make sure that the very first line of your file contains the text %pdflatex, by itself. Then you basically forget about it, and just watch as typeset PDF text automagically materializes in your preview window as you type in the source window. If there is a TeX/LaTeX error, the preview window is simply not updated. When you are done editing your file, switch to the Flashmode window and click the "Quit Flashmode" button. Read the manual for further information.

Now, before you get your hopes up: this thing is not really real-time preview. Working on a small- to medium-sized file, the PDF output gets updated maybe once every 1 or 2 seconds. However, after using Flashmode for a while, a strange thing happens: the source window sort of "fades in the background", and you tend to focus more on the output window. I mean this in a new-agey, psychological sense, not literally of course: you sort of stop paying attention exclusively to the TeX source, and rely a lot more on the PDF output.

I realized that this was happening because I did not find TeXshop’s editor frustrating, which I usually do (addicted as I am to TextMate). In particular, syntax highlighting is really minimal in TeXshop, but in the end it does not matter that much, because you can see the actual typeset output materialize before your eyes, without user intervention.

I think the killer application for Flashmode are Beamer presentations. As long as presentations are short (I tried a 130-slides one, including transitions, and it really didn’t work that well), you can approximate Powerpoint-like interactivity, while enjoying all the advantages of LaTeX typesetting.

Bottom line: I need to experiment more, but the results so far are intriguing. Incidentally, I tried Flashmode on my office iMac (3 GHz, early 2008 vintage) and my Macbook Pro (2.5 GHz, also early 2008 vintage). Not exactly the fastest machines you can lay your hands on. So, your mileage may be even better than mine.

Behind the scenes

Based on Flashmode’s manual, the software works as follows. An Applescript (Apple’s own scripting language) sits in the background, polling TeXshop to see if the topmost document window is "dirty", i.e. contains unsaved text. If so, the script asks TeXshop to save the file, then typesets the document and refreshes the output window.

According to the manual, this is all Flashmode does. However, there are two, maybe three well-known behind-the-scenes tricks to speed up IP. Flashmode perhaps uses the first one, but I’m not sure. In a way, I hope it doesn’t, because this would imply that there is scope for further speed-up! Anyway, here is a brief description of the tricks. [Update: Claus Gerhardt just confirmed to me that Flashmode does not use any such trick.]

The format-file trick. Remember that LaTeX is just a set of macros written in the more basic TeX language. When you compile a LaTeX source file, the tex or pdftex command-line program first loads the LaTeX "format file", a pre-compiled version of the LaTeX macros. More often than not, before you actually start typing any text, your file has a series of \usepackage commands (e.g. for bibliography, theorems, AMS fonts, etc.) that provide additional macros; finally, you may define your own macros in the TeX file. Every time you compile, all those packages, not to mention your own macros, have to be read and compiled. The idea is then to create a new format file that includes all of LaTeX, plus the specific packages and macros that you include in your own tex source file. This needs to be done only once, using the initex command-line utility; then, instead of compiling your original file with, say, pdflatex, you strip the preamble (the initial part of the file containing package invocations and macro definitions) and compile only the actual text with pdftex, specifying the pre-generated format file as additional compiled input. With complex macro packages, this can make a big difference. TextMate’s Watch document command in the LaTeX bundle uses this trick.

Slicing. This trick involves some complex coordination between the editor and LaTeX itself; Whizzytex is a particularly fast implementation. The idea is that, if one can figure out which output page contains the text currently being edited, one need only recompile that particular page ("slice"), not the entire document. Obviously, this can make a huge difference in terms of speed. In particular, longer documents do not slow down the interaction, because in any event only the current page is recompiled. I suspect this would be fantastic in conjunction with Beamer. Whizzytex keeps track of the current page by using a specially-crafted LaTeX style file, but I wonder whether a similar trick could be implemented using the built-in synctex facilities.

Low-level interaction with tex. This is a bit vague, so let me explain. Whenever you run any of the tex programs, the operating system has to load them into memory (which is pretty fast nowadays, at least on the second and subsequent runs); then, the software has to go through a sequence of initialization operations. Only after such operations have been completed can processing of actual source files begin. Hence the idea, due to Jonathan Fine, to essentially keep tex running as a background process, or daemon, and feed it source files as needed, without actually going through the initialization sequence each time. Obviously, this requires tinkering with TeX’s source code. It’s way above my pay grade :-)

As a final consideration, it is worth pointing out that all these techniques were originally developed to handle DVI output. But DVI is a much simpler format, so actually displaying a DVI file is considerably (as in, an order of magnitude) faster than displaying a PDF file. Thus, in principle, one could think of using DVI as the output format of choice for IP. However, this is not satisfactory for two reasons: first, there is no native DVI previewer on the Mac (Skim turns the DVI into a PDF); second, Beamer and the TikZ graphics package do not fully support DVI output (if at all).

Final Considerations

Right now, I think that, after this experiment, I will likely go back to my usual TextMate-based, traditional workflow. Incidentally, the Watch Document command in the LaTeX bundle implements 50% of IP: it regenerates the PDF output each time the source file is saved.

However, I wonder whether something like Flashcode could be implemented for TextMate. This largely depends upon the commands exposed via Applescript; I need to investigate! On the other hand, I am pretty sure that Flashcode could be implemented in Sublime Text, without even touching Applescript, and hence in a fully portable way. The Events and Idle-Watcher examples in the documentation for Sublime Text plugins are tantalizing…