Tag Archives: flashmode

Instant Preview: Some Raw Data

Attached file: latexTest-line.py. Remove .doc extension after downloading!

This is a follow-up to my earlier post on Instant Preview (IP) with LaTeX. After thinking about different IP solutions, I decided to run a little experiment. How can I measure the “instantness” of IP on the Mac, when compiling to either DVI or PDF output?

My first attempt involved a combination of a text editor, a script, and suitable DVI and PDF previewers. As text editor, I used TextWrangler, because it can be easily controlled via AppleScript, the Mac OS X native scripting language (actually, I should say "scripting infrastructure," as will be clear momentarily). In case you were wondering: yes, I tried using my beloved TextMate, and no, it could not be made to work because it does not expose an AppleScript interface. I then wrote a simple AppleScript that routinely asked TextWrangler if new text had been entered in the main document window. If so, the script asked TextWrangler to save the file, then launched the latex compiler to generate either a DVI or a PDF file, and finally asked the viewer app to refresh the document being displayed. In other words, I implemented a (very) poor man’s version of Flashmode.

As PDF viewer, the best choice for this experiment turned out to be TeXshop; it is scriptable and fast. Skim is also workable, but seems a bit slower, and requires setting "unsupported" defaults in order to refresh the current page in the background, without either asking for user confirmation or displaying an annoying (and time-consuming) progress bar. Finally, Apple’s on Preview app is not scriptable.

I was worried about DVI files. There is no native DVI previewer on the Mac, except for the venerable (and slow, not to mention for-pay) MacDviX. Skim will display DVI files, but it actually converts them to PDF files, which is obviously a problem if the whole point of the exercise is to measure DVI vs. PDF rendering speed. However, it turns out that Texlive, the TeX distribution upon which MacTeX is based, ships with the ancient but trusty xdvi. While xdvi runs under the X Windows system, and is a bit offensive to modern aesthetic sensibilities, it gets the job done–fast. Furthermore, getting xdvi to reload the current file can be achieved by sending it the SIGUSR1 Unix signal.

Long story short, my little AppleScript somehow worked, and clearly indicated that DVI is perceptibly faster than PDF as an output format for IP. While both xdvi and TeXshop lagged behind my typing, TeXshop lagged more (pretty much the same as with Flashmode, which is of course what one should expect). The question was how to quantify this difference. After discarding more "creative" solutions (I seriously considered filming myself while typing to a beat set by metronome!) I figured out that the best way to get some hard numbers was to simulate typing via a script. Now, AppleScript is Apple’s native scripting solution, but it just isn’t as powerful as, say, Python for anything even slightly more elaborate that sending a few method calls to a running application. So, Python it was.

One minor issue I had to deal with was how to control TeXshop from Python. The naive, slow way is to use os.system() or subprocess.Popen to launch the osascript command-line utility, passing the appropriate AppleScript instructions as argument. The smarter, much faster way is to use py-appscript. The only catch is that, to install the latter, you must have the GCC compiler installed, and that comes with the whole XCode environment, which is fairly big. Anyway, if you do have a development environment up and running, installing py-appscript is a simple matter. Then, you just import appscript in your Python code, and can send AppleScript instructions using nicely Pythonic syntax. And, again, things are quite a bit faster. As far as controlling xdvi is concerned, sending Unix signals from Python is a simple matter.

The Python script I used is attached to this post, in case you are interested (please remove the .doc extension: it is needed to keep WordPress happy). As described below, I ran a series of tests, each requiring slight modifications to the code; see the comments in the file for details. The basic idea is simple: the Python script "types" a line of text in the "middle" of a file, one character at a time; more precisely, for each integer n it creates a file containing

  1. a LaTeX preamble, the title and author of the "paper," and an initial fixed line of text;
  2. the first n characters of the line being "typed"; and
  3. in some tests, additional "body text," ranging from a few paragraphs to about 21 pages.

Each time a new file is created, the Python file compiles it to either DVI or PDF, then refreshes the viewer–just like my earlier Applescript did, except that, again, typing is simulated. The advantage is that I can easily time the execution of the script, using Python’s time module.

For each output format, I measured the time required to compile, but not display, the "typed" line (from the first to the last character), and then the time required to compile and display it. Furthermore, I experimented with different amounts of text after the "typed" line, so as to simulate IP in a fragment of text vs. in a medium-sized file. The tests were conducted on my Early 2008 iMac (a 3.06 GHz machine, codenamed iMac8,1 according to Apple’s convention). All measurements are taken from a single run of the script; I repeated the experiment several times, but found no significant difference. Without further ado, the results are shown in the following table.

Instant Preview: DVI vs PDF

To interpret the picture, “dvi/pdf, no body” means that there is no further text after the simulated “typed-in” line; “dvi/pdf, 1x body” means that there are a few paragraphs of subsequent text, consisting of a theorem declaration, a displayed equation, and some gibberish text; finally, “dvi/pdf, full body” means that the paragraphs just described are copied 50 times, so that the document is 22 pages long overall.

The results: I think there are four clear conclusions to be drawn from this little experiment. First, DVI files are measurably faster to render than PDF files; the time to display DVI output is negligible relative to the time it takes to actually generate it from LaTeX input. The same is certainly not true for PDF output. Now, this may in part be due to the fact that sending a SIGUSR1 signal to xdvi is faster than going through the Applescript infrastructure; I doubt that this is the most significant source of discrepancy, but in any event there is no other way to control TeXshop as far as I know (and for the time being).

Second, generating PDF files also takes longer. This exacerbates the differences in rendering times. For instance, without any additional body text, just generating the DVI files corresponding to the simulated typed line takes 11.26 seconds; when the output format is set to PDF, 13.98 seconds are needed—that is, 24% longer. If we include rendering, the time goes from 11.44 seconds to 15.13 seconds–an increase of 32%. If we include only a single copy of the body text, then just compiling takes 11.27 seconds for DVI and 15.18 for PDF, i.e. 35% longer; if we include rendering, the total time required is 11.65 seconds for DVI and 18.05 for PDF, i.e. 55% longer!

Third, when working with a medium-sized or large file, recompiling only the current page makes a difference. This can be seen by comparing the results for the "1x body" and "full body" experiments.

Fourth, and finally, PDF display is too slow to keep up with regular typing speed (as opposed to "hunt and peck" typing). To get an idea, consider the no-body test; the simulated typed-in line had 77 characters, and it took 15.13 seconds to "type" it all (i.e. compile and render the corresponding sequence of files). This translates to 77/15.13=5.1 characters per second. With 1x body text, "typing speed" goes down to 77/18.05=4.26 characters per second. You almost surely type faster than that. Try this test for instance (it’s fun!). I am no touch typist, but I got 7.48 characters per minute. If you are like me, or faster (which is likely), a naive implementation of IP that does what my simple Python script does on a machine no faster than my 3.06GHz iMac will lag behind you as you type. It won’t be quite so "instant." This is exactly my experience with Flashmode. In the case of DVI output, with 1x body text the Python script produces 77/11.65 = 6.6 characters per minute, which is not as fast as I type (according to the above-linked test) but not far behind either.

This test is only a starting point; it suggests where one can try to optimize processing. As already noted, focusing on the current page, rather than recompiling the entire file, makes a difference. But my Python script can be modified to experiment, for instance, with the format file trick described in my previous post. Furthermore, to output PDF files it is necessary to include, or embed, all required fonts; the pdftex manual suggests that this may be avoided, if the fonts are available system-wide. Perhaps that, too, saves time. In any case, more experimentation, followed by more serious hacking, is required.

LaTeX and Instant Preview

Despite my love for LaTeX typesetting, even I am forced to admit that the edit-compile-preview cycle lacks the immediacy of other inferior but WYSIWYG solutions. This is not always a problem, but it sometimes is. "Instant Preview" (IP for short) is a potential (if not yet fully realized), almost complete, best-of-both-worlds solution.

What is Instant Preview?

A picture is worth a thousand words; but, in this case, a Flash animation is worth even more. Take a look at this page, which demonstrates IP as implemented in a TeX distribution for Windows called BaKoMa TeX (I hope I got the capitalization right). Basically, you enter text in an edit window, and the output file is automatically regenerated and redisplayed on each keystroke, thereby giving the illusion of actually working "in the PDF file" (or, for BaKoMa TeX, in the DVI file, TeX’s older output format). IP was most famously implemented in the TeXtures TeX system for traditional Mac OS; Whizzytex is a well-known Emacs-based solution that achieves the same effect.

On modern Mac OS X there are two solutions (that I know of). One is the brand-new Latexian text editor, which features a built-in IP mode, but is fairly slow. The other is Flashmode, a solution by Claus Gerhardt based on the well-known TeXshop integrated editor and previewer. I decided to experiment with Flashmode this week.

Flashmode: installation and use

First of all, if you have the MacTeX LaTeX distribution, you most likely already have TeXshop. Then, you need to obtain Flashmode itself from Claus Gerhardt’s page; download the TeXhelpers package, which also contains other goodies. The Flashmode software comes with an exhaustive manual; however, let me point out a few caveats that caused me minor headaches and wasted time.

  1. After opening the TeXhelpers DMG (disk image) file, you must copy both the Flashmode and the Local Switcher apps to your Applications folder; you will need them both
  2. For some reason, neither Flashmode nor Local Switcher are indexed by Spotlight. So, if you are like me and launch most software (that you don’t already have on your Dock) using Spotlight, you’re out of luck: you need to open the Applications folder, or install scripts that launch Flashmode from inside TeXshop. Of course, you can also put Flashmode in your dock.
  3. You must create links to your current TeX distribution in your home directory: this is what Local Switcher is for. Fortunately, this is to be done only when you first install Flashmode, and every time you update or change your TeX distro. Note: the Flashmode manual states that you no longer need to create these symlinks, but my experience is that you do need them. [Update:Claus has just released a new version, numbered 6.1, that addresses this issue.]
  4. Finally, every TeX file you work on must have a format directive as its first line: that is, the very first line must be something like %&pdflatex, by itself (if the file is to be compiled using pdflatex; otherwise, e.g. %&pdftex or %&xelatex).

After installation, Flashmode is relatively painless to use. You open a tex file in TeXshop, perhaps typeset it once, then launch Flashmode and press the "Run Flashmode" button while the tex source file is the topmost window in TeXshop, right below the Flashmode window. Once again, make sure that the very first line of your file contains the text %pdflatex, by itself. Then you basically forget about it, and just watch as typeset PDF text automagically materializes in your preview window as you type in the source window. If there is a TeX/LaTeX error, the preview window is simply not updated. When you are done editing your file, switch to the Flashmode window and click the "Quit Flashmode" button. Read the manual for further information.

Now, before you get your hopes up: this thing is not really real-time preview. Working on a small- to medium-sized file, the PDF output gets updated maybe once every 1 or 2 seconds. However, after using Flashmode for a while, a strange thing happens: the source window sort of "fades in the background", and you tend to focus more on the output window. I mean this in a new-agey, psychological sense, not literally of course: you sort of stop paying attention exclusively to the TeX source, and rely a lot more on the PDF output.

I realized that this was happening because I did not find TeXshop’s editor frustrating, which I usually do (addicted as I am to TextMate). In particular, syntax highlighting is really minimal in TeXshop, but in the end it does not matter that much, because you can see the actual typeset output materialize before your eyes, without user intervention.

I think the killer application for Flashmode are Beamer presentations. As long as presentations are short (I tried a 130-slides one, including transitions, and it really didn’t work that well), you can approximate Powerpoint-like interactivity, while enjoying all the advantages of LaTeX typesetting.

Bottom line: I need to experiment more, but the results so far are intriguing. Incidentally, I tried Flashmode on my office iMac (3 GHz, early 2008 vintage) and my Macbook Pro (2.5 GHz, also early 2008 vintage). Not exactly the fastest machines you can lay your hands on. So, your mileage may be even better than mine.

Behind the scenes

Based on Flashmode’s manual, the software works as follows. An Applescript (Apple’s own scripting language) sits in the background, polling TeXshop to see if the topmost document window is "dirty", i.e. contains unsaved text. If so, the script asks TeXshop to save the file, then typesets the document and refreshes the output window.

According to the manual, this is all Flashmode does. However, there are two, maybe three well-known behind-the-scenes tricks to speed up IP. Flashmode perhaps uses the first one, but I’m not sure. In a way, I hope it doesn’t, because this would imply that there is scope for further speed-up! Anyway, here is a brief description of the tricks. [Update: Claus Gerhardt just confirmed to me that Flashmode does not use any such trick.]

The format-file trick. Remember that LaTeX is just a set of macros written in the more basic TeX language. When you compile a LaTeX source file, the tex or pdftex command-line program first loads the LaTeX "format file", a pre-compiled version of the LaTeX macros. More often than not, before you actually start typing any text, your file has a series of \usepackage commands (e.g. for bibliography, theorems, AMS fonts, etc.) that provide additional macros; finally, you may define your own macros in the TeX file. Every time you compile, all those packages, not to mention your own macros, have to be read and compiled. The idea is then to create a new format file that includes all of LaTeX, plus the specific packages and macros that you include in your own tex source file. This needs to be done only once, using the initex command-line utility; then, instead of compiling your original file with, say, pdflatex, you strip the preamble (the initial part of the file containing package invocations and macro definitions) and compile only the actual text with pdftex, specifying the pre-generated format file as additional compiled input. With complex macro packages, this can make a big difference. TextMate’s Watch document command in the LaTeX bundle uses this trick.

Slicing. This trick involves some complex coordination between the editor and LaTeX itself; Whizzytex is a particularly fast implementation. The idea is that, if one can figure out which output page contains the text currently being edited, one need only recompile that particular page ("slice"), not the entire document. Obviously, this can make a huge difference in terms of speed. In particular, longer documents do not slow down the interaction, because in any event only the current page is recompiled. I suspect this would be fantastic in conjunction with Beamer. Whizzytex keeps track of the current page by using a specially-crafted LaTeX style file, but I wonder whether a similar trick could be implemented using the built-in synctex facilities.

Low-level interaction with tex. This is a bit vague, so let me explain. Whenever you run any of the tex programs, the operating system has to load them into memory (which is pretty fast nowadays, at least on the second and subsequent runs); then, the software has to go through a sequence of initialization operations. Only after such operations have been completed can processing of actual source files begin. Hence the idea, due to Jonathan Fine, to essentially keep tex running as a background process, or daemon, and feed it source files as needed, without actually going through the initialization sequence each time. Obviously, this requires tinkering with TeX’s source code. It’s way above my pay grade :-)

As a final consideration, it is worth pointing out that all these techniques were originally developed to handle DVI output. But DVI is a much simpler format, so actually displaying a DVI file is considerably (as in, an order of magnitude) faster than displaying a PDF file. Thus, in principle, one could think of using DVI as the output format of choice for IP. However, this is not satisfactory for two reasons: first, there is no native DVI previewer on the Mac (Skim turns the DVI into a PDF); second, Beamer and the TikZ graphics package do not fully support DVI output (if at all).

Final Considerations

Right now, I think that, after this experiment, I will likely go back to my usual TextMate-based, traditional workflow. Incidentally, the Watch Document command in the LaTeX bundle implements 50% of IP: it regenerates the PDF output each time the source file is saved.

However, I wonder whether something like Flashcode could be implemented for TextMate. This largely depends upon the commands exposed via Applescript; I need to investigate! On the other hand, I am pretty sure that Flashcode could be implemented in Sublime Text, without even touching Applescript, and hence in a fully portable way. The Events and Idle-Watcher examples in the documentation for Sublime Text plugins are tantalizing…