Tag Archives: latex

LaTeX plugin: build paths are in!

A long-standing stumbling block for people getting started with ST2 on the Mac was the need to specify the TeX binaries path in a somewhat esoteric (and hidden) OSX plist file.

Well, no more! To make a long story short, my hard disk crashed, so I had to reinstall and reconfigure the OS and software on my Macbook Pro. In particular, my now-virgin system no longer has the .MacOS/environment.plist file correctly set up. I decided to use this opportunity to fix things the right way.

Now, the LaTeX.sublime-build file recognizes a new option, “path”. On the Mac, it is preconfigured so it works with any recent default install of MacTex. You can change it if necessary, but most likely you won’t need to: the build system should now just work.

On Windows, recent versions of MikTex do the right thing and set up the path so no special setting is needed, but the option is available in any case to handle non-standard configurations. (Note: you still need to manually add the Sumatra directory to your path; I plan to fix this next.)

As usual, please pull from GitHub, test, and report any breakage.

LaTeX plugin: Terminating a lengthy build process.

I just pushed a few patches to GitHub. If you get a fresh copy of the LaTeX plugin for Sublime Text 2, you will now be able to stop (Unix: “kill”; Windows: “terminate”) a lengthy build process simply by invoking the build command while a build is ongoing.

To clarify: say you start a build, e.g. by hitting CMD-B on the Mac. Suppose that you are compiling a Beamer presentation, which typically takes a while. Suddenly, you realize that there is a nasty typo in a slide. Instead of waiting for the build to end, you can now issue the build command again (e.g. by hitting CMD-B on the Mac): this will stop the currently running build, and print a message to this effect in the output panel. You can then fix the typo, and launch a new build as usual.

Caveats: this is my first stab at build killing. The required code (in makePDF.py) definitely hits the ceiling of my Python threading abilities. So far, it seems to be working (I’ve been using it myself all day, as I prepare for two upcoming conferences), but please test it yourself and let me know how things are.

Second, I have only tested this on the Mac. The Python code I use is fully cross-platform, but you never know. If you are on Windows, I’m even more curious to know if things work for you.

Final note to self and to fellow Python coders: I initially had quite a bit of trouble, because I used to execute the actual TeX commands via Popen.communicate. Trying to use Popen.kill() on a Popen object did not really work, in the sense that considerable time elapsed between the time the kill() method was called and the time the process actually exited. I am now using Popen.wait() to execute the TeX command, and the process is terminated almost instantaneously. There seems to be some funny business having to do with capturing output going to STDOUT in Popen.communicate(); using Popen.wait() avoids this (although you then have to send output to a temp file, if you need to capture it–which I don’t at the moment).

SumatraPDF 1.6 vs. 1.4

A quick post for Windows users of the LaTeX plugin. I just updated Sumatra PDF from version 1.4 to version 1.6. Unfortunately, forward search stopped working 😦 I checked, and this (mis)behavior has been present since version 1.5, which I had skipped. (Inverse search works fine, BTW).

I reported the issue to the developers; in the meantime, please use version 1.4, which works fine. You can get it from here (equivalently, navigate to the Download section of the SumatraPDF, and look for the link at the bottom of the page).

Edited (6/9/2011 at 11:45p): ends up there was a bug in my DDE code that Sumatra 1.4 forgave, but later versions did not. If you pull the most recent version from GitHub, you’ll be fine with Sumatra 1.6 as well. I am not sure I fully understand why the fix was needed, but I checked my code against this and figured out what I was doing wrong.

LaTeX Plugin: Build system activated

This post is a quick update on the revamped LaTeX build system.

If you pull the current master branch from Github, you will now be able to invoke the custom make_pdf command using the default build system: in particular, you can hit Command+B on Mac and F5 on Windows. The temporary ctrl+alt+t shortcut no longer works: on Mac, it does nothing, whereas on Windows it errors out (I’ll fix this soon). So, just don’t use it.

This is achieved by using the “target” option in the LaTeX.sublime-build file. By default (i.e. if no “target” option is provided), the build system invokes the “exec” command. However, you can instruct it to invoke any Python-written ST2 command; in my case, I set “target” to “make_pdf”, and ST2 automagically does the right thing.

In particular, any options in the sublime-build configuration file are passed as named arguments to the make_pdf command. Even better, you can set Windows- and OSX-specific options in the sublime-build file, and make_pdf will automagically receive the settings that are appropriate to your platform!

One final bit of awesomeness. If you open the LaTeX.sublime-build file, you will see that you can pass any configuration options to the latexmk (OSX) or texify (Windows) compiler drivers, IF YOU KNOW WHAT YOU ARE DOING. For instance, on Mac, you can easily change the tex engine from pdflatex to xelatex (it’s not so easy on Windows because texify is not as flexible as latexmk). Now, I was worried that any tweaks to the sublime-build file would be lost every time the user downloads a new version of the plugin. However (and this is the above-mentioned awesomeness), it turns out that you can save a copy of the LaTeX.sublime-build file in the Packages/User directory (rather than Packages/LaTeXTools), then tweak it to your heart’s content: when you invoke the build system, the file in Packages/User will take precedence, so your tweaks will still be there! Thanks jps 🙂

There are two items on my immediate TODO list. First, I want to add an option to the sublime-build file to specify the path of TeX executables (with sane defaults that should work on most systems). This will ensure that, with a minimum of configuration (or no configuration), tex and friends will work even if the PATH is not set (or, on the Mac, you have not created an .MacOSX/environment.plist file, as per this comment). This is the main cause of grief for new users; hopefully this will solve the issue once and for all.

Second, I would like to make the viewer configurable as well. Right now it is not: you must use Skim on the Mac and Sumatra on Windows. There is an issue as to where this setting should be; the sublime-build file does not seem appropriate, strictly speaking, but on the other hand it makes sense to keep all configuration settings in the same file.

One final note: the old sublime-settings file is still available, but I appended an “.OLD” extension to it so it’s not active. However, if you really prefer the old-style build system, you can still use it (remove or rename the new sublime-build file, then remove the “.OLD” extension to the old sublime-build file). In particular, this allows you to set your favorite PDF viewer (on the Mac), although you will lose inverse and forward search.

Comments, are usual, are very welcome!

LaTeX plugin: the new build engine is in!

I have decided to merge my ongoing work on a LaTeX-specific build engine. See my previous post for details on the new goodness. If you now pull the master branch from github, you’ll get the new engine. The relevant file is makePDF.py, but I had to change jumpToPDF.py as well.

A few notes. First, the quality of the code is, er, less than ideal. It works on all the (many!) test cases I have thrown at it, but it’s not pretty to look at. I’ll fix this in time. However, in the meantime, I’d like to get more people to try it out.

Second, the new build engine is bound to ctrl+alt+t (which is control+option+t on the Mac). The old, ST2-based build system is still available and bound to F5 on Windows and Command-B on the Mac (these are the ST2-standard bindings). Eventually, I plan to phase out the old build system, but not yet…

Third, due to the logic of log file parsing, you cannot use file names with parentheses. However, I am happy to report that spaces in file names (and directory names, too) are OK.

Fourth, there is still limited support for multi-file documents. Specifically, the log-file parsing logic is smart enough to pick up errors in different files and place the appropriate lines in the exec panel. However, compilation must be invoked from the master file (i.e. you must switch to the tab containing the master file before invoking compilation). The same is true of the old build system, by the way. Setting master files so you can compile from any file in the project is high on my agenda (together with certain other build and display customizations).

If you see any breakage, please let me know! In particular, if you see the line saying ‘Compiling ‘ in the exec panel, but nothing else happens after a while, hit ctrl+` to bring up the output panel; if you see any Python error messages, that’s breakage! Reports are greatly appreciated.

LaTeX plugin update: new build command, and parsing TeX log files

Once again, I haven’t posted anything in a while… I have been pretty busy at work lately: a couple of trips and PhD admissions took up most of my time. However, I have also made some progress with the Sublime Text 2 (ST2) LaTeX plugin. You can take a look at the current GitHub repository; there have been a few blog-worthy additions to the master branch, but the work I am most excited about has been in the make-pdf branch. Check that out (literally!) and look for yourself if you feel adventurous; however, note that things may be a bit broken, especially on Windows (my main work and dev platform is OSX). In particular, there is (as of today) no keybinding for the new make facility on Windows: take a look at Default (OSX).sublime-keymap to see how to add this on a PC. But, if you are not adventurous enough, read on 🙂

In the master branch, turning your tex file into a PDF file is achieved by using the ST2-standard build framework to invoke the latexmk command on OSX, and the texify command on Windows. In turn, these commands are instructed to launch a viewer. The output from these commands, including any error or warning messages, are sent by ST2 to the so-called "exec panel", i.e. a display area that pops up in the (roughly) bottom third or fourth of your ST2 window, and can be dismissed by pressing ESC if the user so wishes. Furthermore, ST2 provides a "goto next error" command (bound to F4 by default); to configure it, one must provide a regex (regular expression) that captures the file name and line number (and, optionally, column number) where the error occurred.

Using the standard build facility has the advantage of being more, er, Sublime–that is, in line with the philosophy, look, and feel of the ST2 editor. That’s a big plus. However, there are two important minuses. First, the regex approach for error detection works very well with languages, such as C or Java, that produce well-structured output and logs (where, by "well-structured," I mean "sane"). This does not include TeX–by a long shot. I provide some details below, as that’s the main focus of this post, and of my work in the make-pdf branch. The other minus is the fact that, while latexmk and texify can invoke a viewer, it is not possible to instruct said viewers to jump to the current line; the reason is that the build system is controlled by a static configuration file, and has no access to things like the current view (at least as far as I can tell, and as of build 2055). However, the ST2 developer has provided the ability to link the build command to any arbitrary Python script (or, more precisely, any ST2 "command"). This makes it possible to overcome any limitation of the default system. The make-pdf branch of the LaTeX plugin uses just this facility.

Specifically, I implemented a heavily customized version of the ST2 exec command, which is invoked whenever a standard build is initiated. The code is in makePDF.py, and the make_pdf command it defies is currently bound to ctrl+alt+t, the key binding for the old "texify" command in the version 1 plugin (again, this is actually only true for OSX as of today). The overall structure is relatively simple: the system-dependent command line is set up, then invoked in an instance of a worker thread called CmdThread, to avoid blocking. After the compilation command terminates, the log file is parsed (again in the thread), and the results are displayed. Since all output must happen on the main thread, the CmdThread thread issues display commands (basically, adding text to the exec panel) using set_timeout, using callbacks in the make_pdfCommand class. When this is done, the jump_to_pdf command is invoked, which causes the viewer to open the just-compiled PDF file (assuming all went well) and go to the line corresponding to the current cursor position. Notice that right now there is no error checking: the next order of business is to at least check for compilation errors before the viewer is invoked.

The piece de resistance of the make_pdf command, however, is the parseTeXlog function. It, well, parses the TeX log file. This presents five distinct difficulties:

  1. TeX errors are reported over several lines: the first (starting with an exclamation mark) states the type of error, then a few lines of surrounding text are reported, and finally the line where the error occurred is actually indicated. It is possible to force tex to use a file:line:error format like C or Java; however…
  2. …each macro package, beginning with LaTeX, reports warnings in a different format that cannot be forced into file-line-error mode. Thus, you need different regexes to match errors and warnings. I guess in principle one could try to use a single, humongous regex, but I am not sure one could cover all cases, and it would be a nightmare to write and maintain. I live by JWZ’s famous quote on regexes. In any case…
  3. …a regex that catches warnings would not be able to detect which file the warning occurred in—only the line and the warning message. This poses a problem for multi-file documents, which I really want to support (hello book co-authors!). This requires really understanding what the TeX log file is telling you at any given line. In particular, whenever TeX processes a new file (be it a source file, a style file, a class file, or anything else), this is reported on the log file; when processing of that particular file is complete, this, too is indicated. But, to make matters worse…
  4. …TeX "helpfully" breaks lines longer than 79 characters. Why is this a problem? Because certain style and macro files live deep in the texmf tree, so that their full path name easily exceeds 79 charactes. And because, if a user’s account name is RichardJamesDrofnatsTheThird, and he is working on a file called UnexplicableImplicationsOfUnrealisticAssumptionsInEconomicTheory.tex in a directory called CogitationsOfAnUnrepentantEconomist off of his Documents folder, well, that file name will be truncated (as far as I can tell, TeX always displays the full path name of the master file being compiled). And, finally…
  5. …there is no way to know whether a line was truncated or not. You can guess, and you can use heuristics, but there is no specific marker to tell you what TeX actually did with your log file. Fun!

So, here’s what I did. To make sense of this mess, one must start with the last problem: how to reassemble long lines. Again, there is no way to know for sure whether a 79-character line is the beginning of, say, a 90-character line, or if it is a line that just happens to be exactly 79 characters long (In case you were wondering, yes, I have found quite a few such lines in actual log files!). However, there are certain empirical regularities, which I exploit. The code in parseTeXlog joins a 79-character line with the line immediately following it, except under certain conditions. For example, many tex packages output an identification string of the form Package: geometry.sty... as soon as they are loaded. So, if a 79-character line is followed by a line that starts with Package:, I don’t join the two lines. There are other exceptions and heuristics: take a look at the file.

To be sure, I am not certain that I have covered all the cases. However, so far, every log file I have processed has been successfully parsed, and (trust me!) I have processed a lot of files. But, if you find something that breaks the parsing logic, I definitely want to know—thanks!

The next issue is to keep track of the file currently being processed. TeX reports it by printing a ( followed by the path name. Usually, the parenthesis is the first character in the line… but when a macro file loads another macro file, this may not be the case. So, you need logic for that. Whenever I see a new file being processed, I add it to a stack (the files list) and print its name (suitably indented) for debugging purposes (the user does not see this, unless she switches to the Python console).

When TeX is done processing a file, it just prints a closing parenthesis: ). This usually occurs on a line by itself, but for files imported by other packages this may not be the case. Special logic is needed to handle such cases. When I can be reasonably sure that a given file has been fully processed, I "pop" (i.e. remove) it from the stack.

The net result is that, when an error message is encountered, it must refer to the file that is currently at the top of the stack. What’s nice about this approach is that I do not need to know anything about the file being compiled: all the information I need is contained in the log file.

When all this is done, extracting error and warning information is actually easy. I then convert the information into file-line-error format, and tell ST2 to look for errors in that particular guise. This way, the user can hit F4 to navigate through errors. Or, the user can double-click on an error, and—regardless of the file it’s in—ST2 brings up the offending line. Pretty cool! This is actually better than TextMate’s error reporting, I think!

There is still a lot to do. First, as I noted above, I think that the current parsing logic is fairly robust, but it is still full of heuristics and hacks. If you find any problem or issue, please let me know! Second, I need to robustify the support for multi-file documents: there will be a mechanism to invoke compilation of the master file even if the user is editing an included file. Furthermore, there will be logic to only display the PDF output if compilation was successful.

More urgently though, I need to run some tests on Windows. When that is done, I will merge the make-pdf branch with the master branch and make this code the default build system. Comments welcome!

The in-progress LaTeX plugin for Sublime Text 2

EDITED 3/26: fixed URLs to reflect the new ‘official’ SubimeText packages repo / organization.

I have been pretty quiet the last couple of weeks. The main reason is that I was working hard to meet a deadline (which I did, sort of). However, I have also finally begun porting my LaTeX plugin for Sublime Text to the new, awesome, cross-platform v. 2.0 of said amazing text editor. Mind you: Sublime Text 2 (ST2 for short) is still in the alpha stage, and so is my ported plugin. On the other hand, I have been using ST2 as my sole text editor for the past two weeks, tweaking and porting the essential features of the LaTeX plugin as I needed them. It has served me well.

Thus, if you like living on the edge, you can get the ST2 alpha version here; or, if you are really adventurous, grab the latest dev build here. Then, head over to the LaTeXTools public GitHub repository; hit the “Downloads” button to get a .zip file of the current version; you will get a silly name like msiniscalchi-LaTeXTools-38ddb5f.zip (the final part of the file name will change: it’s part of the hash code of the latest commit I pushed to github). Expand this to a folder, which will be aptly called something like msiniscalchi-LaTeXTools-38ddb5f. Rename this folder to something sane, such as LaTeXTools but NOT just LaTeX (read on), and drop it in ST2’s Packages directory, which you can open from ST2 itself by selecting Preferences|Browse Packages (the exact location of these menu items varies by OS).

An important caveat: the new plugin is designed to coexist with, not replace, the existing snippets, language definitions, etc., that can be found in the LaTeX folder that comes with ST2 (in the Packages) directory. This is the reason why the plugin folder must be names something other than LaTeX. Also, of course you must not erase the existing LaTeX folder.

I am pretty excited about the plugin! It’s certainly not complete, and must currently work around limitations imposed by the fact that the ST2 alpha still lacks certain features that ST1 had (for instance, quick panels). On the other hand, ST2 brings a lot to the table, and the plugin attempts to use the built-in features of ST2 whenever possible; the best example is the use of the Completions API to handle references and citations, and of .sublime-completions files to handle quick LaTeX macro insertion. As a guiding principle, whereas the LaTeX plugin for ST1 tried to be very TextMatey, the LaTeX plugin for ST2 will be more… well, Sublime 🙂

I will blog about specific features of the plugin later. However, if you want to kick the tires, so to speak, here’s the bare minimum you need to know.

  • On Windows, the plugin uses MikTeX and the Sumatra PDF previewer, just like ST1. Both must be installed, and Sumatra must be on the PATH. Check your environment variables.
  • Recent versions of Sumatra allow you to set up inverse search from the GUI. You need to do so before you can use this feature. Go to Settings|Options, and under “Set inverse search command line”, enter sublime-text "%f":%l
  • Similarly, on the Mac, you use MacTeX and the Skim previewer. To set up inverse search, go to Preferences, select the Sync tab, choose Custom from the Preset drop-down menu, and enter subl in the Command field and "%file":%line in the Arguments field. Also make sure the Check for file changes box is not checked (same as for TextMate).
  • To compile a tex file, use the ST2 build system, which you invoke with F7 on Windows and Cmd-B on the Mac. The build will start, and if there are n errors the PDF viewer will be shown.
  • When you recompile (again, with the Build command), the PDF file will be automatically refreshed, but will stay in the background.
  • To use forward search (i.e. jump from tex to pdf), use shift+alt+j on Windows and shift+cmd+j on the Mac. For inverse search, double-click (Sumatra) or shift-cmd-click (Skim) at the relevant position in the pdf file.
  • For the time being, after compiling, the PDF file is either opened on page 1, or else refreshed; you do not automatically move to the page corresponding to the current position in the tex file. This is the same behavior as the ST1 LaTeX plugin, but it is different from TextMate’s behavior. I’ll fix this later.
  • Two PDF-viewer caveats: on Windows, if you try to use forward search but haven’t installed Sumatra, ST2 will crash. On the Mac, if you have Skim running without any windows open, and you try to compile a file or view the current PDF file (with ctrl+opt+v), nothing will happen. Close Skim and try again. Again, it’s something I need to fix.
  • Check the .sublime-keymap file for your platform (in the directory where the new LaTeX plugin lives) to figure out other key bindings, or change the existing ones.
  • A quick primer on the new features: references are inserted by typing ref and then hitting ctrl+space (the ST2-standard long completion key combo). This shows a drop-down menu with all possible references. But, if you know the reference you are looking starts with, say, `sec’, you can type ref_sec, then hit ctrl+space: now only references starting with `sec’ will be shown! Citations work similarly: e.g. type cite_myer and all citations with `myer’ either in the key or the title will come up. There are many more features (and a couple of caveats): you can check the comments in the latex-cite-completions.py and latex-ref-completions.py files to get an idea, but again, I’ll blog about them later.
  • In math mode, you have access to many convenient shortcuts (completions): e.g. type a, then hit TAB: you get \alpha. In text mode, you get other completions: e.g. bs expands to \bigskip. Note that, for instance, it expands to \mathit in math mode, and to \textit in text mode. Also, you can cycle among certain LaTeX commands. Check the LaTeX.sublime-completions and LaTeX math.sublime-completions for a human-readable list of available shortcuts.

That’s about it for now. Again, it’s all alpha quality, but I have been using the ST2+Plugin combination as my sole LaTeX editing `solution’ (hate that word…) for a couple of weeks, and it is serving me well so far. How’s that for dogfooding?

Instant Preview: Some Raw Data

Attached file: latexTest-line.py. Remove .doc extension after downloading!

This is a follow-up to my earlier post on Instant Preview (IP) with LaTeX. After thinking about different IP solutions, I decided to run a little experiment. How can I measure the “instantness” of IP on the Mac, when compiling to either DVI or PDF output?

My first attempt involved a combination of a text editor, a script, and suitable DVI and PDF previewers. As text editor, I used TextWrangler, because it can be easily controlled via AppleScript, the Mac OS X native scripting language (actually, I should say "scripting infrastructure," as will be clear momentarily). In case you were wondering: yes, I tried using my beloved TextMate, and no, it could not be made to work because it does not expose an AppleScript interface. I then wrote a simple AppleScript that routinely asked TextWrangler if new text had been entered in the main document window. If so, the script asked TextWrangler to save the file, then launched the latex compiler to generate either a DVI or a PDF file, and finally asked the viewer app to refresh the document being displayed. In other words, I implemented a (very) poor man’s version of Flashmode.

As PDF viewer, the best choice for this experiment turned out to be TeXshop; it is scriptable and fast. Skim is also workable, but seems a bit slower, and requires setting "unsupported" defaults in order to refresh the current page in the background, without either asking for user confirmation or displaying an annoying (and time-consuming) progress bar. Finally, Apple’s on Preview app is not scriptable.

I was worried about DVI files. There is no native DVI previewer on the Mac, except for the venerable (and slow, not to mention for-pay) MacDviX. Skim will display DVI files, but it actually converts them to PDF files, which is obviously a problem if the whole point of the exercise is to measure DVI vs. PDF rendering speed. However, it turns out that Texlive, the TeX distribution upon which MacTeX is based, ships with the ancient but trusty xdvi. While xdvi runs under the X Windows system, and is a bit offensive to modern aesthetic sensibilities, it gets the job done–fast. Furthermore, getting xdvi to reload the current file can be achieved by sending it the SIGUSR1 Unix signal.

Long story short, my little AppleScript somehow worked, and clearly indicated that DVI is perceptibly faster than PDF as an output format for IP. While both xdvi and TeXshop lagged behind my typing, TeXshop lagged more (pretty much the same as with Flashmode, which is of course what one should expect). The question was how to quantify this difference. After discarding more "creative" solutions (I seriously considered filming myself while typing to a beat set by metronome!) I figured out that the best way to get some hard numbers was to simulate typing via a script. Now, AppleScript is Apple’s native scripting solution, but it just isn’t as powerful as, say, Python for anything even slightly more elaborate that sending a few method calls to a running application. So, Python it was.

One minor issue I had to deal with was how to control TeXshop from Python. The naive, slow way is to use os.system() or subprocess.Popen to launch the osascript command-line utility, passing the appropriate AppleScript instructions as argument. The smarter, much faster way is to use py-appscript. The only catch is that, to install the latter, you must have the GCC compiler installed, and that comes with the whole XCode environment, which is fairly big. Anyway, if you do have a development environment up and running, installing py-appscript is a simple matter. Then, you just import appscript in your Python code, and can send AppleScript instructions using nicely Pythonic syntax. And, again, things are quite a bit faster. As far as controlling xdvi is concerned, sending Unix signals from Python is a simple matter.

The Python script I used is attached to this post, in case you are interested (please remove the .doc extension: it is needed to keep WordPress happy). As described below, I ran a series of tests, each requiring slight modifications to the code; see the comments in the file for details. The basic idea is simple: the Python script "types" a line of text in the "middle" of a file, one character at a time; more precisely, for each integer n it creates a file containing

  1. a LaTeX preamble, the title and author of the "paper," and an initial fixed line of text;
  2. the first n characters of the line being "typed"; and
  3. in some tests, additional "body text," ranging from a few paragraphs to about 21 pages.

Each time a new file is created, the Python file compiles it to either DVI or PDF, then refreshes the viewer–just like my earlier Applescript did, except that, again, typing is simulated. The advantage is that I can easily time the execution of the script, using Python’s time module.

For each output format, I measured the time required to compile, but not display, the "typed" line (from the first to the last character), and then the time required to compile and display it. Furthermore, I experimented with different amounts of text after the "typed" line, so as to simulate IP in a fragment of text vs. in a medium-sized file. The tests were conducted on my Early 2008 iMac (a 3.06 GHz machine, codenamed iMac8,1 according to Apple’s convention). All measurements are taken from a single run of the script; I repeated the experiment several times, but found no significant difference. Without further ado, the results are shown in the following table.

Instant Preview: DVI vs PDF

To interpret the picture, “dvi/pdf, no body” means that there is no further text after the simulated “typed-in” line; “dvi/pdf, 1x body” means that there are a few paragraphs of subsequent text, consisting of a theorem declaration, a displayed equation, and some gibberish text; finally, “dvi/pdf, full body” means that the paragraphs just described are copied 50 times, so that the document is 22 pages long overall.

The results: I think there are four clear conclusions to be drawn from this little experiment. First, DVI files are measurably faster to render than PDF files; the time to display DVI output is negligible relative to the time it takes to actually generate it from LaTeX input. The same is certainly not true for PDF output. Now, this may in part be due to the fact that sending a SIGUSR1 signal to xdvi is faster than going through the Applescript infrastructure; I doubt that this is the most significant source of discrepancy, but in any event there is no other way to control TeXshop as far as I know (and for the time being).

Second, generating PDF files also takes longer. This exacerbates the differences in rendering times. For instance, without any additional body text, just generating the DVI files corresponding to the simulated typed line takes 11.26 seconds; when the output format is set to PDF, 13.98 seconds are needed—that is, 24% longer. If we include rendering, the time goes from 11.44 seconds to 15.13 seconds–an increase of 32%. If we include only a single copy of the body text, then just compiling takes 11.27 seconds for DVI and 15.18 for PDF, i.e. 35% longer; if we include rendering, the total time required is 11.65 seconds for DVI and 18.05 for PDF, i.e. 55% longer!

Third, when working with a medium-sized or large file, recompiling only the current page makes a difference. This can be seen by comparing the results for the "1x body" and "full body" experiments.

Fourth, and finally, PDF display is too slow to keep up with regular typing speed (as opposed to "hunt and peck" typing). To get an idea, consider the no-body test; the simulated typed-in line had 77 characters, and it took 15.13 seconds to "type" it all (i.e. compile and render the corresponding sequence of files). This translates to 77/15.13=5.1 characters per second. With 1x body text, "typing speed" goes down to 77/18.05=4.26 characters per second. You almost surely type faster than that. Try this test for instance (it’s fun!). I am no touch typist, but I got 7.48 characters per minute. If you are like me, or faster (which is likely), a naive implementation of IP that does what my simple Python script does on a machine no faster than my 3.06GHz iMac will lag behind you as you type. It won’t be quite so "instant." This is exactly my experience with Flashmode. In the case of DVI output, with 1x body text the Python script produces 77/11.65 = 6.6 characters per minute, which is not as fast as I type (according to the above-linked test) but not far behind either.

This test is only a starting point; it suggests where one can try to optimize processing. As already noted, focusing on the current page, rather than recompiling the entire file, makes a difference. But my Python script can be modified to experiment, for instance, with the format file trick described in my previous post. Furthermore, to output PDF files it is necessary to include, or embed, all required fonts; the pdftex manual suggests that this may be avoided, if the fonts are available system-wide. Perhaps that, too, saves time. In any case, more experimentation, followed by more serious hacking, is required.

LaTeX and Instant Preview

Despite my love for LaTeX typesetting, even I am forced to admit that the edit-compile-preview cycle lacks the immediacy of other inferior but WYSIWYG solutions. This is not always a problem, but it sometimes is. "Instant Preview" (IP for short) is a potential (if not yet fully realized), almost complete, best-of-both-worlds solution.

What is Instant Preview?

A picture is worth a thousand words; but, in this case, a Flash animation is worth even more. Take a look at this page, which demonstrates IP as implemented in a TeX distribution for Windows called BaKoMa TeX (I hope I got the capitalization right). Basically, you enter text in an edit window, and the output file is automatically regenerated and redisplayed on each keystroke, thereby giving the illusion of actually working "in the PDF file" (or, for BaKoMa TeX, in the DVI file, TeX’s older output format). IP was most famously implemented in the TeXtures TeX system for traditional Mac OS; Whizzytex is a well-known Emacs-based solution that achieves the same effect.

On modern Mac OS X there are two solutions (that I know of). One is the brand-new Latexian text editor, which features a built-in IP mode, but is fairly slow. The other is Flashmode, a solution by Claus Gerhardt based on the well-known TeXshop integrated editor and previewer. I decided to experiment with Flashmode this week.

Flashmode: installation and use

First of all, if you have the MacTeX LaTeX distribution, you most likely already have TeXshop. Then, you need to obtain Flashmode itself from Claus Gerhardt’s page; download the TeXhelpers package, which also contains other goodies. The Flashmode software comes with an exhaustive manual; however, let me point out a few caveats that caused me minor headaches and wasted time.

  1. After opening the TeXhelpers DMG (disk image) file, you must copy both the Flashmode and the Local Switcher apps to your Applications folder; you will need them both
  2. For some reason, neither Flashmode nor Local Switcher are indexed by Spotlight. So, if you are like me and launch most software (that you don’t already have on your Dock) using Spotlight, you’re out of luck: you need to open the Applications folder, or install scripts that launch Flashmode from inside TeXshop. Of course, you can also put Flashmode in your dock.
  3. You must create links to your current TeX distribution in your home directory: this is what Local Switcher is for. Fortunately, this is to be done only when you first install Flashmode, and every time you update or change your TeX distro. Note: the Flashmode manual states that you no longer need to create these symlinks, but my experience is that you do need them. [Update:Claus has just released a new version, numbered 6.1, that addresses this issue.]
  4. Finally, every TeX file you work on must have a format directive as its first line: that is, the very first line must be something like %&pdflatex, by itself (if the file is to be compiled using pdflatex; otherwise, e.g. %&pdftex or %&xelatex).

After installation, Flashmode is relatively painless to use. You open a tex file in TeXshop, perhaps typeset it once, then launch Flashmode and press the "Run Flashmode" button while the tex source file is the topmost window in TeXshop, right below the Flashmode window. Once again, make sure that the very first line of your file contains the text %pdflatex, by itself. Then you basically forget about it, and just watch as typeset PDF text automagically materializes in your preview window as you type in the source window. If there is a TeX/LaTeX error, the preview window is simply not updated. When you are done editing your file, switch to the Flashmode window and click the "Quit Flashmode" button. Read the manual for further information.

Now, before you get your hopes up: this thing is not really real-time preview. Working on a small- to medium-sized file, the PDF output gets updated maybe once every 1 or 2 seconds. However, after using Flashmode for a while, a strange thing happens: the source window sort of "fades in the background", and you tend to focus more on the output window. I mean this in a new-agey, psychological sense, not literally of course: you sort of stop paying attention exclusively to the TeX source, and rely a lot more on the PDF output.

I realized that this was happening because I did not find TeXshop’s editor frustrating, which I usually do (addicted as I am to TextMate). In particular, syntax highlighting is really minimal in TeXshop, but in the end it does not matter that much, because you can see the actual typeset output materialize before your eyes, without user intervention.

I think the killer application for Flashmode are Beamer presentations. As long as presentations are short (I tried a 130-slides one, including transitions, and it really didn’t work that well), you can approximate Powerpoint-like interactivity, while enjoying all the advantages of LaTeX typesetting.

Bottom line: I need to experiment more, but the results so far are intriguing. Incidentally, I tried Flashmode on my office iMac (3 GHz, early 2008 vintage) and my Macbook Pro (2.5 GHz, also early 2008 vintage). Not exactly the fastest machines you can lay your hands on. So, your mileage may be even better than mine.

Behind the scenes

Based on Flashmode’s manual, the software works as follows. An Applescript (Apple’s own scripting language) sits in the background, polling TeXshop to see if the topmost document window is "dirty", i.e. contains unsaved text. If so, the script asks TeXshop to save the file, then typesets the document and refreshes the output window.

According to the manual, this is all Flashmode does. However, there are two, maybe three well-known behind-the-scenes tricks to speed up IP. Flashmode perhaps uses the first one, but I’m not sure. In a way, I hope it doesn’t, because this would imply that there is scope for further speed-up! Anyway, here is a brief description of the tricks. [Update: Claus Gerhardt just confirmed to me that Flashmode does not use any such trick.]

The format-file trick. Remember that LaTeX is just a set of macros written in the more basic TeX language. When you compile a LaTeX source file, the tex or pdftex command-line program first loads the LaTeX "format file", a pre-compiled version of the LaTeX macros. More often than not, before you actually start typing any text, your file has a series of \usepackage commands (e.g. for bibliography, theorems, AMS fonts, etc.) that provide additional macros; finally, you may define your own macros in the TeX file. Every time you compile, all those packages, not to mention your own macros, have to be read and compiled. The idea is then to create a new format file that includes all of LaTeX, plus the specific packages and macros that you include in your own tex source file. This needs to be done only once, using the initex command-line utility; then, instead of compiling your original file with, say, pdflatex, you strip the preamble (the initial part of the file containing package invocations and macro definitions) and compile only the actual text with pdftex, specifying the pre-generated format file as additional compiled input. With complex macro packages, this can make a big difference. TextMate’s Watch document command in the LaTeX bundle uses this trick.

Slicing. This trick involves some complex coordination between the editor and LaTeX itself; Whizzytex is a particularly fast implementation. The idea is that, if one can figure out which output page contains the text currently being edited, one need only recompile that particular page ("slice"), not the entire document. Obviously, this can make a huge difference in terms of speed. In particular, longer documents do not slow down the interaction, because in any event only the current page is recompiled. I suspect this would be fantastic in conjunction with Beamer. Whizzytex keeps track of the current page by using a specially-crafted LaTeX style file, but I wonder whether a similar trick could be implemented using the built-in synctex facilities.

Low-level interaction with tex. This is a bit vague, so let me explain. Whenever you run any of the tex programs, the operating system has to load them into memory (which is pretty fast nowadays, at least on the second and subsequent runs); then, the software has to go through a sequence of initialization operations. Only after such operations have been completed can processing of actual source files begin. Hence the idea, due to Jonathan Fine, to essentially keep tex running as a background process, or daemon, and feed it source files as needed, without actually going through the initialization sequence each time. Obviously, this requires tinkering with TeX’s source code. It’s way above my pay grade 🙂

As a final consideration, it is worth pointing out that all these techniques were originally developed to handle DVI output. But DVI is a much simpler format, so actually displaying a DVI file is considerably (as in, an order of magnitude) faster than displaying a PDF file. Thus, in principle, one could think of using DVI as the output format of choice for IP. However, this is not satisfactory for two reasons: first, there is no native DVI previewer on the Mac (Skim turns the DVI into a PDF); second, Beamer and the TikZ graphics package do not fully support DVI output (if at all).

Final Considerations

Right now, I think that, after this experiment, I will likely go back to my usual TextMate-based, traditional workflow. Incidentally, the Watch Document command in the LaTeX bundle implements 50% of IP: it regenerates the PDF output each time the source file is saved.

However, I wonder whether something like Flashcode could be implemented for TextMate. This largely depends upon the commands exposed via Applescript; I need to investigate! On the other hand, I am pretty sure that Flashcode could be implemented in Sublime Text, without even touching Applescript, and hence in a fully portable way. The Events and Idle-Watcher examples in the documentation for Sublime Text plugins are tantalizing…

A quick note on Sublime Text

Two, really. First, I realized that links to specific pages on the sublimetextwiki.com site no longer work. So, the correct URL for the Sublime Text LaTeX plugin is now this.

The aforelinked (pardon the Gruberism) Web page is automatically generated from the README.txt help file that every Sublime Text plugin must include. Again by convention, the latter file is marked up using, ahem, Markdown; a "style sheet" is then used to convert it to HTML code that can be rendered by Web browsers. Well, it appears that a different style sheet is now being employed. I am not too crazy about the choice of grey text on black background; but, more importantly, text in backquotes, e.g.

the word `this` is in backquotes

is now rendered on a line by itself, like this:

the word
  this
is in backquotes

Previously, it was rendered "in-line", as if using the code tag in HTML. I will have to fix up the help file so it does not look too bad; please bear with me…

…which brings me to the second note. I am seriously thinking about experimenting with the current alpha release of Sublime Text X, the next-generation version. Its main advantage is that it is cross-platform! Yay! This means that I will be able to work on the LaTeX plugin on my Mac, under OS X, without either rebooting or running Parallels.

If I do take the plunge, the first order of business will be to refactor the existing code so that all Windows-specific code (in particular, invoking texify and the Sumatra PDF previewer) is neatly isolated, and can be replaced by the corresponding Mac-specific calls and scripts. This would actually be an improvement over the current structure even beyond cross-platform portability; for instance, it would make it easy(ier) for me to add support for different PDF previewers, and perhaps even TeX distributions.

No promises as to the time frame though… if anyone wishes to help out, let me know!