Mike's Musings #2: Thoroughly Modern Mikey

By Mike McCarthy (9 December 2006)

In Mike's Musings #1 "FS2004 and FSX", I did such a good job of selling FS2004 to myself that I decided to go one step further and install my heretofore idle Level-D 767 and DreamFleet 727 add-ons. Both of these aircraft are available for purchase from Flight1, as is Ultimate Traffic, a utility I forgot to mention in the previous article. I can now recommend the two aircraft without qualification because it turns out that they run well even on my antique at-home 900 MHz development system. I haven't yet tried out the FS2004 reinstallation of Ultimate Traffic but I'm sure it will be fine. It's the next best thing to flying online, the subject of an upcoming article.

Of course I had to cut over to Windows XP to be able to run the aircraft products, so now everything at home is au courant except for the hardware. Actually, we have three computers at home, one for my wife and two for me. Hers is a fast one with DVD but no-o-o-o-o, she won't let me install FSX on it. Anyway, at 800 MHz, my second machine is even slower than the 900 MHz development system. It has been running various versions of XP since early beta test days so in that sense I'm not a newcomer to XP at all. This machine is going to become my testbed, to be in a constant state of churn as I continually rebuild it with various versions of Windows and various versions of Flight Simulator.

But that's enough shameless self-promotion for today. Almost all of you are running FS2004, and you have been doing so for quite some time, so there isn't anything further I need to say about it other than that I am well pleased with it. I will continue to test on FS2002 but I now have no plans to fly it for pleasure anymore.

It is Sunday, 4 December and I am at my desk in the office. In Musings #1 I speculated that the reason I was getting stuttering performance from FSX on my office 2 GHz machine was, perhaps, because I had not optimized the hard drive correctly. Well, as I write this, Ultimate Defrag is running in the background, with a new set of parameters to guide the defragmentation process.

Folks, you must get Ultimate Defrag, though it's not the only thing you should get. Available at www.disktrix.com for $40 US, it offers complete control over how your hard drive should be laid out. Using my new set of parameters I have told Ultimate Defrag to give priority attention to the FSX Addon Scenery, Autogen, Scenery and Texture folders. Along with the System Volume Information file (which captures file add/delete/modify activity for future use by System Restore), these will be moved to the outer tracks of the hard drive, with attention being paid to striking a compromise such that these folders, and the sub-indexes and files they contain, will also be kept reasonably close to XP's Master File Table. (The MFT, the multi-level index into the system's file structures.)

To try to eliminate FS stutters, our performance goal must be to minimize disk drive read/write head motion. For example, and oversimplifying a bit, when FS needs a scenery file it will look first in its scenery cache. The software logic of the XP operating system is such that FS does not need to know whether the (cached) file is located in main memory, or in the PageFile.sys swap file, or out in the main file structures. In contrast XP itself does need to know, and that is the purpose of the MFT, to tell the XP file system manager where folders and files are physically located on the hard drive.

The worst case, and this is a common case for dynamically-changing scenery, is that the requested scenery file is not in the scenery cache and is not even in PageFile.sys. In that case XP must look up the file in the MFT. The MFT is organized as a B* Tree, a special kind of multilevel indexed file retrieval mechanism. Without going into detail, finding the file may involve accessing many conceptual layers of the MFT tree structure. If the MFT is badly fragmented (and it becomes so over time), there is the potential for many hard drive read/write head positioner strokes as the file system manager works its way through the MFT to the actual entry for the file in question.

So ... If we have a ten-level MFT search to do, the total time will be 10*40 = 400 ms spent in the MFT, plus 40 ms accessing the file, for a grand total of 440 ms, easily enough to produce a major stutter in the FS scenery playback. That's with a badly fragmented MFT. However, if we fully defragment the MFT, on a typical 100 GB hard drive it probably will reside on one or at most two tracks. Our total search time then becomes roughly 20 ms to get to the MFT, roughly ten disk revolutions = 10*20 = 200 ms to perform the multilevel search within the MFT, and a final 20 ms to get to the file. Now our search time totals 20+200+20 = 240 ms, a bit more than half the previous figure of 440 ms.

This is a considerable improvement, and the more levels there are to the average MFT search, which increases with the number of files on the hard drive, the more important is defragmentation of the MFT. Over-simplifying again, because of the software logic built into the XP file system manager, chunks of a contiguous MFT will be brought into main memory all at once, "for free", speeding up searches even more.

Now ... FS maintains its own set of indexes into its scenery database. In the case of a full install of FSX, the scenery database is enormous so the index file suite also must be enormous. The need for FSX to stream the index files while a scenario is loading is the reason it is slow to get going. But this has nothing to do with scenery caching or scenery lookahead.

Defragmentation is a complex but very interesting subject if, like me, you're a computer nerd. If you want more information, read this article about defragmentation strategies. However, this article is incorrect when it says that MFT entries are never reused. They are, and a Microsoft Knowledge Base article, says so. The Microsoft article doesn't say much about XP's own MFT optimizations, but another one here does. (Look about 40% of the way down for a discussion of what happens every three days or so.)

If all you care about is PageFile.sys, there is a freeware utility for defragging it. The program is PageDefrag, available at www.sysinternals.com. It also treats the registry "hives", another source of system slowdown if left untreated. An alternative to PageDefrag, is a strategy that I've used in the past: Turn off virtual memory (which deletes PageFile.sys), then defrag the drive using the standard Windows defragmenter (perhaps several times), and then reinstate virtual memory. This will cause PageFile.sys to be allocated anew in the free space region which, hopefully, will be sufficiently contiguous to cause the new PageFile.sys also to be contiguous.

I forgot to mention that to make all this PageFile.sys stuff work you need to do Start, Settings, Control Panel, System, Advanced, Performance, Advanced, and then set your paging file to a constant number not managed by the operating system. I recommend 4095 MB, the largest possible value, because that way PageFile.sys will, after you defrag it, act like a file cache in its own right.

Ultimate Defrag has a checkbox for enabling boot-time defragmentation of the critical system files. So does O&O Defrag, available for $50 US at www.oo-software.com. The latter is the program which Andrew Herd uses. I tested the demo version of O&OD (a drug-crazed rock band?) and it did in fact Do The Right Thing, reducing my 12-fragment MFT to a single contiguous file, which Ultimate Defrag was not able to do. So, based on the on the hard evidence that there's something fatally wrong with UD's boot-time defrag logic, here's how to use the pair of tools most effectively, assuming that you're willing to spend the $90 US required to buy both of them.

First, use Ultimate Defrag to perform a "consolidation" of your file system. This will gather all the free space into one adjacent set of hard drive tracks, providing plenty of space in which O&OD can maneuver to perform its MFT and PageFile.sys boot-time defragmentation miracles. Doing the consolidation run could take a few hours so be patient.

Second, tell O&OD that you want to do an offline defrag. (File, Options, Offline Defrag, Execute Once, highlight the drive choice, do Activate, and then reboot.) Be patient again. Depending on the state of your file system this step may take 10-20 minutes, perhaps even longer.

Once the offline defrag finishes, fire up UD again, hit the "analyze" button, and when analysis is 100% complete, right-click on the drive icon in the upper left corner of the screen. Scroll down to Volume Info and then click on the Locked tab. You will then see what O&OD was able to accomplish during its boot-time defrag run. In a perfect world you will see nothing but ones in the Fragments column. Starting with a UD consolidation run will make your world perfect.

Third, in Ultimate Defrag you should then do Tools, Options, High Performance, Custom, Select Files. Once there you should select which files/folders you care about as regards performance. (See my recommendations above.) The program will remember your settings from one run to the next. Having made your selections you should then do not a consolidation run but rather a "recency" defrag. This will not only speed-optimize the files you selected, it will also do a general overall drive optimization based on how recently the various files and folders have been accessed. The least recently used material will be shunted off to the low performance inner tracks of the drive, making the high-performance outer tracks available for the folders and files of interest.

...It didn't ... But O&OD did ...

... And now I'm at home, where I repeated the experiments, on my development system, with comparable results. There is no question about it, folks: Given enough free space in which to maneuver, O&OD does the off-line defrag job perfectly. However, it lacks the precision folder/file placement controls which Ultimate Defrag offers. It is not clear which single approach is best so I'm going to squander my inheritance and buy them both, using them in the three-step procedure outlined above.

I must mention, however, that Zone Alarm, www.zonelabs.com, reports that O&OD tries to act as a server. There is no legitimate reason for O&OD to do this, and it potentially opens a pathway for Trojan horses, so I always forbid it. Incidentally, there are both freeware and payware versions of Zone Alarm. Even the freeware version would have intercepted this suspicious activity. The payware version, $50 US, is able to monitor the contents of outgoing TCP/IP packets for things like your social security number, credit card numbers, and so on. I strongly recommend it, but you must make sure to turn ZoneAlarm off before running FS. Otherwise ZA will intercept each and every file access, bogging the FS session.

Anyway, such is life writing for FSC. Not only don't we regulars get paid, I have to buy stuff myself. I hope that publishers will come around to the idea of sending me utility programs for free. (I'll cede the add-ons review field to the capable and humorous Andrew Herd.) I'd be happy to review such contributions, just as I'm reviewing UD and O&OD here. But make the programs keepers, not demos, oui? [And I do get paid after all, in the form of no-cost self-promotion. Thanks, Nels.]

Now for the payoff: What effect on FSX performance did all these defrag operations have? I have bad news and good news.

The bad news is that time-to-home-screen increased from 40 seconds to 50. Part a) of the good news is that the scenario load time declined from 190 seconds to 160. Part b) of the good news is that there actually has been a reduction of the previously-reported stutter, down from completely unacceptable to simply really annoying. (I can tolerate this but most of you probably wouldn't.)

Now I'm going to speculate about why I couldn't cure cancer, and about why Windows Vista just might. My guess is that Vista will prove to have a greatly improved file-caching scheme, and I will further guess that FSX has improved scenery look-ahead logic which exploits the presumably-improved file cache.

I don't like to admit defeat, certainly not when it comes to technical software issues, so I'm going to consider installing Vista RC1 just to test my hypothesis. However, that would mean my installing and activating FSX for the third time. Before taking that step I would have to contact Microsoft Customer Support to make sure that I don't get shafted for doing three FSX activations in such a short period of time.

There is such a thing as being overly-paranoid, Microsoft. It is not reasonable to inconvenience a large number of customers just in order to make sure that absolutely nobody cheats. Of course we have no choice in the matter, but such is the stuff of which lawsuits are made. There is nothing on the product packaging, or inside the package, which says "Even though you bought the product, we are free to deny you the use it."

Flight1 handles these issues much better, a strong argument for downloads-only product delivery.

Regarding social progress in post-czarist Russia, Lenin said that you can't make an omelet without breaking eggs. Microsoft, where's the omelet?

Mike McCarthy
xxmikexx@qwest.net

Discuss This in our Outer Marker (Feedback) Forum.