Hurricane Screamer

With all the posting on tropical weather, you might think that I’m not taking the time to continue working on Print Preview and HUD-1. On the contrary, I have made quite steady day-to-day progress on the form. Significant parts of the form are complete and have been briefly tested. A lot of work needs to be finished, but the end is in sight.

The most visible change is the appearance of the form. I’ve placed all of the controls, assigned the onscreen help text, and set up the tab order and shortcut keys. And here it is:

The procedural flow of the code is 100 percent settled, but there are still gaps. The code to handle form data in the Data Server and the Open From Form button must still be written, as well as the display, zoom and pan code. But the code to read and render from a file is done and tested. Also, the code to open a file or form from either Print Preview’s command line or the Open From File button is finished.

The biggest pieces of code that has yet to be even considered (IE, planned) is the DoubleClick code and the MouseMove code.

DoubleClick() will determine the field name (such as Line 235) the user just double-clicked upon, and either open a HUD-1 data entry form with the correct file (if the file is not already open in a data entry form) or shift the focus to the correct data entry form in HUD-1’s main window if the file is already open. The HUD-1 form will then jump to the wanted field, and select the text so that the user can instantly correct any errors.

MouseMove() will monitor the position of the mouse (in X, Y coordinates) over the rendered image, and display that coordinate on the form’s status bar. I might be able to add the name of the field, but that might take too much processor time.

The most significant code that is finished is contained in what I call the Render_Container class. This class has a method named Action(). In fact, every class in the Form Framework based on the CodeBase class has an Action() method. This is a throwaway method that presents a common interface to calling code and designers. If the class has one primary task, that task will be found in the Action() method.

In the case of Render_Container, Action() sets up all the contained objects (the data servers, the script generator, the rendering engine, a timer control and a GetFilename control) and controls the flow of execution between them, finally producing images for loading. Finally, Action() invokes the Bitmap_Load() method and waits for further input from the user.

All the code in Action() has been tested and it works fine. It correctly reports all errors and exits gracefully in case of a failure, and resets the form when finished so that the user can start a new rendering operation.

The form itself is capable of quite a lot:

• Produce images representing the separate pages of a normally printed report so the user can check their data input;
• Render an image from either an open data entry form or a file;
• Check that a file is both a valid DBFINI file (a specific kind of FoxPro table file, designed to hold settings and data) and a HUD-1 centric file;
• If opened from the HUD-1’s Print Dialog Box, PrintPreview will automatically render from the Dialog Box’s Current File field (which can be either a file or a passed data entry form name);
• If opened from a HUD-1 data entry form, PrintPreview will automatically render from the data in the form;
• If opened from the HUD-1 program’s main menu, PrintPreview will wait for the user to click either of the Open From buttons, and render;
• Regardless of how PrintPreview is started, allow the user to render again and again until the form is closed;
• Show the progress of reading, scripting and rendering operations and any errors through a status bar display;
• Allow the user to zoom in and out, and pan up and down and left and right;
• Allow the user to select distinct pages of a multi-page report with a combobox;
• Respond to double clicks as explained above.

I have not posted on my progress regarding the production version of the PrintPreview form, and I apologize. The hurdles have been relatively minor, but quite vexing.

I have moved the Data Server, Script Generator and Rendering Engine to the production environment. Since my last post, I’ve been trying to get them to work with no errors. There have been many problems, mainly with the new organization of the data servers.

Most of my errors have been ones of commission, where I forget the effects a single line of code can make. The other errors are ones of omission, where a single missing line of code can abort the entire process.

And it’s a complex process. The data server will grab and format the code from the source, the script generator will read tables describing the script and put the script together with the formatted data, and the rendering engine will create an image with the elements described by the script. In all, tens of thousands of lines of code are involved, in three different languages. And now, all that is working together to produce these two images:

Keep in mind that the images you see are described only as disparate parts of a relational database, composed of seven bound tables and a free table (the data file). These two images were created on the fly, and in less than 30 seconds. Subsequent versions of the underlying code will be much faster.

I still have many other things to do with this form before I am finished. I must add the display, image (page) switching, panning and zooming code, and code to delete the images when the PrintPreview form closes. I must also add the code that will interpret a double-click and open the original data source (in a form) for editing on the double-clicked field. This kind of backward connection is what makes HUD-1 so difficult to write, but so rewarding to use.

I also have the idea of allowing the PrintPreview form to open data files or forms on it’s own. It’s strange, but previous versions of the form could not do this, depending on being passed a command line parameter for its data source. This PrintPreview form will be much more complete!

Over the last few days, I have been copying everything relevant from the beta Script Display form to the production Print Preview form. In the process, I have copied many of the same controls, and added two new ones. I have also encapsulated the GDI+, rendering and data source classes into its own container class, and placed the class into the form.

I have added just enough code for the form to be able to display itself, and here it is in its first operational version:

Can you spot what’s wrong with this form? The “No Preview” panel should read “Please Wait A Moment” instead of “Click View Results.” No matter; that’s a trivial change.

As you can see, the new form follows the original script rendering form fairly closely. The preview from file and preview from form buttons have been eliminated, and replaced with a page selection combo box. A new button, “Test,” has been added that will be visible only when the form is operating in test mode. All the other controls are exactly the same; the zooming, centering and reset buttons are the same, and the panning sliders are the same.

The next step is to begin to connect all the controls so that it can produce useful results. I will be testing this form in FoxPro’s IDE for the time being, and will be taking this whole process very slowly. Note that the script rendering code has never been used to produce more than one page at a time, and this form will have to produce three pages in succession. So I will be testing multiple page creation, and new code that will be added to allow choosing from those pages.

I will also have to add what I call the “double-click” code. This code was a part of the original Print Preview form. When the user double-clicks on a field, HUD-1 will bring up the originating data field from the source. If the data came from a form that is open, the focus will switch to that form, and if the data came from a file, the file will be opened in a new form. The great part about this is that the double-clicked field will then be highlighted in the displaying form, ready to be edited if the user desires.

All that will have to be added over the next few weeks. My target for getting this all working and releasing this as the next operational version of HUD-1 is by the end of July.

There have been many bumps on the road to completing the Print Preview form, including a major upgrade to my computer which diverted me for two entire weeks. The most vexing bump was the fact that I wrote code blindly into the HUD-1 program after upgrading to FoxPro 9, without testing it. After running the main HUD-1 form in Beta Version .9, I ran into the problems created by this lack of testing.

So, I had to write my own code to make up for the deficiencies in the FoxPro functions that didn’t work the way I needed, as well as breaking the previously working HUD-1 form. I tested those new function (in C) and they worked. Except that…. Well, the functions only work when the file actually exists, and will fail when the file isn’t already present in the file system. Which means that writing a brand new file will fail because the functions need to work before the file exists and the file won’t exist until it exists.

Yesterday and today I have spent my time re-writing the new functions to accommodate the possibility that the file does not exist. I am not entirely satisfied with this code because it will split the file and pathname in exactly the same was as FoxPro does (which is what I don’t want) when the file does not exist. I will take the next few days and re-write the code so that the pathname splitting function can be called separately, with the IsFile() and IsFolder() parameters settable outside the function.

I’ll have to make some changes to this entire branch of the FLL code, but it will be backward compatible with upper level code in FoxPro. I’ll work on this tomorrow and Friday, and then switch back to STRUCT for Saturday.

On Monday, I’ll be creating the new Print Preview form. I anticipate about one month’s work on that, and from there we shall see!

I have finished the Isxxx and Justxxx functions in the Form Framework’s C extension module, ff_fll.fll. It took a little longer than I wanted, but I have been pulled aside by end-of-month processing and by a special HTML project for new web site.

All the functions work with the predictability I need to use them in the main HUD-1 project, and by extension, any other Form Framework project.

I have changed the amount of power in the intermediate functions, but have retained the three-tier design I proposed two weeks ago. Now, the top tier handles the interface between FoxPro and the second tier. The second tier has been re-written to act as an intermediary between calling functions and the fileSystemObjectInfo function, which handles all the heavy work of grabbing information about the file and splitting the fully qualified pathname into the parts I want, and how I want them.

Here is the core dump* from my debugging session when I ironed out all the bugs from the main C function, fileSystemObjectInfo:

- myinfo 0×0012e8d8 {attributes=1077936503 isFile=1 isFolder=0 …} FSOInfo_s *
attributes 1077936503 unsigned long
isFile 1 int
isFolder 0 int
+ displayName 0×0012e8e4 “ff_fll.fll” char [260]
+ fileType 0×0012e9e8 “FoxPro Extension Library” char [80]
+ drive 0×0012ea38 “D:” char [3]
+ path 0×0012ea3b “D:\FoxPro Source\FormFoundation\FLL\fll\” char [256]
+ dir 0×0012eb3b “\FoxPro Source\FormFoundation\FLL\fll\” char [256]
+ filespec 0×0012ec3b “ff_fll.fll” char [256]
+ filename 0×0012ed3b “ff_fll” char [256]
+ fileext 0×0012ee3b “fll” char [256]

myinfo is a pointer to a structure that is filled in by the fileSystemObjectInfo function and used by calling functions.

In theory, I could do away with the second tier, and have the FoxPro interface functions create the structure and call fileSystemObjectInfo directly. But I have made enough changes for the moment.

I will take the rest of the week and add to what information the fileSystemObjectInfo grabs (such as size, and time-and-date information) and return to that HTML project I referred to earlier. Next week, I shall begin replacing the FoxPro Justxxx functions with my versions.

*Core dump – well, that really dates me. A “core dump” is a dump to a printer of the contents of a mainframe’s core memory (iron ferrite cores) when a program crashes. This nomenclature became used in the early 1950’s, just days after the first memory cores were installed into vacuum-tube based computers. The term is still used and means basically the same thing, but the “cores” are now microscopic silicon transistors, and the dump can be to the screen during real-time tracing and debugging.

When I upgraded to FoxPro 9 to gain additional enhancements, I had to re-write a great deal of code. Much of my work was based on an old FLL named Focus. Because Focus would not work with FoxPro 9, I had to use whatever FoxPro 9 provided and write a new FLL when FoxPro didn’t have what I needed.

One of my most-used functions in Focus uses an old DOS function to separate the drive letter, path, filename and extension from a file specification. FoxPro seemed to have exactly what I needed in the collection of the Justxxx() functions. So when I re-wrote the code in the HUD-1 form, I used those FoxPro functions.

This was a very bad mistake. I did not check to see if the result was the same for the Focus function as it is in the new FoxPro function. And the results are different. Here’s how:

If I put the following path into both functions:
C:\Full Length Movies\NORAD.911.TAPES

Here are the results I get back:
Focus: C:\Full Length Movies\NORAD.911.TAPES
FoxPro: C:\Full Length Movies

If I use this path:
C:\Full Length Movies\NORAD.911.TAPES\ (note the trailing slash)

Here are the results:
Focus: C:\Full Length Movies\NORAD.911.TAPES\
FoxPro: C:\Full Length Movies\NORAD.911.TAPES\

It’s obvious that, even though NORAD.911.TAPES is a folder, that JustPath strips it out of the returned path as though it were a filename. This gets even more odd when I try to get the extension using the JustExt() function. It returns .TAPES when it should return a blank string! And this is the problem.

These functions require a very tough programming standard, and I don’t always adhere to tough standards, even my own. Other programmers have the same habit.

So I have to make these functions a lot smarter. I propose to do this by extending my FLL with more file and directory functions. Along the way, I’ll also add a few other functions to extend and balance the new ones. Here’s the list of new functions:

APP_IsPath – is this a path to a folder?
APP_IsFile – is this a path to a file?
APP_JustDrive – just the drive letter and colon.
APP_JustFolder – Just the path w/o drive letter, filename or extension, otherwise unmodified.
APP_JustPath – Just the drive letter and folder path.
APP_JustFilespec – Just the filename, the period and extension.
APP_JustFilename – Just the filename, no period or extension.
APP_JustFileext – Just the extension, no period or filename.

The results of these functions will be absolutely predictable. If you put a path to a folder into the APP_JustFolder() or APP_JustPath() functions, you will get the correct answer, regardless if the path ends with a slash. The App_IsPath() and APP_IsFile() functions are added to provide upper-level access to the low level functions that support the others.

I plan on writing these functions and their support functions in a three-level tier structure. At the top are the above functions, which are really there to expose the second tier functions to FoxPro. The second tier accepts calls from the first tier, passes the parameters to the single third-tier function, and interprets the results. Tier two then passes the results back to the top tier which sends the results back to FoxPro.

This is a fairly common approach for me, as I have written a lot of my structure interface code this way. It has two advantages: a common interface and expectations between FoxPro and C using multiple C functions and direct invocation by C level code of the entire second tier by other functions in the FLL. The third tier code calls various Win32 and Shell functions and fills in a structure that is passed from above. All the functions in the second tier use information provided by the third tier to greater or less extent.

There is one additional upper-level function I’ve proposed, but it will have to wait until I resolve some problems with returning strings to reference variables. Or I might use the FoxPro STRUCT class to create a structure that can be passed back to a calling method. This would be an upper-level function and be named APP_FileInfo().

Over the last week, I have created new rendering classes and moved the beta code to them. I also enhanced the error trapping capabilities of the classes and did a lot of reorganization to the top structure of the class library.

This type of reorganizing is a bit unusual for me, as I usually very clearly see the needed structure of a library in the first few minutes of design. But rendering has been so large, using so many different components, that making it simple and easy to use has been a real challenge.

As I tested the various components of the rendering branch the final structure evolved to a class library with five basic classes.

A containing class
This class is provided for project designers in which to drop all the other classes and to write controlling methods and properties. In that way, it is like the FFForm class used by the earlier version of the Form Framework.

A script rendering engine.
The engine reads a script containing position and size information about various visual elements, such as lines, text, images, opacity and color. It then uses GDI+ to render those elements onto an output. The rendering engine need have only two distinct classes: image and printer. Image creates an image, specifically a bitmap, that can be loaded into a form or control (IE, a print preview form). Printer puts output to a local or network printer.

A script generator.
This class reads script description tables, and invokes a provided data server class to grab data from various sources, puts it all together into a single script and makes it available to the rendering engine. There are no provisions to change the way the tables are formatted or organized, but that could be an enhancement for the future. At present, there is only a single class.

A data server.
This is a base class with all the code necessary to read a formatting table and format the incoming data. What the base class does not have is the code needed to read a particular data source, which is handled by specific code written into a subclass. In traditional fashion, I also refer to this class as the Print Computational Matrix, a very old term for the formatting array used to hold the data and format strings.

Sources for the data server would be:
–A HUD-1 flat file.
–A HUD-1 data form, from either a caption name or an object reference.
–A more complex table or relational database structure.
–An HTTP, FTP or other IP source. Note that this would split the data server into two pieces, with the near piece being only a network client and the far piece being a network server, as well as having the code to grab the data.
–An OCDB connection.
–A MSSQL or MYSQL database.

Each one of these sources, or combination of sources, or even sources not listed here would be handled by a distinct subclass. The subclasses would be written for the particular application, and would work with all the other components and the base code, so long as the data server subclass can find each data item by a distinct name. To extend this even further, a name could even be a distinct query string, as in SQL.

It is in the data servers in which the largest changes were made. I first tried to make the server remote from the generator using object references over two forms, but ran into a problem with closing both forms independently. If the object reference is not disconnected the form will refuse to close. Fixing this problem added so much complexity to the project as a whole, I scrapped it. The form framework is supposed to add simplicity, not “impossibly hard to use.”

My eventual compromise is very close to the structure of the original Print Preview: have the generator and server in the same container and adjust the data server as necessary. The final solution is to have the containing control hold the rendering engine, the script generator and as many specific kinds of data server as needed for the project.

The container would receive the command string (IE, “FORM:Billing Invoice” or “FILE:c:\data\mydataform.dbf”), select the proper data server, set up the rendering engine and the script generator and then start the rendering process. After it is done, the container would then signal that it had either finished the rendering and where the results would be found or had failed and indicate the reason why.

At this point, the prototype overall container would not contain the code to make the Print Preview form work. That will still be handled by the FFForm control. I will have to create a new subclass of the control, and migrate all the relevant code from the current version of the Print Preview form’s FFForm to it (such a FormInit, right-click or double-click), and create new methods to handle the image zooming and panning, page switching, and doing the actual rendering.

My plans for the new Print Preview form are a bit ambitious, but that is for the next post.

The entire chain of objects, from report description tables, to data processing and on to rendering and GDI+ class libraries has been verified to work in a development environment to produce a printable and viewable form.

This entire branch of the Form Framework V2 has taken two years to design, write, test and integrate. In all, more than 25,000 lines of FoxPro code and 10,000 lines of C and C++ code contribute to Print Preview. There are still known problems, and several bits of clumsiness that must be resolved before the entire package is moved over to a production environment.

The next step is to further refine the code, and to move it over to classes within the framework, rather than existing in the specially created development classes.

I plan on several refinements to the data server class that will allow a single class to concentrate on one method of grabbing its data. The base class will contain the formatting code, and each subclass will grab the data by different methods. That way, the data server will be more versatile.

Currently, there are two main methods for grabbing data: FORM and FILE. But these methods are specific to the HUD-1 platform, and would not work in another environment. If the data were coming from a SQL database, or a relational database rather than a flat file, or a network protocol, or an object, or a form without the FFForm1 object, the current class would not be able to handle it.

In essence, new subclasses of the data server could be created for a specific application, with only the GetData method overridden and sub-objects (such as WinInet and SimpleINI) added to support the transfer of data.

Another enhancement I have in mind is to encapsulate the rendering engine, data server and script generator into a large containing class that will include object properties to point to the image display, the status and abort controls, and the vertical and horizontal sliders. This class will appear as a prototype in the HUD-1 program, and may be used in other applications.

In all, I anticipate another month or so of work on the migration and enhancements of code over to HUD-1, and then of testing the new Print Preview form. There are several issues that I must work on, especially the scripts for the location-specific tax calculations, but that should not take too much work.

I anticipate releasing the first version of the HUD-1 program in about two months. It won’t have much of a help file, but that’s the breaks.

On the 14th, I promised to explain how to reproduce the errors with Visual ForPro and its buggy interaction with MSXML 3 SP10 (service pack 10).

The Problem

When the metadata property is defined in a VFP class, the property string is turned over to MSXML 3. If the string has content, MSXML returns the results to VFP, but slows down rendering the property sheets. If the string is null, MSXML throws an exception, which is ignored by VFP but not by Visual C++ if the programmer is in a debug session. Another symptom of the exception is that classes with a large amount of code (10,000 lines or more) can take significant amounts of time and processing power to save after changes are made.

I write extensions to VFP in C and C++ and I use Visual C++ 7.1 as the compiler. The resulting file is a multi-thread DLL with special tables so that VFP can call functions directly. This kind of file is called an FLL, or FoxPro Linked Library. To test the FLL’s functions, I write VFP testing code, and enter a VC debug session with VFP as the executable. I run the testing code (a form or program file) in VFP, and, if I have everything right, VC will hit the break points I have set in the C code and allow me to start tracing the execution of the function.

Except when MSXML is invoked with blank metadata. While FoxPro does not display the exception, VC in debug mode does, and sometimes thousands of times.

Reproduce The Error

1. Create a VFP project in a new directory.
2. Subclass a new control from a button.
3. Use the MemberData Editor to create some metadata in the class.
4. Create a form with a single instance of the new button class and put code into the control’s Click event to call an FLL function. You might want to include code to load the FLL in the form’s Init or Load method.
5. Exit VFP with the project file still open.
6. Open VC++ and create an FLL project in the test directory. Write and build the FLL.
7. Remembering to set the executable to debug to VFP and setting breakpoints in your C code, enter a debug session by pressing F5.
8. Keep your eye on the Output window. On my system, there are about 25 lines of output as VFP runs.
9. In VFP, load the test form and click the button. The FLL function is called, and VC++ stops at the breakpoint. Press F5 to continue.

Making The Error Happen

The string in metadata is valid, and MSXML will process the string correctly. The contents of the Output Window may show the FLL being loaded, but nothing much more than that.

At this point, return to the Class Designer and delete the metadata string. Return to the Form Designer (remember VC is still in Debug mode), and run the form. When you run the form, you should see exceptions appear in the Output Window. The line(s) will contain the text “First-chance exception” and the address 0×7c812aeb. This is when the bug manifests itself!

A Workaround

The fix for this problem was so simple, it took a while for me to see it. At first, I considered replacing the MSXML 3 files with a previous version (Service Pack 9), but saw during my tests that the processing power required to save classes was the same as under SP10. The only difference between the two versions seemed to be that SP9 didn’t throw any exceptions visible in VC, even though it was having some of the same problems with blank metadata.

I finally hit on something. A control class inherits many properties and methods from its parent class, but the metadata property is not on that list when the class is newly created. It only appears when I use the MemberData Editor. The metadata property seemed to be acting like a custom property, which was added by the MemberData Editor. And, when I add custom properties, I can always remove them. I asked myself, “is the metadata property considered custom even if added by VFP?” If so, I can remove it from the class!

And that’s what I did. I visited the base class of the Form Framework (ffBaseClass) and removed the metadata property using the Edit Property/Method dialog box. And it went away, and so did the errors in VC!

That’s the workaround: remove the entire property from the class!

A Note Of Caution

This procedure is only a workaround. It may not, in fact, be a real fix, but may be opening the door for other problems in the future. But it seems to work, and Microsoft has not offered any solution to this vexing bug.

I have detected a bug in Microsoft’s XML Core Services DLL msxml3.dll. It was uncovered because of a bug in Visual FoxPro 9 and how it handles metadata.

In short, if MSXML is presented with what appears to be an improper XML string from VFP, MSXML will choke on it and raise exceptions like mad. This bug interferes with API code dealing with objects and with debug sessions under VC 7.1.

The programmer can embed metadata into a VFP class property called MemberData. Metadata is described by MSDN as:

Visual FoxPro 9.0 provides extended metadata attributes for class members. Through this new extensibility model, you now have the ability to extend the functionality of class properties, methods, and events, allowing you to create design-time enhancements such as the following:

* Specify a custom property editor.
* Add a property, event, or method to the new Favorites tab in the Properties window.
* Control the display (capitalization) of a custom property or method in the Properties window and IntelliSense.
* Extend with user-defined attributes.

Early in the development of the second version of the Form Framework, I added a few bits of memberdata to the base control, to capitalize my custom property and method names. Working with subclasses of the base control, the slowdown in property sheet rendering was so bad as to make work impossible. I now know that this was the fault of MSXML, which is very resource intensive. I deleted the value of memberdata, but was not aware of what actually happened.

VFP subclassed controls are stored in DBF files (with a .VCX extension). The DBF files contain the names of any embedded controls, and the names of any custom properties and methods of those controls, or non-default initial values of inherent properties, including metadata. When a custom property is added, or an inherent property is assigned anything other than a default value, the name and value of the property is stored in the vcx file.

In other words, custom properties are always written to the vcx file, because they are given a value as part of their creation. VFP relies on its own internal default values for inherent properties (such as memberdata), so will not write the property names to the vcx until the property’s value is changed by the programmer.

If a property had a value, it might look like this (assuming it was a char type):

Name = “test”

That’s the name of the control: “test.” On the other hand, when the programmer deletes the value of a property, it presents itself to VFP like this:

test =

Note the lack of anything to the right of the equals sign. This is what I believe causes the problem with MSXML: this is not a null string (ie, “”). It looks like a data type I haven’t seen since the 1970’s: Undefined. VFP can apparently process this undefined value type but MSXML cannot. Perhaps it is because VFP interprets this value as a logical False, which is the default value of a newly created property. Of course, its VFP type would then be Logical.

And here’s the second part of the problem: until the memberdata property has an actual defined value, it’s not placed in the control’s vcx file, just like all other properties, which are not stored until they are defined. But once it does have a value, even if the programmer later deletes the value string, the property name remains as part of the vcx table, and because its totally blank, it reverts back to type Logical with a value of False.

And here is the bad interaction with MSXML: when the memberdata property is not present in the vcx table, MSXML will not be invoked. When the property has been assigned a value, it’s placed in the table, and MSMXL will be invoked whenever the object or a reference to it is invoked (such as an API call or in a VC debug session or in the IDE), even when the xml string value is invalid in some way!

Tomorrow, I will write as to how I discovered this bug, and how to re-create it.