Hurricane Screamer

On the 14th, I promised to explain how to reproduce the errors with Visual ForPro and its buggy interaction with MSXML 3 SP10 (service pack 10).

The Problem

When the metadata property is defined in a VFP class, the property string is turned over to MSXML 3. If the string has content, MSXML returns the results to VFP, but slows down rendering the property sheets. If the string is null, MSXML throws an exception, which is ignored by VFP but not by Visual C++ if the programmer is in a debug session. Another symptom of the exception is that classes with a large amount of code (10,000 lines or more) can take significant amounts of time and processing power to save after changes are made.

I write extensions to VFP in C and C++ and I use Visual C++ 7.1 as the compiler. The resulting file is a multi-thread DLL with special tables so that VFP can call functions directly. This kind of file is called an FLL, or FoxPro Linked Library. To test the FLL’s functions, I write VFP testing code, and enter a VC debug session with VFP as the executable. I run the testing code (a form or program file) in VFP, and, if I have everything right, VC will hit the break points I have set in the C code and allow me to start tracing the execution of the function.

Except when MSXML is invoked with blank metadata. While FoxPro does not display the exception, VC in debug mode does, and sometimes thousands of times.

Reproduce The Error

1. Create a VFP project in a new directory.
2. Subclass a new control from a button.
3. Use the MemberData Editor to create some metadata in the class.
4. Create a form with a single instance of the new button class and put code into the control’s Click event to call an FLL function. You might want to include code to load the FLL in the form’s Init or Load method.
5. Exit VFP with the project file still open.
6. Open VC++ and create an FLL project in the test directory. Write and build the FLL.
7. Remembering to set the executable to debug to VFP and setting breakpoints in your C code, enter a debug session by pressing F5.
8. Keep your eye on the Output window. On my system, there are about 25 lines of output as VFP runs.
9. In VFP, load the test form and click the button. The FLL function is called, and VC++ stops at the breakpoint. Press F5 to continue.

Making The Error Happen

The string in metadata is valid, and MSXML will process the string correctly. The contents of the Output Window may show the FLL being loaded, but nothing much more than that.

At this point, return to the Class Designer and delete the metadata string. Return to the Form Designer (remember VC is still in Debug mode), and run the form. When you run the form, you should see exceptions appear in the Output Window. The line(s) will contain the text “First-chance exception” and the address 0x7c812aeb. This is when the bug manifests itself!

A Workaround

The fix for this problem was so simple, it took a while for me to see it. At first, I considered replacing the MSXML 3 files with a previous version (Service Pack 9), but saw during my tests that the processing power required to save classes was the same as under SP10. The only difference between the two versions seemed to be that SP9 didn’t throw any exceptions visible in VC, even though it was having some of the same problems with blank metadata.

I finally hit on something. A control class inherits many properties and methods from its parent class, but the metadata property is not on that list when the class is newly created. It only appears when I use the MemberData Editor. The metadata property seemed to be acting like a custom property, which was added by the MemberData Editor. And, when I add custom properties, I can always remove them. I asked myself, “is the metadata property considered custom even if added by VFP?” If so, I can remove it from the class!

And that’s what I did. I visited the base class of the Form Framework (ffBaseClass) and removed the metadata property using the Edit Property/Method dialog box. And it went away, and so did the errors in VC!

That’s the workaround: remove the entire property from the class!

A Note Of Caution

This procedure is only a workaround. It may not, in fact, be a real fix, but may be opening the door for other problems in the future. But it seems to work, and Microsoft has not offered any solution to this vexing bug.

I have detected a bug in Microsoft’s XML Core Services DLL msxml3.dll. It was uncovered because of a bug in Visual FoxPro 9 and how it handles metadata.

In short, if MSXML is presented with what appears to be an improper XML string from VFP, MSXML will choke on it and raise exceptions like mad. This bug interferes with API code dealing with objects and with debug sessions under VC 7.1.

The programmer can embed metadata into a VFP class property called MemberData. Metadata is described by MSDN as:

Visual FoxPro 9.0 provides extended metadata attributes for class members. Through this new extensibility model, you now have the ability to extend the functionality of class properties, methods, and events, allowing you to create design-time enhancements such as the following:

* Specify a custom property editor.
* Add a property, event, or method to the new Favorites tab in the Properties window.
* Control the display (capitalization) of a custom property or method in the Properties window and IntelliSense.
* Extend with user-defined attributes.

Early in the development of the second version of the Form Framework, I added a few bits of memberdata to the base control, to capitalize my custom property and method names. Working with subclasses of the base control, the slowdown in property sheet rendering was so bad as to make work impossible. I now know that this was the fault of MSXML, which is very resource intensive. I deleted the value of memberdata, but was not aware of what actually happened.

VFP subclassed controls are stored in DBF files (with a .VCX extension). The DBF files contain the names of any embedded controls, and the names of any custom properties and methods of those controls, or non-default initial values of inherent properties, including metadata. When a custom property is added, or an inherent property is assigned anything other than a default value, the name and value of the property is stored in the vcx file.

In other words, custom properties are always written to the vcx file, because they are given a value as part of their creation. VFP relies on its own internal default values for inherent properties (such as memberdata), so will not write the property names to the vcx until the property’s value is changed by the programmer.

If a property had a value, it might look like this (assuming it was a char type):

Name = “test”

That’s the name of the control: “test.” On the other hand, when the programmer deletes the value of a property, it presents itself to VFP like this:

test =

Note the lack of anything to the right of the equals sign. This is what I believe causes the problem with MSXML: this is not a null string (ie, “”). It looks like a data type I haven’t seen since the 1970’s: Undefined. VFP can apparently process this undefined value type but MSXML cannot. Perhaps it is because VFP interprets this value as a logical False, which is the default value of a newly created property. Of course, its VFP type would then be Logical.

And here’s the second part of the problem: until the memberdata property has an actual defined value, it’s not placed in the control’s vcx file, just like all other properties, which are not stored until they are defined. But once it does have a value, even if the programmer later deletes the value string, the property name remains as part of the vcx table, and because its totally blank, it reverts back to type Logical with a value of False.

And here is the bad interaction with MSXML: when the memberdata property is not present in the vcx table, MSXML will not be invoked. When the property has been assigned a value, it’s placed in the table, and MSMXL will be invoked whenever the object or a reference to it is invoked (such as an API call or in a VC debug session or in the IDE), even when the xml string value is invalid in some way!

Tomorrow, I will write as to how I discovered this bug, and how to re-create it.

After a few false starts, I have finished scripting the Invoice Form. In all it took about 20 hours of hand scripting and testing to fully script the form with visual objects and data.

Here is the output of the rendering engine:

In a professional environment, the process would be assisted by using a design surface of some kind. One that can manipulate visual objects, and allow the user to assign attributes and test data, and then create the proper tables and code fragments to access them. That would be easier and much faster, especially in an environment where many forms are needed or make many changes. Maybe that can be added to the Form Framework V2 as an add-in, but it will have to wait for three other project branches.

It’s amazing when a testing tool turns into a real development tool. When I created the form to test the rendering engine, I didn’t know that I could use it to edit and render a script in progress.

So far, I have finished writing into the script all of the permanent artifacts that will appear on the billing form. These are the comment fields, and the horizontal and vertical lines. I will be writing the data elements in tomorrow and Thursday, and getting ready to begin serious testing on Friday. Of course, some of that time will be used to move the individual script items to the FoxPro tables that will be read by the script generator.

To test the script, I am using the form I created to test the rendering engine. I have seen a few problems with the engine that need to be addressed before it moves to a production environment. But nothing too serious.

I am most impressed by the fact that I’ve been able to use the older test form to actually write and test the script. The editing facilities are very primitive, but can be good for single script items. I use a regular editor (notepad++, a freeware replacement for Windows Notepad), and switch between the form and the editor. It’s amazing to me that the form is being used now to help design and test the next form. It’s always nice when a tool exceeds my perceived usefulness.

I have not blogged in quite some time. There’s always something else going on. Sorry about that. Once again I make the empty promise that I’ll blog at least once a day. Or maybe once a week.

I have finished the beta of the rendering engine, and turned my attention to the other half of the task. That is, an object that will create a script. Since the task is really split into two parts, I have decided to create two new objects.

The first object is responsible for generating the lines of script from the data. This object can be embedded in the forms that either hold or point to the data (as a form, file or object reference). It is currently in alpha form. The code is somewhat limited in that it can only understand a file that conforms to HUD-1 standards. But the code that was originally used to calculate the Print Computational Matrix has been split up into separate methods, so is a bit easier to change when the overall application requires it. This change can be done by overriding the data_getfield_file() method.

The second object has overall command of the entire process of generating a script. It will tell the data script object where to find everything and when to do its job, read the chain of report descriptor files and context menu descriptor files, generate the script, tell the calling form where the finished script can be found, and keep the user informed of script generation progress.

The amazing thing about these two objects is how little work I had to do to write them. The Print Dialog Box from HUD-1 contains all this code already. All I had to do was to extract the print() method, re-write it, and put in the code that supports the Form Framework V2, and poof, there it was. Along the way I discovered an egregious error in the menu code that is corrected in the new data object.

I have also created the forms that emulate the HUD-1 print/preview chain, and have adapted the already existing HUD-1 code to the new conditions. I have come to the point where I can begin beta testing the two new objects. I begin this tomorrow. For the rest of today, I will work on writing the script items that will produce a billing form when rendered. For next week, I will switch back and forth between testing the script generation code and writing the script elements.

Can you tell me what’s wrong with this?

if ascan(menuNouns, upper(alltrim(this.pcMatrix[x1,2]))) = 0
   for x2 = len(this.pcMatrix[x1,2]) to 1 step -1
      if len(this.pcMatrix[x1,2]) > 0
         c2 = substr(this.pcMatrix[x1,2], x2, 1)
         c3 = iif((c2 >= "0" and c2 <= "9") or (c4 = .F. and c2 = "."), ;
         c2, "") + c3
         c4 = iif(c2 = ".", .T., c4)
         endfor
      else
         c3 = "0"
      endif
      c2 = alltrim(transform(val(c3), right(c11, len(c11) - 1)))
else

[Note: copyrighted egregious error, all rights reserved]

I have taken advantage of the time available over this holiday weekend to work on aspects of the Form Framework other than GDI+. There are many things that need to be completed before I can begin writing applications based on the framework.

One of the outstanding issues is the Struct class. Struct is a way for FoxPro programmers to easily create, fill and destroy C and C++ structures in memory prior to a Win32 call. Structures are everywhere in Win32, and it would take a lot of work to implement each one in its own separate class complete with individual data items.

There are three basic parts to the Struct class:

• The class library itself, which consists of a container control for each structure, a collection class to hold the actual data, and class definitions to define a structure element, a union element or an enum element.
• The ffCTypesAndSizes class (part of CodeBase) is the VFP front end of C code for conversions, reading writing, etc, and which defines the intrinsic C data types and how to process them.
• The C code which provides the size in bytes of the various intrinsic C data types, reading and writing from memory, allocating and deallocating local heap memory, and code to convert each kind of data to and from a form which VFP can manipulate.

To verify the code, I have to start at the very bottom, with the C code. Over the last few days, I have written testing programs for each C function, and have verified that each one works. The next step is to test the ffCTypesAndSizes class.

Earlier this year, I did something I have never tried. The separate functions of the C code were getting out of hand. There were too many, some of them duplicated code in other functions, they all had calling parameters that were different, and since they were FLL functions, there was no easy way to call any of them from C itself.

It took a while, but I cut down the number of actual functions, made all the argument lists the same and split the ones that needed it into two parts. That little bit of discipline made writing the ffCTypesAndSizes interface not only easier, but possible.

Next, though, I will be jumping back to the Print Preview form. That will be next Tuesday, as I will be traveling tomorrow.

For my Christmas gift this year, I have given myself the full implementation of the panning and zooming code for a Print Preview form. With the completion of the prototype rendering engine for bitmaps, the slider code, and the panning and zooming code for the form, I feel that the GDI+ Beta program is ready for its next release.

You can download the current version by clicking this link:

http://hud-1.net/betaversions/GDI+Beta.setup.4.027.exe

The code exists in prototype form at this point. When it is upgraded to the HUD-1 project, I will be creating a production version of the rendering engine that can take advantage of the full capabilities of GDI+.

I’m thinking of creating a single-page pageset control that will encapsulate a multipage print preview viewpane. On the originating form side, I would have to create a custom control that would be either a script server or a data item server for the viewpane. I’ll begin to work out those details a bit later.

For now, I will continue testing and extending the Struct class, a class that has been sadly neglected over the last year.

This is not a take on the famous Beatles movie, Help! For the last few weeks I have been adding pages relating to the new rendering form to the GDI+ Beta program’s help file. I have also added many pages explaining the syntax and usage of the scripting language used by the rendering engine.

There are two buttons to which I have to add code, and I have to make one correction to the sliding code, but I’m almost done with the form as a whole.

However, I’m going to Florence, SC for Christmas and will have to finish up these details on Friday.

In the meantime, here is some Christmas Cheer from Annie Lennox:

Enjoy!

Visual FoxPro has many benefits, but several serious lacks. One of these is native support for pixel-wise printing when creating a custom report. I have solved this by creating the GDI+ classes. Another serious deficit is the lack of native support for vertical and horizontal sliders. These controls are so useful, I would have thought that Microsoft would have added them at some point, but they never have. Neither has the contributors at CodePlex.

To this end, I ported an already existing sliders class from Visual Basic 6 to FoxPro. This was about four years ago, and each time I have used them, or worked on their code, I think that they are done. Then I used them in the GDI+Beta project, and I again learned that I had not.

Over the last two weeks, I have worked to find out why both the vertical and horizontal sliders were behaving erratically when they were embedded into a page frame, and why they would apparently be normal when not embedded. The problem has to do with the fact that the MouseDown event delivers its data to the receiving control using the absolute position of the click within the borders of the form, and that my sliders would process the event only in relation to any containers into which they might be embedded. That’s fine if the embedding container is a form, then the x and y coordinates match the left and top of the sliders.

But if they are embedded into a container control, such as a pageframe, the left and top coordinates become relative to the form through the pageframe. In other words, the left of the pageframe and left of the sliders have to be added to get the absolute horizontal position of a slider within the form. On top of that, the pageframe can have page tabs on either the top or the left, which adds a further offset of 30 pixels.

Over the last few days, I have had to write code accounting for all of this, to derive a control’s absolute position relative to a form. I have written a small test form to show that this code works, and to return the value of the sliders to a visual control (an edit box). This form tests two sets of sliders, one inside a pageframe and one outside the pageframe. They work perfectly in both places.

I now declare the vertical and horizontal sliders finished, at least as far as version one of the Form Framework is concerned. In fact, the self-discovery of its absolute position is so useful, I’m thinking of writing this code into the base code of all visual controls for the second version of the Framework, with two new properties: AbsoluteLeft and AbsoluteTop.

This removes a major block in completing the Script Test form of the GDI+Beta program. There is only tuning the startup conditions of the bitmap size and placement before I release the next version of the program, with the Print Preview form.

That work will begin on Monday.

In the meantime, I hereby release the code to the public domain:

LOCAL c1, c2, o1
dodefault()
*  set the vertical and horizontal mousedown offsets
o1 = this
c1 = 0
c2 = 0
DO WHILE .T.
    IF o1.baseclass <> "Form"
        IF PEMSTATUS(o1, "left", 5) = .T.
            c1 = c1 + o1.left
            IF o1.baseclass = "Pageframe"
                c1 = c1 + (o1.width - o1.pagewidth)
            ENDIF
        ENDIF
        IF PEMSTATUS(o1, "top", 5) = .T.
            c2 = c2 + o1.top
            IF o1.baseclass = "Pageframe"
                c2 = c2 + (o1.height - o1.pageheight)
            ENDIF
        ENDIF			
        o1 = o1.parent
    ELSE
        EXIT
    ENDIF
ENDDO
this.mousedownoffsetx = c1
this.mousedownoffsety = c2

This is from the Init method of the base class of the sliders, ffScrollBar. Even though released into the public domain, the code is still copyright to me, David M. Hartzell. Please provide proper citations for my work.

I have verified the reason behind the bizarre behavior of the slider controls on the image viewing pane. As you might recall, I last wrote about an odd quirk I observed while clicking on the horizontal slider; the thumb of the slider would endlessly jump from left to right, even while the clicking point was to the left of the thumb.

This would be correct if the thumb had actually moved all the way past the clicking point, but it had not. It was still to the left of the thumb. After examining the code, and creating a test form, and frankly having a shower-time insight, I finally was able to reproduce the error.

Here is what is happening: The MouseDown event is the one that notifies the control that it has been clicked. The event handler figures out where the click happened, and whether it is either to the left or the right of the thumb. The event handler does this by referencing the Left property of the slider control and the left and width properties of the thumb.

Here’s the sticking point: the x-coordinate delivered to the event handler is the x-coordinate relative to the form. If the slider is embedded in a page frame or other container, the left property of the slider will be different than that if it were in the same position on the form, but not embedded in the container control.

So, the MouseDown event handler is confused, by the fact that the slider control is offset from its actual position on the form, by not having that offset accounted for by the Left property of the page frame.

This means that I have to discover a way of informing the slider of its actual position within the form. I’ll start researching that today. I’ll probably find a way for the Init method to discover the hierarchy of containers up to the form level, and set an Offset property.