Wednesday, June 22, 2011
Weltmeisterschaft
I'm off for the next while to see the women's football world cup in Germany, so I'm unlikely to be posting much, and if I do, it probably won't be technical. See you all after the final.
Interesting Bugs
There are lots of interesting ways for software to go wrong. Here's one recent one that we uncovered at Cincom. And I should say that by "we" I mostly mean Tom Robinson, from the Store group.
I first noticed this bug in the middle of a demo in Frankfurt. I was showing some interesting Glorp and StoreGlorp capabilities. I'd run an interesting expression in a Store workbook, inspect the result, then run another expression and get an error. Force a reconnect, and it was fine again, but it occured several times.
Travis Griggs noticed an even odder manifestion of it in trying out some Store expressions. He wrote an expression to find the head of the 7.9 trunk for a particular package or bundle, and it ran fine, returning the expected result. Then he ran it to find the head of the 7.8 trunk, and it returned the exact same package as before, from the 7.9 trunk. Running it again produced the right answer.
Investigation revealed nothing obvious going on at the Glorp caching level. The database appeared to be genuinely returning results that were clearly incorrect for the query that was issued. Sometimes they were obviously for a different query. You might ask for a StorePackage and get back a single column that was obviously for the tw_databaseidentifier. This only happened when using Postgresql. And it apparently only happened when using the Store Workbook. Normal Store operations never showed this.
After much examination, Tom tracked down the cause, an interaction of several causes. First, when you open an inspector, there may be code specific to the type of object being inspected. In particular, the inspector shows the icon for things. The icon for packages shows differently depending if it's the version that's resident in the image or not. Asking that, especially on a completely new connection, could end up doing a database query. That's not really great, but should still work. But...
The inspectors try to be robust. If you write a printString that goes into infinite recursion, raises an exception, or otherwise doesn't return, it tries to stop it and just print an indication that it didn't work. So one of the mechanisms there is a timeout. If it takes too long, it just terminates the process. And in addition...
The Postgresql driver is written purely in Smalltalk. So there's Smalltalk code that manages a socket and the communication on it. This is in contrast to most other database drivers that call out to a C library, which may be just communicating on a socket underneath, but the protocol and details are hidden.
So, what happens is that we issue a query, and open an inspector on the resulting object(s). The inspector is the reason that this didn't affect normal Store operations. The inspector wants to know if this is the current version in the image, so it issues a query to the database. That query will run in a separate Glorp session, but because we're trying to be careful of resources, it'll re-use the underlying database connection. And there's more setup code than is really necessary that runs for each new session (or did, up until more recent builds). If the database is Postgresql, and isn't very close by on a fast machine, the inspector will time out before it gets the answer, so it will terminate the process. The database driver doesn't clean up properly when the process is terminated, so the previous results are left in a buffer. The next query that's issued on that connection will start looking for results, and get the results of the previous query out of the buffer. If the shape of those results matches what we're expecting, we'll just get a wrong answer. If the shape doesn't match, we'll get a confusing error. For example, if it's running the initial setup query to get the tw_databaseidentifier, we might see the result of that.
Fortunately, this only affects operations that use inspectors, so it has no effect on normal Store operations. And databases other than Postgresql keep the individual query results separate by themselves, so that won't happen. But it's a nice example of some very interesting interactions that normally don't show up on their own.
I first noticed this bug in the middle of a demo in Frankfurt. I was showing some interesting Glorp and StoreGlorp capabilities. I'd run an interesting expression in a Store workbook, inspect the result, then run another expression and get an error. Force a reconnect, and it was fine again, but it occured several times.
Travis Griggs noticed an even odder manifestion of it in trying out some Store expressions. He wrote an expression to find the head of the 7.9 trunk for a particular package or bundle, and it ran fine, returning the expected result. Then he ran it to find the head of the 7.8 trunk, and it returned the exact same package as before, from the 7.9 trunk. Running it again produced the right answer.
Investigation revealed nothing obvious going on at the Glorp caching level. The database appeared to be genuinely returning results that were clearly incorrect for the query that was issued. Sometimes they were obviously for a different query. You might ask for a StorePackage and get back a single column that was obviously for the tw_databaseidentifier. This only happened when using Postgresql. And it apparently only happened when using the Store Workbook. Normal Store operations never showed this.
After much examination, Tom tracked down the cause, an interaction of several causes. First, when you open an inspector, there may be code specific to the type of object being inspected. In particular, the inspector shows the icon for things. The icon for packages shows differently depending if it's the version that's resident in the image or not. Asking that, especially on a completely new connection, could end up doing a database query. That's not really great, but should still work. But...
The inspectors try to be robust. If you write a printString that goes into infinite recursion, raises an exception, or otherwise doesn't return, it tries to stop it and just print an indication that it didn't work. So one of the mechanisms there is a timeout. If it takes too long, it just terminates the process. And in addition...
The Postgresql driver is written purely in Smalltalk. So there's Smalltalk code that manages a socket and the communication on it. This is in contrast to most other database drivers that call out to a C library, which may be just communicating on a socket underneath, but the protocol and details are hidden.
So, what happens is that we issue a query, and open an inspector on the resulting object(s). The inspector is the reason that this didn't affect normal Store operations. The inspector wants to know if this is the current version in the image, so it issues a query to the database. That query will run in a separate Glorp session, but because we're trying to be careful of resources, it'll re-use the underlying database connection. And there's more setup code than is really necessary that runs for each new session (or did, up until more recent builds). If the database is Postgresql, and isn't very close by on a fast machine, the inspector will time out before it gets the answer, so it will terminate the process. The database driver doesn't clean up properly when the process is terminated, so the previous results are left in a buffer. The next query that's issued on that connection will start looking for results, and get the results of the previous query out of the buffer. If the shape of those results matches what we're expecting, we'll just get a wrong answer. If the shape doesn't match, we'll get a confusing error. For example, if it's running the initial setup query to get the tw_databaseidentifier, we might see the result of that.
Fortunately, this only affects operations that use inspectors, so it has no effect on normal Store operations. And databases other than Postgresql keep the individual query results separate by themselves, so that won't happen. But it's a nice example of some very interesting interactions that normally don't show up on their own.
Friday, May 20, 2011
Checking overrides when upgrading
Here's a question that came up in the vwnc mailing list. Suppose that you have an application which has overrides of a number of system methods. I'm also going to start referring to overrides as redefines here, because I never liked the ambiguity between an override in a subclass and the replacement of a definition that VisualWorks uses the term for. So I'll try calling them redefines here, even though it makes for a lot of backspacing.
Anyway, let's say that this was built in VisualWorks 7.6, but now you're upgrading to 7.8, and you want to check which of these methods no longer apply. That can be complicated in general, but a basic first check is if the method we were redefining changed between VisualWorks 7.6 and 7.8. That's a bit tedious to check manually, but we can script it.
Of course, this depends on how we've organized things. The original question came up in the context of having grouped redefinitions by the package that contained the thing being redefined. So, if we redefined a method in "Assets", we'd create a package "Assets patches" and put the redefinition there.
So here's an example of a workspace script for finding these. It assumes that we've published the old version of the base into our local database, and that we've loaded our code into a new image.
mySession := StoreLoginFactory currentStoreSession.
versionToUpgradeFrom := '7.7 '.
patchPackages := Store.Registry allPackages select: [:each |
each name like: '% patches'].
So, first we need to get a StoreGlorp session which we'll use to do our queries, and we define a variable for how we'll find the old versions. We'll also need the list of patch packages that we have in the image. I used the #like: method to do the matching, which does it in SQL style, rather than the more traditional matches:, or even regular expressions, but they'd all work.
patchPackages do: [:eachPatchPackage |
basePackageName := eachPatchPackage name readStream upTo: Character space.
currentOverriddenPackage := Store.Registry packageNamed: basePackageName.
Now we start a loop over each of our patch packages. The first thing we need to know is what the basic package is, which we do based on the simple naming convention described above.
baseQuery := Query readOneOf: StorePackage where: [:each |
(each name = basePackageName &
(each version like: '%', versionToUpgradeFrom, '%')].
baseQuery orderBy: [:each | each timestamp descending].
baseVersion := mySession execute: baseQuery.
Now we do a query to find the appropriate old version. So this involves a Glorp query to find packages by that name, whose version string matches the variable we set at the beginning. If there are multiples, we take only the latest, by sorting them in descending order and just taking the first one. I'm dealing with the Cincom internal repository, so there will be lots and lots of versions of base code. In a project repository there are probably only a few, making it fairly simple to find this.
Once we've done that, we'll loop over the methods in the patch package, and get the three different versions of the method: ours, the new base image version, and the old base image version. These will be three different kinds of objects, and the APIs for manipulating them and getting them are, well, let's just say not as polymorphic as they might be. Our method will be a CompiledMethod. The new version in the image will be an OverriddenMethod. And the one we read from the database will be a StoreMethodInPackage, mapped to the database. Getting that one is a bit fussy, in that we need to make sure we're asking for the class by name, and the name should be exactly the way it will be in the database.
eachPatchPackage methods do: [:eachMethodDescription |
imageMethod := eachMethodDescription method.
oldOverridden := baseVersion
method: imageMethod selector
forClassNamed: imageMethod mclass instanceBehavior
absoluteName meta: imageMethod mclass isMeta.
newOverridden := Override
selector: imageMethod selector
class: imageMethod mclass
in: currentOverriddenPackage.
Finally, we get the source code for the old and the new versions and check them. If the new base version doesn't exist, then that method was either deleted or moved to another package, and we definitely need to think about our redefinition. And if the source code is different, we also want to think about it. Then there's the question of what to do if there's something to think about. We could easily write that to the file or to the Transcript, but in this case what I've done is open a simple text comparison window on the two. That could get ugly if there are a large number, but is nice to work with for just a few.
(newOverridden isNil
or: [oldOverridden sourceCode ~= newOverridden sourceCode])
ifTrue: [
| view |
view := SideBySideTextComparisonView new
leftText: oldOverridden sourceCode
rightText: newOverridden sourceCode.
ScheduledWindow new component: view; openWithExtent: 800@600]]].
This would need to be tweaked for particular environments, but seems like it might be a good start towards making that sort of migration a bit easier. And if I can figure out how to get the syntax highlighting on this blog working, it might get prettier to read here.
Anyway, let's say that this was built in VisualWorks 7.6, but now you're upgrading to 7.8, and you want to check which of these methods no longer apply. That can be complicated in general, but a basic first check is if the method we were redefining changed between VisualWorks 7.6 and 7.8. That's a bit tedious to check manually, but we can script it.
Of course, this depends on how we've organized things. The original question came up in the context of having grouped redefinitions by the package that contained the thing being redefined. So, if we redefined a method in "Assets", we'd create a package "Assets patches" and put the redefinition there.
So here's an example of a workspace script for finding these. It assumes that we've published the old version of the base into our local database, and that we've loaded our code into a new image.
mySession := StoreLoginFactory currentStoreSession.
versionToUpgradeFrom := '7.7 '.
patchPackages := Store.Registry allPackages select: [:each |
each name like: '% patches'].
So, first we need to get a StoreGlorp session which we'll use to do our queries, and we define a variable for how we'll find the old versions. We'll also need the list of patch packages that we have in the image. I used the #like: method to do the matching, which does it in SQL style, rather than the more traditional matches:, or even regular expressions, but they'd all work.
patchPackages do: [:eachPatchPackage |
basePackageName := eachPatchPackage name readStream upTo: Character space.
currentOverriddenPackage := Store.Registry packageNamed: basePackageName.
Now we start a loop over each of our patch packages. The first thing we need to know is what the basic package is, which we do based on the simple naming convention described above.
baseQuery := Query readOneOf: StorePackage where: [:each |
(each name = basePackageName &
(each version like: '%', versionToUpgradeFrom, '%')].
baseQuery orderBy: [:each | each timestamp descending].
baseVersion := mySession execute: baseQuery.
Now we do a query to find the appropriate old version. So this involves a Glorp query to find packages by that name, whose version string matches the variable we set at the beginning. If there are multiples, we take only the latest, by sorting them in descending order and just taking the first one. I'm dealing with the Cincom internal repository, so there will be lots and lots of versions of base code. In a project repository there are probably only a few, making it fairly simple to find this.
Once we've done that, we'll loop over the methods in the patch package, and get the three different versions of the method: ours, the new base image version, and the old base image version. These will be three different kinds of objects, and the APIs for manipulating them and getting them are, well, let's just say not as polymorphic as they might be. Our method will be a CompiledMethod. The new version in the image will be an OverriddenMethod. And the one we read from the database will be a StoreMethodInPackage, mapped to the database. Getting that one is a bit fussy, in that we need to make sure we're asking for the class by name, and the name should be exactly the way it will be in the database.
eachPatchPackage methods do: [:eachMethodDescription |
imageMethod := eachMethodDescription method.
oldOverridden := baseVersion
method: imageMethod selector
forClassNamed: imageMethod mclass instanceBehavior
absoluteName meta: imageMethod mclass isMeta.
newOverridden := Override
selector: imageMethod selector
class: imageMethod mclass
in: currentOverriddenPackage.
Finally, we get the source code for the old and the new versions and check them. If the new base version doesn't exist, then that method was either deleted or moved to another package, and we definitely need to think about our redefinition. And if the source code is different, we also want to think about it. Then there's the question of what to do if there's something to think about. We could easily write that to the file or to the Transcript, but in this case what I've done is open a simple text comparison window on the two. That could get ugly if there are a large number, but is nice to work with for just a few.
(newOverridden isNil
or: [oldOverridden sourceCode ~= newOverridden sourceCode])
ifTrue: [
| view |
view := SideBySideTextComparisonView new
leftText: oldOverridden sourceCode
rightText: newOverridden sourceCode.
ScheduledWindow new component: view; openWithExtent: 800@600]]].
This would need to be tweaked for particular environments, but seems like it might be a good start towards making that sort of migration a bit easier. And if I can figure out how to get the syntax highlighting on this blog working, it might get prettier to read here.
Wednesday, May 18, 2011
Looking at the public repository easily
A quick note. Today someone was wishing there was a way to see what was in the public repository easily without having to fire up Smalltalk. And there is. There's an index of it that's google searchable. It's fairly rough, and it omits things that it thinks aren't interesting, including packages without comments. But it's still quite useful.
Monday, May 16, 2011
Fun with Mail (part 1)
A description of switching around with email clients, and writing some IMAP code in Smalltalk.
I've been a Eudora user for many years. I started using it back when I was a student, and mostly stuck with it, with a bit of time off using VM in Emacs as my primary client. I never liked Outlook, but was able to so I just kept using Eudora in preference to that for corporate email. One of the very nice things in the later versions of Eudora was the search system. The UI was slightly awkward, but it was fast and gave good results. And I keep a lot of email, so that's important to me. I'm at about 5GB right now, with some of it going as far back as 1990.
Unfortunately, Eudora has been abandonware for quite a while now. I resisted switching for a long time, but there were starting to be enough problems that it had to happen.
I wasn't sure what client I'd end up with. The various webmail solutions seemed to be out because none of them seemed to have ways for me to upload huge archives of old mail. So I figured I'd end up with an actual mail client on my machine. In order to be able to try out different ones, and also because I thought it'd be fun, I set up a small Linux server with IMAP (using dovecot) and had it fetch the messages from my various accounts to one location.
Getting the email over was a bit of an adventure. It was possible, in Eudora, to set up an account on the IMAP server, and then to drag folders over onto it, copying their contents. But that was very slow, and tended to crash. Eventually I found a Mac called Eudora Mailbox Cleaner that can also convert the format, and I was able to use that to import both my old Eudora mail, and some other stuff that wasn't in Eudora, but was saved in more or less Unix mailbox format. It crashed a couple of times part-way through, but with a bit of babysitting I got it all converted into local folders in Thunderbird. Thunderbird would let you move stuff from local folders into IMAP as well, and it wasn't quite as slow and didn't crash quite as often.
So ultimately I had my mail converted, and was able to try out some clients. Apple's mail was the main other one I tried, but it missing features I wanted, so I ended up mostly using Thunderbird, particularly with the QuickFolders and Archive This extensions, which were helpful for quick filing and finding folders. But Thunderbird had some problems. The search was not particularly fast, at least not on the kind of volume of mail that I had. Worse, the filters didn't seem to work reliably. I don't know if that was just an issue with Thunderbird and IMAP, but the spam filtering wasn't moving things out of the Inbox reliably, and mailing list messages weren't reliably going into the right folders.
I decided to take this as an opportunity to write some Smalltalk code, and made myself a little cron job that would run a Smalltalk program to filter the messages. Along the way, I learned a bit about IMAP and the VisualWorks libraries for using them. So in part 2, I'll talk about some of the code that I ended up with.
Wednesday, May 4, 2011
New blog address
Hello World,
I'm Alan Knight, the engineering manager for Cincom Smalltalk and principal developer of the GLORP object-relational mapping framework for Smalltalk. I did have a blog on the Cincom Smalltalk site for a while, which I posted to, um, occasionally, but without Jim Robertson around to maintain that infrastructure I've decided to use a more conventional service. And since Blogger seems to have at least the potential for automatic Smalltalk syntax highlighting that seemed like as good a reason as any to choose it.
Once, many years ago, when I was a columnist for The Smalltalk Report someone asked me at a conference if I was "the" Alan Knight. And I really wasn't sure if I was or not. But now I know - I'm not "the" Alan Knight, but rather this is - LEGEND! I, on the other hand, am just a computer guy, live in Ottawa, play and referee soccer, and am married to Kirsten Carlson, a flute player and teacher.
Before I worked at Cincom I was with The Object People, a training and consulting company in Ottawa, Canada. They ended up best known for developing TOPLink first in Smalltalk and then in Java, and for a while I was the chief architect. TOPLink eventually ended up owned by Oracle, but TOPLink/Smalltalk is still interesting as an early example of O/R mapping software that is clearly prior art to any number of patents. For that matter, so is the ObjectLens software (which patent lawyers seem to mostly know by its very early and short-lived name of Infobase) that my employer Cincom inherited from ObjectShare/ParcPlace-Digitalk/Parcplace and still sell today.
In this space I hope to blog about Smalltalk and programming in general, Cincom Smalltalk in particular, Glorp and other O/R mapping issues, and anything else that seems interesting enough to post. And hopefully post things with more actual content and not quite so many links. Wish me luck.
I'm Alan Knight, the engineering manager for Cincom Smalltalk and principal developer of the GLORP object-relational mapping framework for Smalltalk. I did have a blog on the Cincom Smalltalk site for a while, which I posted to, um, occasionally, but without Jim Robertson around to maintain that infrastructure I've decided to use a more conventional service. And since Blogger seems to have at least the potential for automatic Smalltalk syntax highlighting that seemed like as good a reason as any to choose it.
Once, many years ago, when I was a columnist for The Smalltalk Report someone asked me at a conference if I was "the" Alan Knight. And I really wasn't sure if I was or not. But now I know - I'm not "the" Alan Knight, but rather this is - LEGEND! I, on the other hand, am just a computer guy, live in Ottawa, play and referee soccer, and am married to Kirsten Carlson, a flute player and teacher.
Before I worked at Cincom I was with The Object People, a training and consulting company in Ottawa, Canada. They ended up best known for developing TOPLink first in Smalltalk and then in Java, and for a while I was the chief architect. TOPLink eventually ended up owned by Oracle, but TOPLink/Smalltalk is still interesting as an early example of O/R mapping software that is clearly prior art to any number of patents. For that matter, so is the ObjectLens software (which patent lawyers seem to mostly know by its very early and short-lived name of Infobase) that my employer Cincom inherited from ObjectShare/ParcPlace-Digitalk/Parcplace and still sell today.
In this space I hope to blog about Smalltalk and programming in general, Cincom Smalltalk in particular, Glorp and other O/R mapping issues, and anything else that seems interesting enough to post. And hopefully post things with more actual content and not quite so many links. Wish me luck.
Subscribe to:
Posts (Atom)