A caterpillar builds a chrysalis in order
to shield it as it undergoes the metamorphosis that will tranform it
into a butterfly.

METAMORPHOSIS helps to resolve the forces that arise in evolving systems by providing a means by which a system's behavior can be augmented without changing its primary interface. These forces are described in the SOFTWARE TECTONICS pattern. Providing the means by which a system may dynamically extend itself also resolves the gradual evolution criteria of that pattern.

v v v

A mutable system is one in which the behavior of existing parts of the system can be changed. This is in contrast to an extensible system, which allows new elements to be added to a system, but does not allow the modification of existing parts.

One example of a mutable system is one where extensions to existing tools can be incorporated in the menus for those tools. One does not create a new tool. Instead one adds capabilities to the existing tool. In order to do this, the tool's menus must be mutable.

A mutable language allows the behavior of existing constructs to be changed. Debugging tools can make use of such facilities. Because of the potential for circularity, programmers must exercise extreme care when they change the way existing language constructs behave. Typical programming environments implement debugging facilities in an ad-hoc fashion beneath the language level. Hence, programmers cannot access and augment these facilities.

An extensible system allows users to add features. A mutable system allows users to change existing features.

Consider an analogy drawn from the theater world. There are two ways to change what happens on-stage during a play. The most direct way is to change the script. When that is not desirable, there is an alternative: you can change the performers, or the nature of the performance. Most readers will, I trust, concede that a performance of MacBeth might take on a different character with someone like Jerry Lewis rather than Sir Laurence Olivier in the title role, even though the script is unchanged. Similarly, productions of MacBeth in Kabuki-style have a decidedly different character than do traditional productions.

What has this to do with software? If one wants to change the way a program works, one could change its code or data directly. However, sometimes this is not desirable, or even possible. For, instance, assumptions about a subsystems primary interface may pervade the system. When this is the case, one can intervene indirectly, by changing the way that some underlying element of the system on which the application depends works.

One layer that underlies every system is the machinery associated with the programming language in which it is written. Were one interested in changing all the formatted I/O in a C program, one can either change every call to printf in one's program, or provide a new printf. However, other linguistic facilities are more difficult to modify. Languages that do provide full, runtime access to such mechanisms are said to be reflective.

This sort of relationship between an application and its substrates is not limited to the language layer. Facilities such as operating system calls, window systems, network services, mathematics libraries, or even other user written layers can all have the sort of relationship with an application such that these kinds of interventions are possible. When this is the case, and when this is the easiest place to get at these facilities, changing the performance rather than changing the script can be an effective way of solving otherwise intractable design problems.

Not only must systems provide a way for programmers to get under the hood, they must provide user-serviceable parts as well. What's more, they should ensure that these parts retain this serviceability in the field. For instance, languages without significant metalevel architectures such as C++ trap objects in binary object files. Once C++ programs are compiled, information pertaining to layout, object and class identity and the like is usually completely discarded. Some more modern systems have found it useful to retain some of this information in vendor-specific browsing and debugging structures. The growing movement to standardize runtime type information (RTTI) in the C++ community is evidence of a genuine need for metainformation. Some of these proposals go so far as to all but establish what are in effect first-class class objects in C++. This movement is driven not by linguistic purists, but by the requirements of real programmers in the field. There is frustration that often programmers must construct mechanisms themselves to reestablish information that the compiler knew in the first place, but threw away. A good sign that a linguistic facility is necessary is that a number of people go to a great deal of trouble to independently invent it. This would seem to be the case with metainformation. One can make the case that the extraordinary success of Visual Basic is due in part to that fact that the language is more reflective than C++.

Languages such as Smalltalk-80 and CLOS have what is in some ways the opposite problem. These languages provide fairly complete metalevel architectures, but usually imprison objects in snapshots or images.

For truly autonomous objects to break the umbilicals that tie them to single processes and images, the traditional division of responsibilities among system components must be refactored.

Autonomous objects must have access to global namespace services, so that they can find the other objects to which they are tied. Autonomous objects that interact with object-bases would benefit from the knowledge that truly first-class objects can glean about their own layouts.

For autonomous objects to function on platforms other than their home platforms, code must be bound to them at runtime. Smalltalk, Self, and Java provide code portability by defining code in terms of byte codes for a virtual machine. Dynamic translation of the sort found in Smalltalk-80 [Deutsch & Schiffman 1984] and Self [Chambers et al. 1989] might be factored into the operating system, or provided as a runtime service, so that native code can be made available to any object on demand.

A vital factor in realizing autonomous objects is genuine first-classness. They will require a system-wide object model with a fully realized metalevel architecture, and not one based exclusively on v-tables.

A good sign that a programming language feature is needed is when a lot of people go to a great deal of effort to build it themselves atop or beside existing languages. There is abundant evidence that first-class, dynamic, metalevel objects are such a feature.

Most programming systems that support graphical user interfaces now support mapped, dynamic data structures called resources. These are usually cast at about the level of C structs. However, since they often are created and manipulated by using resource editing tools, they must usually employ their own conventions for manipulating what is, in effect, metainformation. Some realizations add unique symbolic objects that resemble Lisp atoms, and powerful, dynamic evaluators to allow runtime resource expressions to be processed. Often, elaborate schemes must be devised to set up runtime correspondences between names for routines in the resource namespace, and the same routines as they were known to the compiler and linker. The irony here is that all the facilities that have to be created in an ad-hoc, implementation specific fashion by the architects of these systems are essentially duplicating things that the original programming system knew how to do as well. In fact, the information that the programmer must redundantly recreate for these systems is often information that the compiler knew in the first place, but compiled away.

For similar reasons, many applications are adding, simple interpreted macro languages to their applications, while building these applications in a more powerful object-oriented language that by virtue of the system's architecture cannot be reused, and is hence out of reach.

Consider the difficulty one encounters in trying to construct a query for an object in an object-oriented database for an object one has not encountered before using C++. The only way to address this issue is to once again construct a dynamic language, with its own metalevel data structures, atop of C++.

The potential for Balkanization can be seen most acutely in current efforts to define object brokering services and object models. In many respects, these seem to be architectural end-runs around the linguistic community. This evasiveness is justified, given the degree to which mainstream language designers have avoided the issues these efforts are trying to address. In the end, it will be objects themselves, and not languages that will be the central focus of system design efforts.

When both applications and their substrates are built from objects, they can evolve together as requirements evolve, or as specific users present specific needs. When the runtime mechanics of a system are thus accessible, that system can integrate better into a changing community of applications and services than a system that is set in concrete.

Metamorphosis is a powerful technique. However, many conventional systems will place a premium on support for these facilities, if they provide them at all. When they are not provided, runtime support for them will have to be provided by the user. Therefore, this technique should not be employed cavalierly. When objects can be redesigned so that their layouts are knowable in advance, such a redesign should be considered, and weighed against any loss of potential generality. In those cases where applications simply cannot be given prior knowledge of certain kinds of objects, metamorphosis can be the only viable solution.

v v v

In order to stay competitive, Caterpillar has noticed that they must be able to quickly evolve to new ways of doing business. They need ways to be able to make new decisions quickly and change the way they do business according to those decisions. In order to do this they need to be able to dynamically choose their variables and business logic and then query from their data sources accordingly.

VisualWorks by ParcPlace has provided a framework for creating static SQL database queries. The framework allows for the developer to graphically create SQL queries that map to Oracle and Sysbase Databases. These queries then gets converted into a Smalltalk method that can be called upon when desired. Smalltalk objects can also be passed into the generated methods and conversions and comparisons are supported by the framework. This framework can also query the database for the current data model the developer is interested in and then create objects to map to the desired tables within the database. It is also easy to extend the framework to add undeveloped database functions or extend the mapping to other Database vendors.

Basically what happens is that the generated methods are parsed and SQL code is generated that includes the joins, projections, select-where, group-by, and order-by clauses. The generated SQL code is then packed up and shipped across the network via SQLNET. The returned database values are converted into objects that describe the attributes for each table within the database. These returned values can then be displayed, evaluated, or processed dynamically just like any Smalltalk object.

The problem arises when one wants to dynamically change or create SQL queries during run time. Since the SQL framework supplied by ParcPlace only provides ways to pre-defined static queries we built a framework of SQL Query objects that allows for the creation of dynamic SQL queries through the use of Smalltalk expressions. For example, the experienced Smalltalk Programmer might desire to be able to write code very similar that portrayed in

example1
    "Perform a natural join on Models and ProductFamily
    projecting model number and family description.	
    Then return the values."
    | models productFamily tmpQuery|
    models := TableQuery tableFor: #Models. 
    productFamily := TableQuery tableFor: #ProductFamilies.
    tmpQuery := models naturalJoin: productFamily.
    tmpQuery := tmpQuery orderBy: (models fieldFor: 'modelNumber')).
    tmpQuery := tmpQuery project:(models fieldFor: 'modelNumber'),
        (productFamily fieldFor: 'familyDescription').
    ^tmpQuery values

Also, it might be nice to take a query that is formed as above and then "wrap" some new constraints on the query such as select only those in the above query for the current month. Adding new constraints to the queries such as this might only be realized during run-time, thus making the static creation of queries insufficient.

Our solution to allowing for the dynamic creation of SQL objects was to define GroupQuery, OrderQuery, ProjectQuery, SelectionQuery, and TableQuery classes which are all subclasses of the QueryObject abstract class. We also created QueryExpression objects that allow for the developer to build query expressions as in the example above.

The Query objects know how to respond to the appropriate message to build the queries and wrap constraints to themselves during run-time. The design pattern that fits here is the INTERPRETER pattern from Design Patterns [Gamma et al., 1995].

We were able to reuse all of the code from VisualWorks original framework of parsing a method into SQL and submitting the SQL across the net and then creating objects representing the desired values returned from the database.

Our dynamic SQL framework has allowed for late binding of constraints to the SQL objects by allowing the developer to build a parser for developing queries and "wrap" additional constraints to SQL objects as the application runs.

This example demonstrates the principles of reuse, extensions, and modifications. Reuse by the simple approach of blindly reusing the framework for generating the SQL code and using SQLNET to get the desired results and populate objects with them. Extensions are based upon the addition of the "Query Expression" objects for allowing the developer to write queries in Smalltalk like expressions and to also allow for the ease of extending these objects dynamically by wrapping additional constraints before the SQL code is generated. Modifications can be done to the existing framework by adding in behaviors for additional desired SQL functionality or the framework can also be extended by adding database drivers not supported in the default image provided by ParcPlace.

Smalltalk allows dynamic translation of the sort done in our SQL Object example. Our example parses a field of possible queries and simply wraps additional constraints, that the end user can create, around SQL Objects that are passed around. The dynamic translation is done through the simple parsing of strings that build the SQL Objects with the desired constraints and wraps them around the original SQL Object.