XCMTDMW: Characteristics of an XML CMS, Part 2

In part 1 of this discussion of CMS characteristics I outlined those features that apply to the management of storage objects. Storage object management is a prerequisite for any more sophisticated semantic management. It is also generic in the sense that the storage object manager (the repository) does not, by itself, know or care what is inside the storage objects. It just holds the data, manages version relationships, manages arbitrary metadata for storage objects, and manages other arbitrary dependency relationships among storage objects. We also discovered that the Subversion system does all of these except the last (and it might be made to do that one too--I haven't had the time to look).

This means that for a zero up-front software cost and a modest amount of custom integration effort (a few person weeks to a few person months) you can implement an adequate and quite useable XML-aware content management system. It won't do everything you might like it to do but it will be freaky cheap compared to any commercial system you might buy and it will be relatively easy to replace or refine as needed.

With a system of this sort, any XML awareness would be in the boundary code, that is, the code that takes XML and puts it into the repository, examining the XML data to extract any metadata to be applied to storage objects, to instantiate dependencies, and so on. This is the integration effort and its scope is entirely a function of the local requirements. In the Woodward case it was minimal because we only cared about part number metadata and nothing else. In your case you might care about a lot of stuff or have much more complex links or whatever. But since it's integration you're driving you can choose how quickly and how complete you need it to be. You can get a minimally-useful system up and going pretty quickly and with a very modest investment. I think this is very important, especially in the context of technical documentation where budgets tend to be modest anyway and all the other aspects of moving to XML-based documentation are probably sucking up a good chunk of it (for example, converting legacy documents into XML, developing your schemas, developing your publishing systems, acquiring and customizing authoring tools, and so on).

I will also observe that it's easy to imagine lots of cool things that an XML-aware content management system could do for you or could enable you to do with your information that in practice you would never do, for the simple reason that they're too complicated for authors or too fragile at scale or too difficult to quality assure or make the system too expensive or whatever. I certainly see this with fine-grained re-use. Enabling re-use at the content management and processing level is not that hard but doing large-scale re-use across large numbers of publications or across large numbers of products and authors exposes a number of difficult human management and communication issues that have no easy solution (because they depend on people communicating with each other and doing their jobs well).

Just saying.

So what are the semantic processing features that an XML-aware content management system needs to provide? They include:

- Maintain knowledge of element-to-element link relationships (cross references, Xincludes, etc.). This should include the information needed to answer "where-used" questions for individual elements.

- Enable quick location of and access to elements via standard addressing methods (XPointer, XPath, XQuery, etc.)

- Enable quick location of and access to elements based on their text content (full-text search)

- Maintain and manage element-level metadata that is not necessarily inherent in the underlying XML source.

- Provide services (APIs) by which software components can use the foregoing

These sets of features could be classified as "XML-aware indexing". The link-awareness component is part of what most people think of as "link management" but is not the whole thing (like storage-object dependency management, it's a necessary but not sufficient prerequisite for full link management).

Each of these features can be implemented in different ways with different levels of completeness. For everything except element-level metadata management, they could all be done through brute force with no persistence at all. Of course this would not scale but it's very cheap to implement (just hack up the XSLT script you need).

The location of elements using XPath or XQuery can be implemented using a more or less brute-force technique of simply locating the documents you want to examine and then applying the xpath or query to them directly with no pre-indexing or anything. [Remember that our storage-object-to-storage-object dependencies let us quickly determine which documents to search given a starting document.]

For example, say you want to resolve a link from element A in DocA to element B in DocB. To do this you will have to take the address uttered by the link (let us assume it is an XPointer) and resolve it to the element itself. As every element exists in exactly one document you first resolve the storage-object-to-storage-object dependency implicit in the link (that is, the LinksTo depenedency from DocA to DocB). That gives you DocB. You then apply the XPointer to DocB to get you to the element.

In practice, this scenario works well regardless of repository scope as long as the individual documents are not too large (say 10's of megabytes). That's because it just doesn't take that much processing time to parse a document of typical size and find a particular element. The only real scale issue would be posed by large numbers of users all doing resolution at the same time. In that case it would probably make sense to have some sort of cache or index, but that's an optimization that can be applied later. The core functionality can be implemented quickly and easy using a little XSLT or DOM programming. In most technical documentation environments individual XML documents tend to be quite small, usually less than 100K bytes, because the information is usually chunked up for ease of authoring or re-use. Any technical documentation environment where individual XML documents are larger than 1 Meg of XML are not doing something right.

For element-level metadata you do need some sort of persistent store or index, but it doesn't have to be complicated. The minimum you need is a four-column table: attribute name, attribute value, containing document, location of element. Totally brain dead but it would work. Of course you'd immediately realize that you should probably have one table per attribute and if any attributes represent pointers to other attributes you'll need to capture those relations but that's all workaday stuff for anyone with a little RDBMS experience and again represents optimizations that aren't strictly necessary just to make the thing work.

With these features in place you now have all the core information management features needed to implement arbitrarily sophisticated business logic over your body of information. You can implement workflow (using dependencies and metadata to track status, ownership, and so on), you can implement publishing (using dependencies and link information to feed your composition or HTML generation process). You can implement author support features like general search and retrieval over the repository, integration with authoring systems, convenience UIs for creating links or other complex structures, can build more specialized indexes to satisfy different business processes or authoring tasks, and so on.

Let's take a moment to catch our breath and talk about the overall shape of the system as a collection of components.

If you've been able to hold everything I've described so far in your head (and if you have, I'm impressed, because this is not an optimal medium for presenting these complex concepts) then you should be starting to see that the system as I've described it breaks down into three main layers:

Layer 1: Storage object management. This is the layer that manages storage objects as opaque objects. There is no XML awareness at this level in the system: storage objects are just sequences of bytes (e.g., files) or collections of pointers to other storage objects (e.g., directories)

Layer 2: Generic semantic management. This is the layer that provides generic facilities for managing information about the XML as XML (the first level of XML awareness). These facilities are generic because they can be applied to any XML data regardless of schema or application semantics.

Layer 3: Business-Process-Specific semantic management. This is the layer that provides schema-specific or business-process-specific functionality, the second level of XML awareness. This is the layer that binds knowledge of the application-level semantics of a given bit of markup to the generic features needed to make implementation and use of those semantics possible or practical. For example, a simple workflow mechanism that uses element and storage-object metadata to track the progress of a specific XML element through the development process is going to bind the specific development process (the series of stages, the business rules for when you can move from one stage to another, what happens when you do or don't move, etc.) to the generic metadata facilities to both set and examine the relevant metadata values.

When most people start thinking about their need for a content management system they of course focus on the features that will be exposed to users or that reflect the business-process-specific processing of their documents. If you think about these features, whatever they are, in terms of the foregoing layers, it should become clear that all of them can be modeled as Layer-3 components that use the generic facilities in Layer 2 and Layer 1.

It should also be clear that each layer can itself be organized into peer components that can be (and usually should be) separable. For example, in Layer 2 you can have a full-text index component that is separate from the link index component that is separate from the element metadata component, as least as exposed via APIs. Under the covers all these components might be implemented using the same underlying software component or they might not. Dilligent engineering suggests that even if they are implemented using the same component there should be a level of abstraction between the API exposed to Layer 3 and the underlying implementation so you can change the implementation decision without having to change your APIs or the implementation of Layer 3.

Note also that the code in Layer 3 is, by definition, use-case-specific. That means two things: it's unlikely you will ever find an off-the-shelf system that provides more than part of the Layer 3 functionality you need [although see my comments about DITA at the end of this post] and it's where the bulk of your investment in building the system will go.

This is very important. To the degree that the code that implements Layer 3 is loosely coupled to layers 1 and 2, your investment in that code is protected. To the degree that it is tightly bound, your investment is at risk.

It should also be clear that one implementation choice that cannot in any way reflect this model is the element-decomposition approach, for the simple reason that it combines the storage layer, the element metadata layer, and the other Layer 2 components into a single, monolithic component (the object store containing the objectized elemements). While it might provide some generic features for element metadata and so on, as I discussed in an earlier post, if the object database schema directly reflects the XML schema, then it is also combining Layer 3 components as well. If it the object schema is generic then it is probably not providing any particular optimization for indexing and retrieval relative to other, simpler, more componentized approaches. Thus, no matter how you slice it, it comes up suboptimal at best, counterproductive at worst.

In most businesses that do technical documentation, the basic business processes and information schemas are pretty stable once established. For example, in any type of product documentation, the nature of that documentation is not likely to change drastically because the nature of the product is not likely to change. For products with very long service lives, such as commercial aircraft, this is obvious (commercial aircraft typically have rated service lives of 50 years or more but are sometimes kept in service longer than that--some DC3s built in the 40's are still flying). For products with much shorter service lives, such as mobile phones it's still the case. While the individual manual might only have a life span of two years, the product line might have an expected service life of 10 years or more and the business of mobile phones will continue indefinitely until some technology completely replaces them (which seems unlikely to happen in the next 20 or 30 years).

So if you're going to invest 100,000 or a million Euros supporting your business processes you'd like that investment to serve for at least a decade if not much longer. But you can be sure that any implementation technology you use today will be either obsolete or unavailable or unsupported 10 or 20 years from now. So it seems to me to only be prudent to design your system to minimize the dependency between your highly customized Layer 3 components and the generic services provided by Layers 1 and 2.

If you currently have an XML-aware content management system, you might ask the question: to what degree are my business-process-specific components irrevocably tied to the generic parts of the system? Can I even clearly separate those two?

When Documentum first started rolling out their built-in XML support I was talking to the person who was the primary technical contact for those features. I don't remember the exact reason for the call but I do remember that I made it very clear that, as an integrator, I would never use Documentum's built-in user interface components for accessing the repository as part of an integrated solution for the simple reason that the user interface is a key part the integration investement my client would be making and it needs to be as independent of the underlying repository as much as possible. The user interface reflects the client's business processes and schemas and it costs a lot to develop so why should I tie myself to a single underlying system? She didn't seem to get it. Note that I'm not singling out Documentum here: I would provide the same answer for any repository: unless it can't be avoided, I won't do it.

Of course, some systems give you no choice by not providing an API by which you can bind your own UIs to the underlying system. As far as I'm concerned, those systems (and you know who you are) are not acceptable candidates regardless of any other features they might have.

You have to ask this question: if a repository vendor doesn't have enough confidence in the compelling value of his system as a set of services to let you use your own UI implementations, why not? If a vendor has to use UI lockin to keep you as a customer, that should make you think. Of course, the vendor may not have done this intentionally to create lock-in, it might just have seemed like an appropriate engineering choice. In which case I would say that the engineers or architects of the system don't fully understand the requirements that drive repository design because if they did they would have realized that not exposing a UI connection API is counter productive.

OK, maybe I'm being a little harsh. Maybe my bitterness as an integrator forced to work with such systems is coming through. But if you're going to charge someone millions of dollars for a system I think you have an obligation to think a bit more deeply about their larger requirements and not just about what's best or easiest for your software.

But that's just me, Bitter Integrator Guy.

The Bonnell system I mentioned before was designed as the antithesis of these types of systems. Bonnell was a collection of services and components separated by clear and public APIs designed to enable piecewise implementation and modification of a complete content management system. It was a repository framework or toolkit and satisfied our desires as integrators for the system we would like to be able to integrate on top of. As I've mentioned, for reasons beyond our control, the system is currently lost to the general public. We did use it in a project for one client, which went pretty well in that we were able to build the system per the client's specs. Unfortunately that client happended to be a telecom company that was a victim of the crash of 2000 and the collapse of the telecom market. And then DataChannel sold us to Innodata.

It should be no surprise that XIRUSS has the same basic architecture and design and goals and implementation model.

And to be fair in my ranting and bitterness, it's not just repository vendors but repository customers who don't think things through. I don't know how many projects we've had where a company choose a repository because it had a slick demo UI only to discover that it otherwise couldn't really satisfy their requirements, at least not without a lot of expensive integration work from us. While I don't mind making money doing integration work (that is after all my day job), I would rather do five straightforward projects that result in clean, stable, maintainble, reliable systems at a reasonable cost, rather than one painful project that results in a less-than optimal, hacked, fragile system that never completely satisfies the customer's requirements.

To summarize what we have so far:

- An XML-aware CMS can be modeled as a three-layered system consisting of generic storage object management (Layer 1), generic XML metadata, searching, and retrieval (Layer 2), and business-process- and schema-specific business logic (Layer 3).

- The cost of implementing layers 1 and 2 should be low (ideally free) and does not entail any particular technical difficulties other than those always imposed by large scales (and that would accrue to any similar information management system).

- Most of your investment in deploying an XML-aware content management system will be in Layer 3 where you are binding your specific business processes and application semantics to the generic facilities in layers 1 and 2. The expected service life of this layer is likely to be decades. You want to protect this investment as best you can.

One thing I should mention at this point: for some enterprises there will be no choice but to pony up non-trivial money to build a system that reflects the sophistication of their information and how it is used but there will be a return on that investment in terms of faster development, easier repurposing, better information quality, increased writer productivity, and so on. But for a lot of enterprises they simply can't justify that level of investment but they still need some basic XML content management features.

Fortunately, emerging standards like DITA (and to a lesser degree, DocBook before it) are starting to lower the cost of entry to sophisticated XML-aware content management for the simple reason that DITA provides a standard set of sophisticated linking and re-use semantics that can therefore be implemented as generic services. That is, vendors are now motivated to implement DITA-specific Layer-3 components. This means that if you are using DITA out-of-the-box your initial cost may only be the cost of licensing a given DITA-aware content management system. [I build some generic DITA awareness into XIRUSS-T specifically to demonstrate this point.]

If you are using a specialized DITA-based XML application, if your DITA-aware repository provides the appropriate APIs and whatnot, you should be able to implement the additional layer-3 features you need at minimal extra cost, because you only need to account for the delta, you don't have to build it from scratch.

Of course DITA is still in its formative stages as are commercial systems that support it, so it's not quite that simple in practice (for example, there are still a lot of interoperability issues that reflect ambiguities in the DITA spec itself or implementor misunderstanding [or willful perversion] of specific DITA features). But this picture should improve dramatically over the next few years.

Next up: Boundary Code Spelunking: Import and Export (really, this time I mean it)

XCMTDMW: Characteristics of an XML CMS

I was going to talk about boundary complexity (see previous post) but I realized that before I can do that we need to have a clear understanding of what an XML CMS does (or should do).

For this discussion I will use the term "repository" to mean the overall content management system and the body of data it manages, is in "a repository of XML documents". I like "repository" because it's shorter than "XML CMS".

In addition, I feel that the term "XML CMS" is unnecessarily specialized. In my world, content management is a much more general problem and 90% of what you need to manage XML well applies to everything else too. That's another reason I chafe at over-specialized XML repositories--they really can't manage anything else. But what if you happen to have a 20-year legacy of Framemaker documents and you really don't have a need (or budget or time or patience or stomache) to convert them to XML just so you can get some decent content management features? If things are architected correctly, it shouldn't even really be an issue. But I digress.

There are lots of services and features that a repository could provide and there is of course a fuzzy boundary between features that are core repository features (such as raw data storage) and features that are really business-process-specific data processing that happen to be implemented at the repository level. That is, for a given enterprise or community of interest that needs a repository there will of course be requirements that are core repository requirements (check-in/check-out) and requirements that are unique to that enterprise or community ("on Tuesdays during the session we produce a full report of all ammendments that have gone through third reading. This creates a load spike that the system must handle without significant degration of other operations during that time"). Obviously, in this sort of general discussion I will be focusing on the former but you have to keep the latter in mind as well as you design the general architecture of the system such that the system will be able to react to those sorts of unpredictable and idiosyncratic requirements.

Another way to think about it is that a repository is a layered system, with core functionality, the functionality everybody needs, in the core components, with more and more specialized functionality built on top of those components as layers building outward. This is consistent with the separation-of-concerns/modular implementation view as well: each layer represents a more or less clear boundary between concerns that can be implemented as either a literal boundary (via an API) or as a layer of abstraction within the code. Each layer can in turn be organized into discrete components that further separate concerns within a layer. [This is pretty standard systems engineering but it still amazes me the number of commercial systems that either don't reflect this model at all (that is they are internally monolithic) or that may be implemented this way but don't usefully expose the component boundaries to the outside world. For shame.]

So what are the features that a repository needs to provide?

- Reliable, persistent versioned storage of storage objects (e.g., XML documents, Framemaker files, graphics, HTML pages, program source code, whatever).

By "storage object" I mean "file" or "resource" (in the HTTP sense) or whatever. In XML a syntactic XML document is exactly one storage object (it may make reference to other storage objects, such as external parsed entities).

In XML terms, this means the repository must manage XML documents in the strict syntactic meaning of the term "document" in the XML specification. Note the plural. Many purported XML document management systems in fact manage exactly one document, making them a "document management system". What you usually want is a "documents management system".

By "versioned" I mean "versioned in time" such that for the same logical storage object ("resource" in the terminology I normally use) you can retrieve its data state as it was at any point in time. The simplest useful versioning system only needs to provide a single sequence of versions for a given resource--that is, it does not need to support branching [in my experience, very few people who use versioning systems, even very sophisticated people, actually use branching. It introduces any number of complexities that are best avoided. The only exception to this tends to be in code bases for products where you have to maintain multiple release versions and therefore have no choice but to do branching and merging of code. But in those cases the process is very carefully controlled and managed. In the context of document authoring, with the possible exception of legislative bill drafting systems, I have never seen either a compelling need for branching and merging or a body of authors who could reliably take advantage of it.]

Note that the other meaning of "version", that is different alternative versions of the logical content at a single moment in time is a linking problem (or at least it can be well modeled as a linking problem) and not a core data storage problem. For example, problems surrounding the management of localized documents where you have different language versions of the same base text is either a linking problem or an XML schema design problem, not a core storage problem. Of course some systems that are optimized for managing localized documents do address these through their core storage mechanisms. I feel strongly that that's the wrong way to go about it.

Thus I distinguish "storage object version" from "alternative version". Storage object versioning is purely data storage and is only about versions in time. Alternative versions are semantic concepts orthogonal to storage object versioning.

Note that per my previous post, a system like Subversion or CVS satisfies the requirement for reliable, persistent versioned storage of storage objects. Subversion is a little bit better because it provides support for all Unicode encodings and binary diffing, but CVS will work (as I've demonstrated).

- Distributed access to storage object versions. That is, you have to be able to get the data back out and that access needs to be distributed in the sense that the repository and its clients do not need to be on the same physical machine or running in the same process space. There are many ways to do distribution but in today's world providing an HTTP-based access method (i.e., a Web server) is a minimum requirement. Other technologies, like CORBA, DCOM, proprietary internet protocols (i.e., CVS's protocol, which predates HTTP by 10 years or more) can also be useful. But if it does HTTP you can pretty much play with anybody.

- Provide appropriate storage object access control such that, at a minimum, users who are not authorized to access a given storage object cannot do so. Some scenarious may futher require, for example, that users not even be able to see evidence that particular storage object exists, much less access its content or metadata. Systems like CVS and Subversion (and most operating system file systems) provide sufficient minimum access control features.

- Provide appropriate locking or conflict resolution facilities such that one user cannot destroy another user's changes. This is usually expressed as either write locking or optimistic locking. Both approaches are useful. CVS and Subversion use optimistic locking (checking for conflicts on commit) but a lot of content management systems use write locking (check out, check in). I prefer optimistic locking as a rule but it's not a religious issue, rather one driven by what will work best for the users and the business process.

Up to this point simple off-the-self versioning systems like CVS and Subversion at least minimally satisfy the requirements. [Update: Stan D points out that Subversion provides support for storage object metadata as well., so this statement can be moved down below the next bullet.]

- Management of arbitrary metadata associated with storage objects. Storage objects can of course have lots of metadata associated with them, such as who created it when, where it is in some defined workflow sequence, what larger collection it is a member of, its root element type, the schema that governs it (if it's an XML document), the namespaces it reflects (if it's an XML document), and so on. A repository should provide a generic mechanism for setting and interrogating these metadata values.

Note that here I'm not talking about higher-level metadata, such as descriptors on individual XML elements, except to the degree that that metadata might appropriately characterize the entire storage object.

For example, in a DITA-aware repository you might choose to manage each topic as a separate storage object. In that case it might make sense to capture all the topic-level metadata (keywords, index terms, etc.) and reflect them as storage object metadata. Or it might not. But if you managed multiple topics in a single storage object it probably wouldn't make sense to put all the descriptors from all the topics as metadata on the single storage object that contains them.

Rest assured that I will require a higher-level metadata management facility (otherwise known as an index [see the previous post]).

This is the first requirement that cannot be satisfied by CVS or Subversion directly. There may be other generally available versioning systems that do provide an appropriate metadata feature but I haven't looked for one yet. I did however, implement one (see the initial post in this series). In any case, this sort of metadata management is not hard--at its simplest it's just a sequence of name/value pairs that can be easily captured in a relational database or using generic objects or whatever. It's not a hard problem.

- Management of knowledge of dependency relationships among storage objects, both version-specific and version-independent.

Almost any non-trivial data will, in its semantics, express or imply dependencies with other storage objects. This is certainly true for documentation. For example, a Framemaker file might have a reference to an external graphic, an XML document may be governed by a schema or link to another document or include a graphic in some way or include an external unparsed entity. Regardless of their specific semantics and meaning, these all define dependencies between the storage objects involved. For example, an XML document that is governed by a schema establishes a dependency between the storage object that is the XML document and the storage object that is the root storage object of the schema.

A number of operations that need to be performed require you to know, for a given starting storage object, what other storage objects it depends on. For example, if you want to package up all the storage objects needed in order to render a publication you need to know what the root XML document is and what other files it depends on are: schema, graphics, XIncluded (or conreffed) documents, and so on.

You also need to be able to answer the "where used?" question. If, for example, I want to delete or significantly change a storage object I need to be able to find where it is used (what other storage objects depend on it?) in order to, for example, determine if there are no dependencies or, if there are, inform the owners of those other storage objects of the impending (or performed) change.

At its simplest, the facility can simply capture "A depends on B". In practice, it is usually useful to be able to type the dependency so that processes can make informed decisions based on different dependency types. For example, in XML, you can usefully process a document even if you don't have access to any graphics it might point to but you cannot process it if you don't have access to any external parsed entities it includes. Therefore, an export processor needs to know if a given dependency is a "parsed entity reference" dependency or a "graphic inclusion reference". For any well-defined data format there will be a set of defined dependency types and you should expect a format-aware repository to have those types built in. In addition, there will be dependency types specific to a given enterprise's or community's data. Therefore, you should expect the repository to allow you to define new types.

In addition, these dependencies should be first-class objects with their own metadata (of which the relationship type is one instance, possibly priveledged).

The repository should provide facilities by which you can examine the dependencies, get their types, metadata, traverse them in either direction ("where used") and so forth.

By "version dependent and version independent" I mean that dependencies must be able to be expressed in terms of exact version-to-version relationships ("this is a link from version 1.3 of object A to version 2.6 of object B") or expressed in terms of rules, that when applied at a particular point in time, will resolve to one version or another "this is a link from object A to the current (latest) version of object B" or "this is a link from object A to the version of object B whose metadata properties X, Y, and Z match the values S, T, and U". This is where things start to get a little tricky, but not impossibly so. But systems that only provide direct version-to-version relationships or only provide version-to-latest relationships are not, in the general case, useful, because they are much too limiting for most non-trivial use cases. There are some documentation environments where document life cycle is so short that older versions are of no interest but those environments are rare. In the typical software or consumer product documentation case there can be quite complex version-to-version dependencies and the repository needs to support those complexities.

This facility is not, by itself, link management as most people think of it. It is, however, a prerequisite for complete link management. By "link management" I mean mananging the knowledge of and resolution of individual links within storage objects.

For example, say XML document A has an XInclude of some element from document B and it includes that element about 100 times (it's a warning used in every subtask in every procedure in the engine manual for the X104 personal spacecraft). That's 100 individual links of type "XInclude" but it can be captured as a single storage-object-to-storage-object dependency (object A Xincludes from Object B).

It's the core repository's job to know that A depends on B for a given reason. It's the link manager's job to know about all 100 XIncludes from A to B and to provide the services for resolving or modifying those XIncludes as needed. That problem is one that I will talk about in much more detail before too long, as it's the problem (versioned hyperdocument management) that has most engaged me for the last 15 years or so.

But you have to walk before you can run and not everybody does a lot of linking (or can afford a full link management system). So just having basic dependency knowledge is very important and goes a long way.

Finally, storage-object-to-storage-object dependencies need not be inherent in the storage objects. They might reflect dependencies created as a side effect of some business process. For example, you might have documentation for two related products that will be released to market at the same time. This imposes as schedule dependency between the the publications for those products even if there are no other relationships among them. This scheduling dependency could be captured in the repository as a dependency between the root documents of the publications involved. Given this dependency you could, for example, create a report that looks at your publication, determines what its publication date is (which could be captured as metadata on the storage object), look for any "published with" dependencies, find those publication roots, then traverse the tree of dependencies from those roots to see if any subcomponents are not far enough along in their development workflows. I.e., if all the components should be in the "approved" state (again, a metadata value on the storage objects) but some are still in the "under review" state, you probably want to track down the owners (another storage object metadata value) and ask what's up and when do they expect to get those things approved?

You get the idea.

By capturing just dependencies at the storage object level you are again separating concerns (storage-object-specific metadata from semantic object (element-level) metadata) and enabling a large set of useful functions without the overhead and expense of a full link management system.

Everyone needs to know about some set of storage-object-to-storage-object dependencies. Not everyone needs link management (and even when they do they can limp along without it quite effectively for a long time--buy me a beer and I'll tell you about my first non-trivial link management exercise that was implemented entirely using line-oriented text editors and some simple markup conventions).

At this point we have the minimum features needed for complete version-aware, dependency-aware storage object management. These features are a prerequisite for more sophisticated semantic management that is applied to the content within the storage objects.

Please note a few things about the foregoing set of requirements:

- Most of it is satisfied by off-the-shelf versioning systems like CVS and Subversion

- The storage object metadata capture and access is not a hard problem and can implemented in quite simple ways. The simple CVS and Gadfly approach I took at Woodward would work just fine for storage object metadata as well in a lot of use environments.

- For dependencies that are inherent in the semantic or syntactic content of the storage objects, is is not absolutely necessary to hold those persistently. That is, any persistent storage of knowledge of those relationships is just optimization of what could be done by brute force. That is, in theory, to answer any dependency question you could just examine all the storage objects to see what they point to. Of course this won't scale (but it might well perform for smaller data sets) but the point is you don't absolutely have to start with a sophisticated database to hold the dependency information.

- All the features I've talked about are not in any way XML specific. That is, a system with these characteristics can be used to manage any type of storage objects whatsover.

- I still haven't talked about how you get the metadata or capture the dependency knowledge. That's all boundary stuff and I'll get to it soon.

With a system that does just the things outlined here you've gone a long way toward satisfying your XML (or equivalent structured data) management requirements. Of course I haven't said anything about, for example, full-text search and retrieval, but if you've been paying attention you can probably predict what I'll say about how to satisfy that requirement....

Next up: semantic processing features of a repository