Delivering HTML from DITA in The Face of Reuse

Delivering HTML from DITA in The Face of Reuse of Topics in a Single Publication

In DITA you can use the same topic multiple times from the same map. For example, the same user interface component might be used by several different parts of a program and you want to include that topic in the descriptions for each of those parts. Or you might have a common installation topic that uses conditional content to apply to different operating systems.

In monolithic output formats like PDF and EPUB this reuse does not present any particular practical problems: because the rendered publication is a single linear flow of content, each use simply occurs in the flow and reflects whatever conditions are in effect at that point in the publication.

However, with multi-file output formats like HTML, there are several practical problems.

The most obvious problem is the "one result file or multiple result files?" question: When a topic is used multiple times, do you want to have just one result HTML file or do you want one result HTML for each use? The DITA Open Toolkit, through version 1.8.5, only generates a single result HTML file unless the map author specifies @copy-to on the topicrefs to the topic. (The @copy-to attribute specifies the effective source path and filename for the referenced topic so that the processor then treats that use of the topic as though it was a copy of the real topic with the specified filename.)

The "every page is page one" philosophy says you should have just one result HTML file for a given topic. Likewise, searching is usually more effective if there is just one HTML file, otherwise you end up getting multiple search results for the same content, which confuses users and makes it hard to know which version to use (and may throw off search ranking algorithms that take the number of copies of a file into account in some way).

On the other hand, if a user comes to a topic that is used in multiple places in the publication, how do they know which use they care about in their current access session?

For the re-used installation topic example, if it reflects multiple operating systems and there is only one copy, you would appear to be required to show all the operating system versions and use flagging to distinguish them. On the other hand, if you have one HTML file for each copy of the topic, each HTML file only reflecting a single operating system, a search on installation will find all the copies, making it hard for the user to choose the right one.

DITA 1.3 adds an important new feature, key scopes, which allows keys to have different values in different parts of the same map. This lets you reuse the same topic in different contexts and have content references, hyperlinks, and key-defined text strings resolve to different values in the different use contexts.

For the installation example, you could have three key scopes, one for each of the operating systems Windows, OSX, and Linux.

DITA 1.3 also adds the new branch filtering feature. With branch filtering you can apply different filtering conditions to branches within a single map. This lets you use the same topic in different parts of the map with different filtering conditions applied.

For the installation topic you can now have a single topic as authored with content that is conditional to each operating system and then have only the operating system for the branch filtered in.

It should be obvious that this must result in either three result HTML files, one reflecting each different set of filtering conditions, or a single HTML file constructed so that the browser can do the filtering dynamically, such as through different CSS files for the different filtering conditions or through Javascript or some combination.

This all means that, with DITA 1.3, DITA-to-HTML processors must handle multiple uses of the same topic in a sophisticated way. The OT 1.x approach of generating a single HTML result will not work. Likewise, the OT 2.x approach of always generating a new result file works (in that it ensures a correct result) but does not necessarily satisfy requirements for minimizing content duplication in the result.

So basically there is a fundamental conflict between ensuring correct content in the generated HTML when branch filtering and key scopes are in effect and satisfying the "every page is page one" philosophy.

If every use of a topic results in a new HTML file then searching is impaired but HTML generation is as simple as it can be. 

In the context of the Open Toolkit, branch filtering (and @copy-to) is applied to create new intermediate topic files and then those intermediate topics are filtered to produce another set of intermediate topics which are then the input to the normal HTML generation process. All the data processing complexity is in the preprocessing.

In order to produce a single result HTML file the processor has to determine, for the conditional content in a given topic, which content would be filtered out of all uses and which content would be filtered in for any use context and produce an intermediate topic that omits the globally-excluded elements but retains the elements included in any use. It also has to somehow record each use and how it relates to the included conditional elements so that the final HTML generation stage can retain that information in the generated HTML so that CSS or Javascript can act on it. For example, the processor might translate each unique set of filtering conditions into a single value included in the conditional element's @class values or it might embed some JSON data structure that establishes the map context the element was referenced in.

Given this kind of information in the generated HTML it would then be possible to have the browser dynamically show or hide specific elements based on the active conditions selected by the reader. By default the content could be flagged as it would be in the normal flagged output result produced by the normal Open Toolkit flagging preprocessing.

However, with this dynamically-filtered HTML file there's still the problem of the reader knowing what use context they want to view the topic in terms of.

For example, if you do a search and find the installation HTML page and open it you then have to decide which operating system you want to view it in terms of. 

How is this decision presented to the reader? 

How does the Web site track access in order to establish this use context automatically when it can?

And of course the situation could be much more complicated: there could be a number of conditions against which the content is filtered, e.g., operating system, hardware platform, region, product features active, etc.

I think this is a delivery challenge that the DITA community needs to address generally by both establishing best practices around content authoring and delivery, by implementing the DITA-to-HTML processing that supports generating these more-sophisticated HTML pages, and by implementing general CSS and Javascript libraries for use in DITA-based Web sites.

Rethinking DITA "Custom" Attributes

I had a bit of an epiphany today as regards the definition of "custom" attributes and their DITA conformance or lack thereof.

To date I think I have been consistent in saying "You can't define attributes for individual elements in a way that is a conforming DITA specialization."

But in discussing this today with some colleagues I realized I have been too limited in my thinking. In particular, I now think that @base is an appropriate specialization base for any "custom" attribute and can be combined with constraint modules to limit the appearance of those attributes to specific element types.

If the initial requirement is, for example, "I want the attribute @foo on table" then I think you can make this a conforming @base specialization as follows:

1. Declare a normal attribute domain module for your new attribute, specializing it from @base:

me_fooAttDomain.ent:

<!-- @foo attribute domain module: -->
<!ENTITY % me_fooAtt-d-attribute "foo CDATA #IMPLIED">

<!ENTITY me_fooAtt-d-att "a(base foo)" >
<!-- End of domain module -->

2. Define a constraint module that allows @foo on the elements you want it to be allowed on:

me_fooAttributeConstraint.mod:

<!-- @foo attribute constraint module: -->

<!ENTITY me_footAttTableOnly-constraints
"(topic me_fooAttTableOnly-c)"
>

<!ATTLIST table foo CDATA #IMPLIED >
<!-- End of constraint module. -->

3. In your shell, include the domain module but *do not* add it to the @base overrides (the not including it is a constraint and needs to be declared as such on the @domains attribute, which our separate constraint module will do for us):

<!ENTITY % me_fooAtt-d-dec
      PUBLIC "urn:pubid:example.com:dita:attributes:me_fooAttDomain.ent"
      "me_fooAttDomain.ent"
>%me_fooAtt-d-doc;
...
<!-- Constraint: Not including @foo in base attribute extensions: -->
<!ENTITY % base-attribute-extensions
""
>

4. In your shell, include the constraint module:

<!-- ============================================================= -->
<!-- DOMAIN CONSTRAINT INTEGRATION -->
<!-- ============================================================= -->

<!ENTITY % me_fooAttributeConstraint.def
    PUBLIC "urn:pubid:example.com:dita:constraints:me_fooAttributeConstraint.mod"
           "me_fooAttributeConstraint.mod"
>%me_fooAttributeConstraint.def;

5. In your shell, add the constraint domains contribution to the @domains attribute:

<!ENTITY included-domains
    "&concept-att;
     ...
     &me_footAttTableOnly-constraints;
    "
>

You've now declared a specialization of @base but then constrained it to only be allowed on <table>, as defined in the separate constraint module.

The 1.3 specification says this about the @base attribute:

"A generic attribute that has no specific purpose. It is intended to act as a base for specialized attributes that have a simple value syntax like the conditional processing attributes (one or more alphanumeric values separated by whitespace), but is not itself a filtering or flagging attribute. The attribute takes a space-delimited set of values. However, when acting as a container for generalized attributes, the content model will be more complex; see Attribute generalization <http://docs.oasis-open.org/dita/v1.2/os/spec/archSpec/attributegeneralize. html> for more details."
-- http://docs.oasis-open.org/dita/v1.2/os/spec/common/select-atts.html#select-atts

As long as your attribute's values will satisfy the requirement that "The attribute takes a space-delimited set of values." then your attributes will be conforming instances of @base.

An interesting implication of using @base is that normal generalization processing will convert @foo="bar" to base="foo(bar)". Which means you can also author @base attributes using that syntax (just as you can author @props using the same syntax, e.g. props="mycondition(myvalue)", which is equivalent to having a @props specialization named "@mycondition" with the value "myvalue"). In particular, this provides a standard way to interchange documents in terms of the OASIS-defined vocabulary without loss of information by generalizing specialized attributes to their @base and @props bases (which is part of the point of attribute specialization in the first place).

The result of all this is that you have an attribute this is a conforming specialization of @base and it's limited, via constraint, to only those places that you want to allow it. Your intent as a grammar designer is clear: I want this attribute to be limited to these element types.

If the constraint module isn't used (but the attribute domain is in the normal way), all your existing documents that have @foo on <table> will continue to be valid, which is one test for the correctness of a constraint. If they are generalized so @foo becomes base="foo(bar)" they will also continue to be valid.

So I reverse my earlier position on the appropriateness and conformance of element-type-specific attributes: the @base attribute combined with the constraint mechanism allows it without having to play any semantic tricks. Interchange is preserved through generalization, everyone is happy, and peace and prosperity reigns across the land.

Some DITA and DocBook History: Common Origins, Very Different Results

The following was originally posted to the DITA Users' Yahoo Group 4 Jan 2016 in the context of a discussion of DITA vs. DocBook. My intent with this bit of history is to show how both DocBook and DITA (through it's ancestor, IBM ID Doc) started development around the same time, more or less from a single meeting. Those of us at IBM took things in one direction, those in the Unix-focused community went in a different direction.

The original post (edited for typos):

If you look at the history of DocBook and DITA both descend from the same time period, the late 80’s, when the technical communication industry in particular (but not exclusively) was trying to figure out how to apply this new SGML technology to their particular information management and document production challenges.

In the case of DocBook the genesis was primarily standardizing Unix man pages. In the case of DITA it was IBM’s attempt to standardize the markup used across the many different divisions and product groups within IBM as well as satisfy the requirements of online delivery of hyperlinked documents, something IBM was doing in the 80’s, long before anyone else outside of hypertext research groups, as far as I know.

There was a meeting the late 80’s, I think 1989, where representatives from the major software and hardware vendors met to discuss ways of standardizing the markup across their documentation, including IBM, HP, Digital Equipment, Group Bull, and one or more Unix vendors (the names escape me now—all except IBM and HP are long gone) in order to have some hope of interchange among them.

The meeting was hosted by Fred Dalrymple of the Open Software Foundation at offices in the Boston area. The work was led by Eve Maler, who was pioneering approaches to DTD design and modularization (she popularized the “pizza” model, adopted by the TEI and also reflected somewhat in DocBook and DITA). I was there with Wayne Wohler representing IBM. (Eve wrote the first book on SGML DTD design: "Developing SGML DTDs: From Text to Model to Markup”, with Jeanne El Andaloussi, who was at Group Bull at the time.)

One of the key things that Eve did was make a table that related the markup vocabularies of each participant to each other vocabulary. There was a row for “paragraph”, a row for “H1”, etc. [I’m sure I don’t have a copy of this table anywhere but it would be interesting to see it now—I have a clear picture of it in my mind but not clear enough to reproduce. But this table was, in many ways, the direct inspiration for my approach to markup design and set the direction of my technical career from then to now.]

What this table made clear was that all these languages had the same basic set of semantic elements but they all used different tag names and had different detailed rules for the content. But they all had some kind of paragraph (, , ,, etc.), headings, tables, lists, etc. (Remember that this was before HTML had been defined by Sir Berners-Lee—he based HTML off of the basic tag set in IBM’s GML Starter Set language, which predated SGML and was in use at Cern at the time Berners-Lee developed HTML.)

What Wayne and I got from this meeting was that A) there was this semantic correspondence and B) we needed a way to allow differences in markup details (tag names, content models) that still allowed interoperation. I realized that one could define a layered architecture with these base types as its foundation and, given a way to map specific element types to their bases, allow variety in the markup naming and content details while allowing interchange and common processing.

Soon after this Wayne and I, along with Don Day, Simcha Gralla, and others, started working on IBM’s SGML replacement for the GML-based BookMaster language, which was used for most of IBM’s documentation and had more than 600 element types, reflecting a very broad range of requirements. BookMaster allowed for very efficient creation of documentation delivered in print and online on 5 different computer platforms using IBM’s BookManager tool, which provided electronic books starting in the mid 80’s. But BookMaster was also big and difficult to change or extend. It suffered the same problems that all large all-encompassing vocabularies suffer: it became a tarball that was difficult to adapt to new requirements. IBM had a committee that considered BookMaster change requests and it worked on a 6-month cycle at best. BookMaster was also based on proprietary IBM composition technology, the Document Composition Facility, which was becoming obsolete with the development of PCs and more modern processing languages and systems.

At this same time Dr. Charles Goldfarb, inventor of GML and SGML, was now working on HyTime, an SGML-based language for hypertext representation. Dr. Goldfarb knew that he couldn’t impose a specific tag set but had to have a way to allow any element type to indicate what kind of HyTime thing it was. His solution was “architectural forms”, a mechanism that relied on specific SGML features to allow elements to declare how they related to the HyTime-defined element types and attributes. It also imposed basically the same content model constraints that DITA specialization imposes, namely that the content models of the derived element types had to be consistent with those of their architectural bases, but HyTime was necessarily less restrictive.

For the SGML BookMaster replacement, which we called IBM ID Document Type (IDDoc), we needed robust linking and we needed something like architectural forms. So we adopted HyTime both for linking and for the architectural forms mechanism. [As a side effect I became involved with Dr. Goldfarb and Dr. Newcomb with the development of the HyTime standard itself. You can ask my wife about “No, Charles.” sometime…]

For IBM ID Doc we defined a base set of elements that reflected the 25-or-so basic semantic elements that Eve had identified at that meeting at the OSF. The rest of the vocabulary was then build up from those base types. This layered architecture allowed the implementation of common processing while allowing local creation of new vocabulary to meet new requirements. Interchange and interoperation were preserved but the overall system became more flexible. This design was completed in about 1993 and implementation and use proceeded and continues to this day, although I understand that use of IDDoc is almost completely replaced by use of DITA within IBM. I left IBM in 1994. Don Day stayed.

Thus DITA reflects one ancestral branch from those early days of SGML application design.

Soon after or at the same time as the OSF meeting, another group of people founded the Davenport group, focused on standardizing Unix MAN pages. I was not directly involved in these meetings so I can’t comment on the details but their work became the basis for DocBook. I did attend one DocBook meeting sometime in the early 90’s (I remember I was still wearing suits per the IBM dress code, so it had to be before ’92 or ’93) and presented my attempt to use architectural forms to formally map DocBook to IDDoc and to try to plant the idea of architectural forms and layered architectures but I was not successful. I think I was seen mostly as a disruptive crank, which I probably was to some degree.

[From Fred Dalrymple’s LinkedIn page, on his time at OSF: "Designed the book style and created formatting tools for all OSF technical publications, published by Prentice-Hall. Led migration of OSF technical publications from legacy format (UNIX nroff/troff) to SGML, including definition of the OSF DTD and development of transformation tools. This work led directly to the creation of DocBook and the Topic Maps standard, ISO/IEC 13250:2000.”]

Don and Michael Priestley can give the history of the development of DITA within IBM after I left at the end of ’93 but the result is apparent today: the DITA we know and love.

In the ensuing decade between 93 and 2003 I became an editor of HyTime 2nd Edition and a founding member of the XML Working Group. I did a lot of client work developing custom SGML and XML vocabularies and tried to apply the same layered architectural model that we had defined at IBM. XML omitted the SGML features required for HyTime’s architectural forms mechanism (which is why DITA has the @class attribute it does) and the publication of the XML standard in 1997 made HyTime instantly obsolete (we published HyTime 2nd Edition in 1996, just in time for it be completely ignored by most people, although its influence is still felt in newer applications, including DITA, XLink, TEI, JATS, and DocBook).

When Don approached me in 2000 or 2001 about this DITA standard thing he was staring I was very eager to participate because I saw it as a potential way to fully realize many of the ideas I’d been working with over the previous decade or so.

[This is the end of the original posting. Obviously there is lots more history here but I think this provides some insight into how DITA and DocBook came to be. Would definitely like to hear the DocBook side of this story as I'm sure I've either omitted important events or misrepresented important aspects.]

Why DITA Requires Topic IDs (And Why Their Values Don't Matter)

The DITA standard requires all topics to have an id= attribute.

Why?

The reason is simple: so you can point to elements within topics. And for no other reason.

Surprised?

Most people seem to assume that topics are required to have IDs so you can point to the topics. And they further seem to assume that topic IDs need to be unique within some fairly wide scope (e.g., within their local topic repository).

But that's not the case at all.

For the case of topics that are the root elements of their containing XML documents and that contain no elements that themselves have IDs, the topic ID isn't needed at all. In this case the topic can be unambiguously addressed by the location of the containing XML document (e.g., "mytopic.xml").

In the case of topics that are not root elements and that are not themselves pointed to and that do not contain any elements with IDs, again the ID is not needed (because nothing points at the topic or its elements).

So why does the DITA standard require topics to have IDs?

It is because topics establish the addressing scope for their direct-descendant non-topic elements.

By the DITA spec, to point to an element that is not a topic you use a two-part pointer: {topicid}/{elementid}.

Without a topic ID it would be impossible to point to a non-topic element.

By requiring all topics to have some ID, it ensures that any non-topic elements with IDs are immediately addressable without the need to also add an ID to their containing topic.

In general by normal DITA practice, non-topic elements are given IDs only when they are intended to be either used by conref or be the target of a cross reference. Both of these tend to be carefully considered decisions driven by editorial and business rules, not arbitrary author decision. Which means you would tend to know, in advance of creation of a given element, that it is a candidate for conref use or xref use, which means you know to give it an ID at the time you create it.

By requiring that topics always have IDs it means that authors don't have to worry about adding IDs to topics just because they also happened to put an ID on an element. [In normal XML practice, elements are addressed directly by ID within their containing document, which means it is sufficient to simply put an ID on the element with no other dependencies. That is not the case in DITA, which defines its own unique syntax for non-topic element addressing.]

Because topic IDs are XML IDs (as opposed to non-topic-element IDs, which are just name tokens and have no special XML-defined rules), any XML editor will both require topics to have IDs and ensure that topic IDs are unique within the scope of their containing document.

If topic IDs were not required, DITA-aware editors would have to have special rules to know to require topic IDs whenever non-topic elements got IDs and it would mean that generic XML editors would not ensure that this important DITA rule was met (topics with elements with IDs must themselves have IDs).

So the DITA spec requires that all topics have IDs.

But the fact that topics must have IDs does not imply that topic IDs need to be either descriptive or unique within any scope wider than the XML documents that contain them.

In the case where every topic is the root of its own document, the topic ID can be the same for every topic. To make this point I have standard practice of using the value "topicid" for the IDs of all my root topics. There is absolutely no need to generate unique topic IDs for document-root topics as a matter of standard practice.

The only other case is ditabase documents.

If you are using ditabase documents, stop.

Sorry.

There are some legitimate uses of ditabase documents, for example, as a first-pass target for data conversions and as a way to hold otherwise unrelated topics that need to be managed as a single unit of storage, such as topics that exist only to hold reusable elements.

[NOTE: Using ditabase simply to allow the mixing of different topic types in a single document during authoring* is the wrong thing to do. You should have already created local shell DTDs and within those shells you can allow whatever topic type mixing is appropriate for your local environment. There is no need to use ditabase in that case and many reasons not to. See my many other posts about why you should always create local shell DTDs as the first step in setting up a production use of DITA.]

In that case, the topic IDs must be unique within the scope of the ditabase element, simply because XML rules demand it. But the IDs need not be unique beyond that scope and they need not be meaningful.

One of the implications of this is that if you always edit topics as individual documents and never have nested topics you never have to think about topic IDs. Your topic document template should already have an ID value and it can be something like "topicid" and there is no reason whatsoever for that ID to ever be changed.

In the case where you do edit topics with nested topics (for example, you're authoring more or less narrative documents or you've designed some topics types that need nested topics to allow a bit of hierarchy where the nested topics would never be meaningful in isolation) then you either have to configure your editor to assign IDs to the nested topics for you (if your document template doesn't already have the subtopics with IDs assigned) or you have to think about it. But even in that case, the IDs can be pretty generic, e.g. "st1", "st2", etc. The IDs in that case still don't need to be unique beyond the scope of the containing document.