Re: [Groff] The future redux

[Walter Alejandro Iglesias]

On Mon, Feb 24, 2014 at 09:59:43PM -0500, Peter Schaffter wrote:
> I'm afraid this will be a long post.  Sorry, but I don't see any way
> around it.
>
> It's ironic and instructive that the thread, "Future direction of
> groff", which became a semantic-vs-presentational debate eerily
> similar to a previous discussion on the same subject, originated
> in the "space-width" thread, which deals with the minutiae of
> presentation.
>
> Re-reading the posts, a number of things become clear.  First,
> groffers on the list value quality typesetting, and there's
> considerable discomfort with moving away from the physical page,
> conceptual or otherwise, to the multi-modal paradigm.  Secondly,
> there's a perceived conflict between backward compatibility and future
> development.  Thirdly, more than one list member expressed fear that
> improving groff will somehow break it.
>
> Lastly, there's Eric's insistence that "if it ain't semantic it's a
> dinosaur," which was greeted with as much resistance this time as
> the last time the subject came up.  Allow me to say here that Eric
> may well be right.  His indispensable contributions to open-source
> development lend considerable weight to any argument he makes.  He's
> been an inspiration to me for years.  I'm disinclined to dismiss out
> of hand any opinion he offers.
>
> If groff is to have a future, if it is to remain vital, it is clear
> these issues need to be looked at.  More than groff is at stake.  Do
> we live in a world where typography itself is becoming an archaic
> craft, like stone masonry?  Is print really dead, as some have
> asserted?  Is there a future for documents that are beautifully
> typeset but only render properly as PDF or PostScript?
>
> And, if the answers are, in order, "no," "no," and "yes," do we
> really need both groff and TeX when TeX is, at present, more
> typographically sophisticated?
>
> I think, too, we need to consider the distinction between
> "documentation" and "a document".  Is the primary purpose of groff
> to produce documentation, which requires an ordered structuring of
> the subject matter, or to produce material of a more fluid nature
> where the presentation (the typography) serves expressive rather
> than structural needs?
>
> * Backward Compatibility
>
> Mike Bianchi summed up the backward compatibility concern best:
>
>  "The fact that I can still format documents I wrote in the 1970s
>   and beyond is valuable to me, and, should any of them ever become
>   classics, possibly to others in the future.  So no, do not break
>   groff by 'modernizing' it."
>
> Backward compatibility has been a mainstay of *roff throughout its
> history, and any effort to improve groff should remain true to that
> goal.  However, ultra-strict adherence to backward compatibility
> must not stand in the way of improvements to existing functionality
> and the quality of final output.
>
> While it's a wonderful convenience to use groff in 2014 to format
> documents from the 1970s, it must be remembered that a) those
> documents exist as plain text, thus changes to groff would have no
> impact on the readability of their content, and b) the formatting
> directives in those documents, equally, is plain text, and thus
> easily understood presentationally and semantically--by humans, if
> not by Eric's "baroque AI baby." :)
>
> In short, if some backward compatibility were to be lost in future
> versions of groff--and frankly, I doubt it would--documents written
> in the 70s would remain as useful and valuable as ever.  The only
> thing that might be lost is the time it would take to write a Perl
> script to parse and update said documents to reflect probably very
> minor changes to groff's behaviour.
>
> Furthermore, let's not forget that under Werner's leadership, groff
> underwent a considerable amount of modernization.  Nothing got
> broken, and the whole groff picture improved splendidly.  I see no
> reason to fear that future development will be any different.
>
> * Fear of modernizing groff
>
> Mike wasn't the only one who expressed fears about the modernization
> of groff.  Walter Alejandro Iglesias wrote:
>
>  "In general terms what in the name of *modernity* developers have
>   being doing with Unix, by Unix I mean the idea behind, what made
>   of it a well designed OS, has being just adapting it to those used
>   to the MSWindows experience.  Marketing.  First that modifications
>   took place in userland, but today, base, init and even the kernel
>   are suffering them."
>
> While I share Walter's feelings, I don't think they apply to the
> kinds of changes we've all been dancing around that need to be
> made to groff.  For example, groff's line-at-a-time approach to
> formatting, if unchanged, will remain an impediment to high quality
> typesetting and ensure groff's demise for anything other than
> writing manpages.  Since the point of implementing page-at-once
> formatting (or, as Werner dreamed, document-at-once) would be
> to improve the quality of typeset output, not to change the
> fundamentals of groff usage, resisting such a change seems like
> misplaced Luddism.
>
> * Groffers love good typography
>
> I scarcely need comment on this.  The most interesting discussions
> on the list, the ones that generate the most responses, deal with
> fussiest of typographic details: how much space to put between
> sentences, the length of dashes, hanging punctuation and margin
> correction for glyphs, letter-spacing versus font expansion, and so
> on.
>
> Why this love is important relates to the future of typography in
> general.  We are concerned about these things because we do not
> accept shoddiness.  Since the advent of DTP, and hastened by the
> rush toward generalized input suitable for multi-modal display,
> there has been, globally, a serious decline in the quality of
> typesetting, by which I mean the balanced, aesthetically-pleasing
> arrangement of words regardless of the display medium.
>
> Good typography lends weight--gravitas--to content, in addition
> to facilitating the act of reading.  It makes words come alive
> and encourages contemplation of the ideas they represent.  Good
> typography says, "These words matter and deserve respect."
>
> It is an ongoing battle, tending the fires of quality in the face of
> widespread dilettantism.  Perhaps I am guilty of adopting a doomed
> Romantic stance (wouldn't be the first time), but I believe groff,
> and the community of users it attracts, are essential to keeping
> the craft of typography alive.  The same is true of TeX and its
> community.  People respond well to quality typesetting, and there's
> no reason why they shouldn't expect it, if not now, at least in the
> future.  If we allow groff to remain as-is, if we cease to look for
> ways to improve the quality of output and refine the tools used to
> create documents, we are doing a disservice to the future since it
> is in our hands to preserve what, of the past, should not be lost.
>
> * The great presentational vs semantic markup debate
>
>>From a typographic standpoint, markup, whether presentational or
> semantic with linked stylesheets, is only as useful as the program or
> device interpreting it.  There's a reason why _The Binbrook Caucus_,
> a novel I put online a few years back, isn't in html: browser
> rendering of type sucks.  I doubt that's going to change any time
> soon.  The novel uses typography to express changes of tense and
> POV, as well as to convey things like verbatim newspaper articles,
> train station announcements, email correspondence, and typewritten
> copy.  For readability, it requires fixed margins and a degree of
> control over justification that's impossible to achieve with, eg,
> xhtml, which is my preferred way of formatting web pages.
>
> For similar reasons, the novel isn't ePubbed or mobied on Kindle.
> Only PDF, page-centric and type-specific, is capable of rendering
> the work according to my intentions.
>
> Eric says:
>
>  "What I don't believe is that there will ever again be enough
>   demand for printer-*only* output to justify markup formats
>   and toolchains that don't also do web and ePub or functional
>   equivalent."
>
> In this he may well be right, but he is speaking of a world where
> precise control over typography no longer plays the role it
> does presently in document design.  A world where "approximately"
> rules--fine for certain types of writing, notably documentation,
> technical papers, blogs, and the like, where, whether on paper,
> a monitor or a smartphone, it is enough for a headings to "look
> like" headings, paragraphs to "look like" paragraphs, nested lists
> to "look like" nested lists, and so on.  And it is certainly in the
> best interests of future markup formats and toolchains to do their
> best at generating output renderable on the greatest number of
> devices.
>
> Interestingly, this was what the various *roffs aimed at originally:
> device-independent output that could be fed to various drivers.
> But somewhere along the line, I believe, that mandate lost its
> importance.  After the valiant attempt of grohtml, no new drivers
> were added until Deri James contributed gropdf.  Had the mandate
> retained its importance, the question of whether groff should have
> an XSL-FO driver would be moot because, in all likelihood, it would
> already have been written, along with several other much-needed
> drivers, grortf being the most pressing for many of us (and I wish
> to God we didn't all hate RTF so much so somebody would, in fact,
> write the driver).
>
> I think what happened is that, over time, near-exclusive use of the
> PostScript driver caused many of us to confuse groff output with
> grops output--if not intellectually, at least at the conceptual
> level.  We began to think of groff as a PostScript typesetting
> engine.  Few are the posts in the last ten years dealing with, say,
> terminal output issues, while legion are the posts about what are
> clearly PostScript presentational issues (like the "space width"
> thread).
>
> I suspect the uneasiness with what Eric has to say about groff's
> future, and the whole semantic-vs-presentational debate, stems from
> Eric addressing the issue in terms of groff's original mandate
> (device-independent output, a concept largely replaced by the notion
> of display-neutral xml), while everyone else looking at groff as a
> typesetter for the paper or full-screen-viewable documents that,
> Eric's doom and gloom predictions aside, continue to form the bulk
> of lists members' groff usage.  Consider Tadziu Hoffman's comment,
> which mirrors my observations, above:
>
>  "Personally, I use groff exclusively for printing out stuff (or
>   creating PDFs as a sort of virtual paper)--something to be read,
>   as is, *by humans*--nothing else.  (For the web (if I must), I
>   write HTML.)  In principle, I could also print from the Browser,
>   but the results are ghastly.  I would not be willing to compromise
>   typesetting quality in exchange for additional media I have no
>   plans of using."
>
> We cannot ignore the need for groff to accommodate the Web and
> ePub-y type things, despite its paper-centricity.  Eric is quite
> right that "printer-only" will never again be enough.  However, as
> he also points out, attempting to extrapolate semantic meaning from
> groff output is impossible "... because the information required to
> do that is thrown aware at macro expansion time."  The conclusion
> he draws from this strikes me as self-evident: "The difficult but
> correct thing to do is to recover structural information by looking
> for cliches in the source markup *before* it goes through troff."
>
> But why "difficult"?  Well, mostly owing to historical groff
> (mis?)use, which fostered conditions where, in Mike's words,
> "...presentation and formating are horribly intermingled."  The
> classical macrosets are not well set up for creating stylesheets
> "that really sing" (esr).  Groff-hands got in the habit of dealing
> with presentational issues for semantic elements on-the-fly,
> either with groff primitives or by writing their own macros--the
> cliches of which Eric speaks.  Had there been, historically, a
> macroset that provided a fairly complete set of semantic tags and
> an easily-parsable mechanism for applying presentational markup
> to those tags, Eric might never have had to write the miracle of
> cleverness that is DocLifter.
>
> Without a single change to groff as it stands now, all that's really
> required to generate xml from groff source files are a) well-formed
> source files, and b) mechanisms for parsing and transforming them
> into xml.  (I speak here of present and future documents, not the
> vintage stuff.)  If I can convert a mom file to xhtml using sed
> (gasp!), xml and xml stylesheets are perfectly feasible with more
> sophisticated tools.
>
> The trick, of course, is writing well-formed source files, with
> clear a distinction between metadata, stylesheet, semantic tags,
> and discardable presentational markup.
>
> * The mom macros
>
> Mom is my baby, but that's not why I'm mentioning it here.  I was
> really surprised by Mike's comment:
>
>  "Done right, a really great macro package would have to clearly
>   separated parts: presentation and format.  But it seems *roff has
>   never really provided the architecture to support that sort of
>   separation, hence macro packages that mush the concepts together."
>
> The mom macros were conceived, from the start, to do exactly what
> he describes: keep presentation and formatting separate.  Every
> semantic tag has a bevy of "control macros" that permit the styling
> of tags separately from the tags themselves, macros that furthermore
> begin with the name of the tag, making their intent instantaneously
> clear, both to human eyes and to a parser assembling a stylesheet.  A
> mom document begins with metadata, is followed by a stylesheet, and
> begins formatting, in the sense Mike means, only after the .START
> macro is issued.  Throughout the remainder of the document, there's
> virtually no need for presentational markup, except discardable
> markup like tightening or loosening a line with track kerning
> (which, moreover, is done in macro space with the easily identified
> EW [extra white] and RW [reduce white]).  In cases where it's
> desirable to make available "in-document" presentational markup of
> a tag, say to adjust the position of a table, the presentational
> markup is attached directly to the tag as an optional argument, which
> can be flagged and ignored at parse time.
>
> Furthermore, mom's typesetting functions can be used independently
> of document formatting (again, in Mike's sense) to create documents
> never likely to be rendered anywhere but on paper (a poster
> advertising kittens free to a good home, for example).
>
> In short, groff has always had the "architecture" to support source
> files that are both human- and xml-friendly, it's just that it's
> rarely been taken advantage of.
>
> * Summing up
>
> Groff can, and does, have a future, one that accommodates both
> printer (or analogous) output and display-neutral xml.  The
> typesetting engine/pipeline (ie ditroff=>grops/gropdf) needs to be
> overhauled to remove the hurdles posed by line-at-a-time processing.
> Historical baggage like not respecting order of precedence can,
> I believe, be fixed without earth-shattering consequences.  Adding
> arrays, as has been suggested, wouldn't break anything.  (Myself, I
> wouldn't mind case statements, as I find I'm increasingly using
> while loops to simulate getopts functionality.)
>
> I think we all agree some requests could do with a dose of sanity,
> which could be accomplished by creating new requests in the manner
> of .de1 or .am1 whenever altering their historical behaviour would
> break existing documents.
>
> And all this is all doable, it seems to me, without affecting backward
> compatibility.  I don't think we have to consider forking; in fact,
> I'm against it.  What we do have to do is acknowledge that groff, as
> a typesetting engine, has the potential to stand next to TeX, and,
> as such, remain a viable choice for quality typesetting.  I really
> don't believe that printer output, or the physical page paradigm for
> screen documents, is anywhere near falling into desuetude.
>
> As for xml output, I'm convinced that's a source file, macro level
> issue.  The mom macros point the way for xml-friendly structuring of
> source files; who knows what a joint-development effort in a similar
> vein could accomplish?
>
> Again, sorry for the long post.
>
> --
> Peter Schaffter
> http://www.schaffter.ca


I wouldn't call it fear, I was "sure" the, let's call it "pottering
solution" or "mswindows integration process" would more or late
contaminate Groff too.

After spotting on space-with subject I've noticed that Groff do the
right thing between sentences.  LaTeX doesn't.  And despite what other
Lennart disciple here said, Groff and LaTeX quality is superior to
most proprietary solutions, at least the ones I've known (he showed up
praising their "GUIs with visual manipulation" feature).

If you are so convinced that "modern tendencies" (pure fashion) should
rule the future of Groff start NOW thinking about a functionality that
does not imply to read and write.  Because in the near future the
average will not be able to stablish any communication in that way.
You know, it isn't a hyperbole (of course, to appreciate "how pretty
the fonts look on the page" is not necessary to be able to read).

Finally, assuming you have the skills to improve Groff why don't you
write your own *better* software and everybody's happy?

Why do you people don't let other species live?, why do you want to
cover all with your shit?  What do you want to do with Groff?
Converting it in one more friendly full featured app to seduce idiot
people?  Is that your "rescue strategy"?  Before that let it die with
dignity.



        Walter