Semantic UI

Emacs and web browsers seem to be used rather often for UIs, even when the programs are not really for text editing (in case of Emacs), and aren't just documents (in case of web browsers). Though web browsers are more like thin clients.

Some of the advantages of emacs UI over GUI toolkits are extensibility and minor modes, and of web browsers – perhaps ease of use (by beginners in particular), and still more extensibility than with regular GUI toolkits (i.e., one may use vimperator, firemacs, or open pages from emacs). The "Cross-platform GUI Toolkit Trainwreck" article sums up the situation with GUI toolkits quite nicely, btw.

There are projects such as emacsy, which tries to bring extensibility to regular GUI, but it's abandoned, and isn't quite about rendering; readline is there, too, though it's just a small part of UI. I haven't found much of HCI/UI materials, particularly nothing along the lines of emacsy, so here will be initial speculations on that topic.

1 Semantics

While some approaches seem to simplify drawing, I think that the key to both extensibility and accessibility is to rely on semantics, rather than on drawing: i.e., a program may provide something like this:

(section "Greetings"
         (paragraph "foo")
         (image "bar.svg" :description "bar")
         (paragraph "baz"))

Instead of, well, what's usually there: precise colors, layouts, and so on. That way, there's much more flexibility for a user: a theme may decide whether to render those sections as windows, or use something like emacs buffers, or simply put them in the same window; it can list the sections, provide fancy or minimalistic elements for inputs, and so on.

Knowledge representation is a related and interesting topic, but it's also a rabbit hole (though one can settle on RDF).

2 UI layers

To translate semantics into actual UI, and to make the translation customizable, it seems that the layered approach may be nice:

  • The first layer is the program-defined elements, reflecting semantics – as in the example above.
  • The second layer is a rather abstract representation: textual, raster, or vector. Those could be translated into each other using generic functions, so that if an element from the previous layer gets mapped into text, it's also easy to render it as raster.
  • The third layer is just for backends. Optionally, the elements from the first layer may be mapped to backends directly.

If each element on the first layer has at least one mapping to the second layer, and at least one element of the second layer has a mapping to the backend, it's possible to render the element. I wrote a prototype using a weighted directed graph of those elements, it's pretty simple and seems to work fine; probably will refine and publish it in the future.

It may look like this (+ weights):

plaintext, curses, cairo+gtk, sdl+freetype
         text ↔ raster ↔ vector
 section, timestamp, table, plot, diagram

If a program requires a custom component, it may just define it, and add into the graph. Although, will it be that different from JS abuse on the web? That's a tricky part: bare HTML itself is more or less semantical, most of the issues don't arise from it alone; they arise from the abuse of CSS, JS, and fonts, tearing apart all the semantics. But maybe HTML is still a viable option, if you abstract from the way it is [ab]used nowadays.

3 Interaction

It's pretty hard to explain in a generic, computer-readable way what a program is capable of doing, and what inputs it requires. Though it's an interesting thing to try and to think of, apparently the more practical way is just to borrow the controls from emacs (REPL, M-x, keybindings to invoke functions, internal documentataion, apropos, etc), and provide the means to update the UI (e.g., sexp diffs).

Graphical input elements may be rendered out of [optionally?] typed functions, using their docstrings for descriptions that seem to make GUI more appealing to newbie users. While describing input fields themselves brings a whole bunch of other problems (blocking and/or state changes between checks and updates, code clutter, etc), which are better to avoid.

4 Functionality mapping

To map things into such an UI, it's straightforward to consider a program ≈ emacs major mode ≈ website's server side, while UI ≈ the rest of emacs ≈ all the client-side bits (including a web browser).

Minor modes may alter the elements, the graph, keybindings, and so on. Themes may be applied by changing the ways things are translated between layers.

5 Further work

I'm currently a bit stuck on picking a graphical back end for a prototype (only implemented a textual one). Also didn't use Scheme for a while, so not satisfied with my code – should refine that (or maybe just prototype it in Haskell or Idris: it's awkward to use something like a functor in Scheme, without really writing it all down, and arbitrary deciding how to encode it).

Then I could write a program to try it out – a feed reader, maybe, since I'm not satisfied with the ones I've seen so far. Those are relatively basic programs, but the UIs tend to be unsatisfactory, and emacs is not that nice for pictures, while I use the feed readers mostly to read web comics.

6 Another prototype

Made a semantic-ui repository on github, with another prototype – without the middle layer yet, in Haskell with SDL. I'm quite happy with the result, but there's a couple of things to decide next:

  1. Name: "Semantic UI" is taken by a CSS framework, and binary names such as view, feed, and doc are too generic. Maybe something like "igless" for "interactive and graphical less", since it's somewhat like a pager, or "sigless" to add "structured" (or "semantic").
  2. Before adding fancier things and optimizing it further, I should decide whether to continue it in Haskell, or use something like C (or Rust, or Idris). Haskell seems like the easiest option, but C compilers are much more common on machines, and Guile aims to work with it. And Idris is nicer.
  3. It's not clear what elements it should support, and how interactive it should be. Perhaps there's no "right" answer, and it's just about balancing.

Guile is quite important in making it customizable, though the viewer can always just be replaced by another viewer: the only part that should be fixed is the interface between programs and UI. Perhaps the initial viewer should be just nice and snappy.

7 Update (2017-08-28)

Though I'm still using the prototype mentioned above to read web comics, it seems that restricted HTML may be more suitable for the purpose: it's trying to be semantic nowadays, and it might be more handy for processing.

There are other ways to get more or less declarative UIs, though those seem to focus on representation, not on semantics.

8 Update (2018-04-18)

Here is an article arriving to a somewhat similar conclusion (not a semantic description, but some kind of a language-agnostic declarative GUI): "Gui development is broken".

Thinking of it again, and re-reading this note, it seems that maybe I shouldn't have dismissed knowledge representation/RDF.