About the Lexer category

This is for discussions about how to process UTF-8 text into Rebol elements.

Generally speaking, historical Rebol tended to break things up into separate parts more often:

rebol2>> load "(a b)/c"
== [(a b) /c]

Modern ideas have many more ways of merging parts into single elements:

>> transcode "(a b)/c"
== [(a b)/c]

There are a particularly large number of nuances about how TAG! is handled. As the tip of the iceberg, <> is a WORD!, and not an empty tag.