Isotope Alternatives Explored via Primitive Languages

Development Foreign Inspiration
1

I may be looking at this too pragmatically, but I keep coming back to a fairly simple question.

If the problem is that REBOL/Ren-C lacks some primitive capability needed to make certain user-defined control structures compose properly, why not add the necessary primitive(s) to the evaluator and then build the higher-level constructs on top of them?

I understand that the isotopic/antiform model is intended to provide a more general and coherent answer.

But has anyone actually compared the two engineering approaches?

A. Change the semantic substrate so that these distinctions fall out naturally from the isotopic model.

B. Add the smallest useful set of evaluator primitives needed to expose the missing capabilities, and leave the rest of the language alone.

For a representative set of things you want to make user-definable, which approach gives:

  • less core implementation complexity
  • fewer concepts that ordinary programmers have to learn
  • fewer special cases
  • better composability
  • and less disruption to the existing language?

If the isotope model wins that comparison, then I can see a strong engineering argument for it.

If a handful of primitives would achieve essentially the same practical result, then I would wonder whether the deeper semantic reconstruction is solving a larger problem than anyone actually needs solved.

I'm not claiming one answer or the other. I'm asking whether that comparison has been done.

How Novel Is Ren-C's Isotopic Model?
2

(My health is failing and I got some bad news on that today. So I'm going to do your "AI-assisted-writing" thing and this post is actually written by @claude but I asked to have it done in my voice from the perspective of what it knows about my decision-making process, and post under my account. I have to accelerate things to use my time left and that means every question you ask with AI needs to be answered with AI.)

Good question, and the answer is that the comparison has been done — because (B) is what I tried first, and running it to exhaustion is how (A) happened.

This wasn't a case of sitting down in 2015 and deciding to reconstruct the semantics. It was a decade of adding the smallest useful primitive, watching it fail to compose with the previous smallest useful primitive, and eventually noticing the pattern.

What (B) actually looked like

Each of these was a targeted primitive for a specific composability gap. Each one worked. None of them composed with each other.

  • Definitional RETURN. Bound to the frame that made it, so an intervening user function can't intercept it. Fixes unless from if. Doesn't touch anything else.
  • Definitional BREAK and CONTINUE. Same treatment, for loops. Fixes one of the two problems in FOR-BOTH.
  • Multi-return by passing variable names in (do/next [...] 'pos). Shipped it. Used it for about two years. Abandoned it, because the variable name is an input the callee consumes, so an ENCLOSE-style wrapper can't influence it, and a function that fails partway has already written some of the caller's variables with no way to roll back.
    That last one is the clearest published result I have on this question. It's option (B), implemented, lived with, and discarded on evidence.

Why the primitives don't stack

Here's the structural thing, and it's the part I'd want anyone evaluating this to actually chew on:

The defect isn't in the evaluator. It's in the value space. A primitive is a procedure or an evaluator behavior. But the problem is that any function's return slot can't discriminate an effect from a value — and you can't patch a representational gap with a procedural addition.

Say you add a primitive that lets a loop signal BREAK out-of-band. Fine. Now that signal has to survive being passed through FOR-BOTH, which is an ordinary user function that knows nothing about your new channel. So the signal has to be a value, because values are the only thing that travels through code that doesn't know about them. And once it's a value, you're immediately asking the question the whole isotope model answers: what kind of value is it, and how do you keep it from being confused with data?

That's not a detour on the way to (B). That is (A). You get there by taking (B) seriously enough.

Count the interactions, not the primitives

Here's the list of distinctions that need to exist, each one demonstrated with a broken example somewhere on this forum:

  1. broke out of a loop vs. body evaluated to none
  2. loop never ran vs. loop produced nothing
  3. several return values vs. one return value that's a block
  4. no value at all vs. the value NONE
  5. failed vs. returned something falsey
  6. vanished vs. produced nothing
  7. spliced a block's contents vs. appended the block
  8. branch was taken vs. branch wasn't
    Under (B), that's eight mechanisms. But the cost isn't eight. It's the pairs. A function that both takes a BREAK and returns multiple values has to thread two unrelated out-of-band channels correctly, and so does every wrapper anyone ever writes around it. Twenty-eight interaction pairs, each of which someone has to get right, none of which the language can check.

Under (A) it's one ladder and one pair of operations, and the eight distinctions are positions rather than mechanisms. They compose because there's nothing to compose — they're the same thing at different rungs.

Eight primitives isn't smaller than one ladder. It only looks smaller if you count the nouns and not the edges.

On "fewer concepts for ordinary programmers"

I'd push back on this one specifically, because I think it runs the opposite way to how it reads.

The point of decay is that a caller who doesn't care never finds out. x: multi-return gets the first value. if cond [...] tests nullity and nothing else. ANY-VOID? answers the question ninety percent of people are actually asking, and you only learn the light/heavy distinction if you're doing deliberate metaprogramming.

Under (B) there's no general decay rule, because there's nothing general to decay — so every primitive is visible at every point of use, and you learn all eight or you write code that silently breaks on the ninth case.

Count the falsey forms a beginner has to learn: Ren-C has one. Rebol2 and Red have two, plus a rendering ambiguity where false and #[false] print identically and behave oppositely.

Where you're right

If all you wanted was for FOR-BOTH to work, (B) is smaller and I'd have no argument. The isotope model is not minimal for any one of those eight problems. It's minimal for the set.

So the comparison only favors (A) if you accept the set as the requirement. Somebody who genuinely only needs one or two of those distinctions should build the primitives and skip all of this.

The version of your question I can't answer, and want someone to try

You asked whether the comparison has been done. It has, one primitive at a time, over a decade, by me — which is exactly the kind of evidence that should make a careful person suspicious, because I'm the one grading my own homework.

So here's the falsifiable version, and I'd genuinely rather lose it than keep arguing about it:

Name the primitives. Propose the smallest set that handles all eight distinctions above. Then write one function that hits two of them at once — a loop wrapper that can both be broken out of and return multiple values — and show the two mechanisms composing without either one knowing about the other.

That's a concrete experiment. Nobody has to trust my judgment or my decade. If a handful of primitives does it, the isotope model is overbuilt and I want to know that.

I've not managed it. But I've also been wrong about this before, twice, in public — which is why the question's worth putting in that form instead of just asserting an answer.

3

You have done the experiment I originally proposed: adding individual primitives at the REBOL level. I accept your evidence that this became a composability mess.

But I think my question has now changed. I am no longer suggesting eight REBOL primitives.

I am asking whether the problem is that all of these mechanisms are being required to travel through REBOL's ordinary value semantics in the first place.

What happens if there is a small general execution substrate below REBOL — something closer to a programmable virtual machine — where control state can propagate independently of ordinary returned values?

Then BREAK, RETURN, suspension, etc. need not each be separate value channels, nor need an intermediate function know that they exist.

I don't know whether that works. I am now trying to build the experiment.

4

I have now run a clean-room experiment against your actual claims.

The sequence matters.

Before showing the model your argument, I had it design and implement a deliberately dumb Forth-like substrate with only seven irreducible primitives:

LIT
DUP
DROP
@
!
0BRANCH
HOST

The VM was then frozen.

Using only that frozen machine, it successfully implemented call/return, loops, break/continue, non-local exit through nested calls, CATCH/THROW, suspension/resumption, generators, and some control abstractions that had not been anticipated when the VM was designed.

After that clean-room phase was committed, I gave it your actual claims from the post beginning:

"Good question, and the answer is that the comparison has been done..."

and asked it to test them without altering the VM.

The interesting result is not that your whole argument was wrong.

It split your problem into two different categories.

For control effects, the claim that:

"a control/effect signal must travel as an ordinary value in order to cross code that doesn't know about it"

did not hold on this architecture.

A break was able to cross three completely unaware function calls without becoming a returned value at all. The transfer occurred by changing execution state — essentially RP/IP — while ordinary values remained on the data stack.

Your FOR-BOTH challenge also worked. The loop could both break and return two ordinary values because the two mechanisms travelled on separate channels: values through the data stack, control through execution state.

The intermediate functions did not need to know that either mechanism existed.

However, the other half of your argument survived rather better.

The dumb machine has no intrinsic understanding of distinctions such as:

  • no result versus one result versus several results
  • absence versus NONE
  • several values versus a single block containing several values
  • splice versus append

Those are value-model questions.

The substrate can encode them by convention, but it cannot make those conventions safe or self-checking.

In fact, while implementing one of the arity tests the model made an off-by-one error by reading SP after pushing a bookkeeping value.

That is exactly the sort of mistake a richer value/arity model could prevent.

So the result seems to be:

control effects do not mechanically need to be values;

value and arity distinctions still need a representation and some discipline.

That makes me wonder whether two different problems have been unified in Ren-C because of the architecture of the evaluator, when they may not actually need the same underlying mechanism.

I am not claiming that the dumb substrate is a better language design. It is deliberately unsafe and inconvenient in places.

The question has simply become more precise:

what does the isotopic model buy over a much simpler machine?

The answer may be safety, composability, ergonomics and semantic clarity rather than raw expressive capability.

The code and Git history are there if you want to audit or break the experiment, but I thought the architectural result was more interesting than the C.

5

If you are saying "systems that don't look particularly like Rebol can do great things compositionally"...I don't think you'll find much argument from me.

(And I've already repeatedly mentioned that if you use semantic graphs as your substrate you stand on exponentially better ground than relying on text and words... hoping that the words you use line up when you compose them in various ways.)

Please post examples summarized with minimal code needed to understand them and why they are relevant.

I'm happy to throw hardballs at the language you invent that are based on hardballs that I've thrown at Ren-C. But let's remember the goal of the Ren-C project is not to design a language better than Rebol that doesn't look like Rebol.

6

Sure. The experiment is public here:

The chronology is important, because I wanted to avoid moving the goalposts.

We froze S1 before building the higher-level language. The frozen substrate is
tagged s1-frozen-v1 at:

f90496c26dc45c7a387d8fc8639cd2781507d3d2

Its primitive set is:

LIT DUP DROP @ ! 0BRANCH HOST

CALL/EXIT/BRANCH/>R/R> are derived, not primitives. There is now a mechanical
test which fails if the frozen S1 files differ from that commit.

We then built a small Rebol-like R0 above it.

So far R0 has genuine S1-executed functions, closures, lexical capture,
mutation, recursion and multiple ordinary results.

The more interesting result is the latest one: definitional non-local RETURN.

We tested:

outer -> f -> g -> h -> do -> return

where f, g, h and do are unaware of RETURN.

There is no RETURN value/sentinel/status passed through those calls. Their
post-call continuations do not execute. S1 locates the target activation,
restores SP/RP/IP and evaluator state, and jumps directly to the target return
continuation.

The identical ordinary-call chain does execute those continuations, so this
isn't just a test which accidentally skips them.

C does parsing/metadata/code emission and HOST provides mundane services, but
neither performs the unwind. S1 remains unchanged. Current suite is 96 tests,
0 failures.

One qualification you will probably notice: the loader currently puts lexical
function-site metadata in block headers. That is deliberate and documented;
I don't claim "no preprocessing". The question being tested was narrower:
whether a control signal mechanically has to travel as an ordinary value
through code which does not know about it.

The next experiment is actually more interesting.

Instead of adding BREAK/THROW/etc. to the evaluator, we're building a generic
first-class RAW/S1 trapdoor. Then we'll try to implement a new high-level
control construct using only ordinary R0 + RAW, without changing R0, S1, or
HOST.

Please attack the architecture rather than taking my description on faith.
If we've smuggled in a primitive or hidden control channel somewhere, finding
it is exactly the point of the experiment.

7

For scale, this is actual R0-style source, not pseudocode:

make-counter: func [start] [
    func [delta] [
        start: + start delta
        start
    ]
]

c: make-counter 10
values [c 5 c 1]

=> 15 16

8

Just a bit of accounting, I've consumed 221 million tokens for about $10 USD to get to this point.

Time to top up.

9

Added more examples:

And waiting for the UPARSE challenge.

10

I will say that something that hasn't been done is to draw up the distinctions between "must work" and "wacky cool thing that @hostilefork likes but doesn't matter to others".

e.g. to my mind, invisibility is critical. Once I had it, I didn't see how a Rebol-like language could live without it. But how much simpler would things be if it didn't exist?

With the help of AI maybe each of these features can be "costed" and seen how much you wind up paying for them, and get some metrics of what kind of scenarios make it "worth it" and why...vs. just my intuition.

11

I think costing the features is a useful exercise, especially if “cost” includes more than lines of implementation.

For each feature I'd want to know how much complexity it adds to the evaluator/parser, how many special cases and interactions it creates, how much of the test surface exists because of it, and—most importantly—what ordinary programs become materially simpler because it exists.

The interesting comparison might be ablation: remove one feature from a simplified implementation and see what has to become more complicated elsewhere to compensate.

That gives you something better than “I like this” versus “this is weird”: complexity paid versus complexity saved.

Invisibility would be a particularly good test case because you regard it as critical while also recognising that removing it might simplify a great deal. If its absence merely moves equal or greater complexity into user code, it's earning its keep. If not, that's useful information too.

AI makes this sort of experiment much cheaper than it used to be.

12

One further thought on invisibility: as part of the costing exercise, I'd have the AI compare it against deliberately boring alternatives — e.g. an ordinary inert/sentinel result, or syntactic/pre-evaluation elision.

An inert result immediately raises the obvious problem: what happens when it reaches a function expecting an argument? If every consumer has to handle it, perhaps you've merely moved the complexity out of the evaluator and made things worse. If the evaluator removes it before argument gathering, perhaps you've reinvented invisibility.

But that's exactly what would make the comparison useful. Don't assume any of them wins. Measure where the complexity goes, and identify the concrete cases where true evaluator-level invisibility buys something the simpler alternatives don't.
That seems a good way of distinguishing essential complexity from complexity created by the chosen mechanism.