Evaluator Hooking ("RebindableSyntax")

[hostilefork]

After debating the topic for some time, I decided FENCE! should indeed be evaluative.

But rather than setting what it does in stone... let's imagine that what FENCE! does when it evaluates is actually to pass the fence to a function you can redefine in the environment.

demo: func [] [
    let fence!-EVAL: func [fence] [
        print "I got a fence of length" length of fence
        return try second fence
    ]
    return {a b c}
]

>> fence!-EVAL: identity/  ; inside DEMO has a LET of its own choice

>> demo
I got a fence of length 3
== b

>> {x: 10 y: 20}
== {x: 10 y: 20}  ; ...!

By allowing FENCE! to be evaluative, you can even allow it to be unevaluative if you choose, in a context:

That's an idea so powerful...that having block!-EVAL, group!-EVAL, etc. functions all being looked up and run seems like it needs to be done right now. Whatever optimizations are needed to make it not slow in the general non-overridden case can be attended to.

Since FENCE! doesn't do anything at all yet, it's a perfect guinea pig for the technique.

The Default Evaluation Should Be Likely Be Dialected "MAKE"

I think I like the idea that CONSTRUCT is actually is what FENCE! does by default, and you can direct it to make something that's not an object with some special notation (perhaps just something that looks up to a DATATYPE! in the first slot?)

>> string: "0201"

>> {integer! reverse string}
== 1020

I've proposed this before. But that was before pure virtual binding. Now the parts are here, and it's right within reach.

It's time to go for it, because this is what the whole thing is supposed to be about.

---

20 Minutes Later...

It works.

>> {x: 10 y: 20}
== #[object! [
    x: 10
    y: 20
]]

>> fence!-EVAL: func [f] [print ["Length is" length of f]]

>> {x: 10 y: 20}
Length is 4

>> fence!-EVAL: construct/

>> {x: 10 y: 20}
== #[object! [
    x: 10
    y: 20
]]

Years to ponder, minutes to implement. (Of course, it hinges on the blood sweat and tears of pure virtual binding.)

---

Despite being extensible... IT'S FASTER THAN MAKE OBJECT! because instead of looking up MAKE and looking up OBJECT! and building a frame and all that, it calls a native arity-1 intrinsically, with no frame at all! So you're paying for fewer word lookups and not even making a frame the resulting function (if you are using an intrinsic, which CONSTRUCT is, and presumably whatever other default maker would be too).

But crucially here...you should always have fallbacks for doing the creation without needing to use the lexical form. So you can use FENCE! creatively, however you like... but still have MAKE OBJECT! (or whatever) passed blocks to get the behavior if you need it.

And if there's only hookpoints for BLOCK!, FENCE! and GROUP!... but not their variations.. you'll always have quoted '[blocks] and $(groups) etc to fall back on if necessary.

Wild Example #1 : Progressive Parsing

Let's say you wanted to do a parse, but not all at once... rather continuing it a little piece at a time with handling code.

data: [
    The Sharp Gray @Fork "Quantum Leaped" Over The Lazy @Red
]

f: lambda [rule [block!]] [
    parse data [accept [rule, elide data: <here>]]
]

The short (meaningless) name gets things about as brief as you can get in "historical" code:

designer: f [some word!, one]
assert [designer = @Fork]

occurrence: f [text!, elide 'Over]
assert [occurrence = "Quantum Leaped"]

other: f [collect [
    keep one, keep ('Intellectually) keep spread across to <end>
]]
assert [other = [The Intellectually Lazy @Red]] 

But what if you wanted to do it so that a FENCE! was an implicit call to the parse steps?

fence!-EVAL: lambda [rule [fence!]] [
    rule: as block! rule
    parse data [accept [rule, elide data: <here>]]
]

Then your calls could look like this:

designer: {some word!, one}
assert [designer = @Fork]

occurrence: {text!, elide 'Over}
assert [occurrence = "Quantum Leaped"]

other: {collect [
    keep one, keep ('Intellectually) keep spread across to <end>
]}
assert [other = [The Intellectually Lazy @Red]] 

Yes, it works!

Is It THAT Different Than Using A Function? YES.

The difference is significant, and more than just cosmetic.

We've seen arguments against "more parts" before... e.g. saying that GROUP! is not necessary if you have BLOCK! (or vice-versa) because you can always split your intent up into multiple tokens. But it's very Turing Tar-Pit to say "oh it's all the same"... because it is not the same.

When the FENCE! can encode itself in one value vs. needing a word-and-a-value, you get better compositional properties. You have smoother meta-analysis when building higher level things that the fences are composed into (vs. WORD!+BLOCK!)

I see mountains of potential here.

[hostilefork]

@hostilefork

But what if you wanted to do it so that a FENCE! was an implicit call to the parse steps?

fence!-EVAL: lambda [rule [fence!]] [
    rule: as block! rule
    parse data [accept [rule, elide data: <here>]]
]

That method makes multiple calls to PARSE, updating the position of a data variable as it goes.

I'll just point out you could do this with a YIELDER (GENERATOR that takes parameters on each call). That avoids having to update DATA's position...the PARSE just gets suspended on the stack but with the parse state preserving the current position, all as part of a single operation:

fence!-EVAL: yielder [rule [fence!]] [
    rule: as block! rule
    parse data [opt some [let ^result: rule (yield ^result)]]
]

And if you use the ACTION! combinator you can avoid the temporary variable:

fence!-EVAL: yielder [rule [fence!]] [
    rule: as block! rule
    parse data [opt some /yield rule]]
]

[bradrn]

@hostilefork

That's an idea so powerful...that having block!-EVAL, group!-EVAL, etc. functions all being looked up and run seems like it needs to be done right now. Whatever optimizations are needed to make it not slow in the general non-overridden case can be attended to.

This reminds me strongly of RebindableSyntax in Haskell. It’s not used very much, because Haskellers prefer other means for extending the language, plus variable shadowing is frowned upon. In Ren-C, of course, those aren’t problems — and, besides, this feels like the natural conclusion of dialecting as a paradigm.

[hostilefork]

@bradrn

This reminds me strongly of RebindableSyntax in Haskell.

That does indeed look to be very much in the same spirit:

• "So the RebindableSyntax extension causes the following pieces of built-in syntax to refer to whatever is in scope, not the Prelude versions"

• "An integer literal 368 means fromInteger (368::Integer), rather than Prelude.fromInteger (368::Integer)".

(Perhaps similar enough that calling it "Rebindable Syntax" might be a good idea. I hate making up new names for things that already have serviceable names.)

@hostilefork

And if there's only hookpoints for BLOCK!, FENCE! and GROUP!... but not their variations.. you'll always have quoted '[blocks] and $(groups) etc to fall back on if necessary.

So I called the FENCE! hook "fence!-EVAL" based on the idea that you wouldn't want to redefine the natives that perform the actions. But given that you can call LIB/XXX versions, maybe you do want to redefine the natives?

e.g. perhaps it should have been CONSTRUCT instead. "Want to redefine FENCE!'s behavior? Redefine CONSTRUCT."

If that pattern extended, quoted things would run UNQUOTE, I guess?

>> unquote: func [x] [reduce [lib/unquote x]]

>> 'x
== [x]

>> '''{a}
== [''{a}]

GROUP!s could run EVALUATE. EVAL might be rethought as an alias for EVALUATE, e.g. look up whatever EVALUATE is defined as in the current context and run that, vs. a hard reference to LIB.EVALUATE. Although you'd want LIB/EVAL to run LIB/EVALUATE, I guess the aliasing would need to take where it was being run from into account.

BLOCK!s could run... uh... I dunno. What does the [] say?

Er, Nevermind... xxx!-EVAL is Better

Not knowing what the BLOCK!'s corresponding native is kind of drives that home.

And think I like that fence!-EVAL stands out enough to make people go "something weird is going on here", and you don't have to corrupt the meanings of natives to get it. You can call CONSTRUCT from inside your fence!-EVAL instead of having to call LIB/CONSTRUCT.

But I got on this track when crossed my mind that the hooking could call regular looking functions... this was particularly because I'm thinking about being able to redefine SET and GET.

>> set-EVAL: func [var ^value] [print ["var is" mold var "value is" value]]

>> x: 1000 + 20
var is x value is '1020

But it's probably for the best you have to say something like set-EVAL instead of requiring you to override set. Again: probably the first thing you're going to be reaching for in implementing your SET customization is... SET.

@bradrn

[RebindableSyntax in Haskell is] not used very much, because Haskellers prefer other means for extending the language, plus variable shadowing is frowned upon. In Ren-C, of course, those aren’t problems — and, besides, this feels like the natural conclusion of dialecting as a paradigm.

It does seem that if you buy into Rebol's odd bargain enough to use the language in the first place, then this shouldn't raise eyebrows. You'd be doing these alterations of behavior anyway in a dialect. So why not make it easier to bind a block to some altered evaluators and just run it, vs. accomplishing the same goal with FOR-EACH or PARSE and having to worry about all the binding manually?

Not fit for all purposes. But when every invocation of something like IF is making a function call... well, you might as well be getting something for your money.

[hostilefork]

@hostilefork

why not make it easier to bind a block to some altered evaluators and just run it, vs. accomplishing the same goal with FOR-EACH or PARSE and having to worry about all the binding manually?

Not fit for all purposes

I was contemplating the idea of having things like FIND pick up on your operating idea of equality:

>> find [a b c] 'B
== /~null~/  ; antiform

>> eval [
       let equal?: lib.lax-equal?/

       find [a b c] 'B
   ]
== [b c]

That's appealing--in terms of letting people make their own choices.

...BUT... it doesn't work if you call a function that wraps find. Imagine something that searches 2 lists:

Let's say you're using some library and it defines FIND2:

find2: lambda [list1 list2 thing] [  ; <-- gets EQUAL? from this block's binding
    any [
        find list1 thing
        find list2 thing
    ]
]

If you use that FIND2, then it will use whatever notion of binding was on the block it received for its body.

But this isn't any kind of profound or novel problem...and it's not unique to this language. It's not like having an argument to pass to FIND gets you any further.

If you call somebody who calls something else, the something else won't get your parameterization unless they explicitly account for it in some way.

Though in Ren-C, if you do something like specialize FIND, then it will still fundamentally be a call to find... and (for better or worse) that specialization would still use the callsite's notion of EQUAL?

We could make it possible to say "using caller's notion of..." in functions... hand waving a bit but something like:

find2: lambda [list1 list2 thing <caller-environment> env] [
    let equal?: env.equal?/
    any [
        find list1 thing
        find list2 thing
    ]
]

Anyway... it's still a new idea to be able to do this kind of thing at all, so I don't really know what the applications will be.