Evaluator Hooking ("RebindableSyntax")

After debating the topic for some time, I decided FENCE! should indeed be evaluative.

But rather than setting what it does in stone... let's imagine that what FENCE! does when it evaluates is actually to pass the fence to a function you can redefine in the environment.

demo: func [] [
    let fence!-EVAL: func [fence] [
        print "I got a fence of length" length of fence
        return try second fence
    ]
    return {a b c}
]

>> fence!-EVAL: identity/  ; inside DEMO has a LET of its own choice

>> demo
I got a fence of length 3
== b

>> {x: 10 y: 20}
== {x: 10 y: 20}  ; ...!

By allowing FENCE! to be evaluative, you can even allow it to be unevaluative if you choose, in a context:

:exploding_head:

That's an idea so powerful...that having block!-EVAL, group!-EVAL, etc. functions all being looked up and run seems like it needs to be done right now. Whatever optimizations are needed to make it not slow in the general non-overridden case can be attended to.

I've been working out the definition of CONSTRUCT as FENCE!'s default behavior, a dialect specific to the efficient and essential creation of objects.

But since FENCE! doesn't have any usages in the wild right now, it's a perfect guinea pig for the technique. So let's give it a shot, because this is what the whole thing is supposed to be about. :rocket:


20 Minutes Later...

It works. :slight_smile:

>> {x: 10 y: 20}
== #[object! [
    x: 10
    y: 20
]]

>> fence!-EVAL: func [f] [print ["Length is" length of f]]

>> {x: 10 y: 20}
Length is 4

>> fence!-EVAL: construct/

>> {x: 10 y: 20}
== #[object! [
    x: 10
    y: 20
]]

Years to ponder, minutes to implement. :hourglass_done: (Of course, it hinges on the blood sweat and tears of pure virtual binding.)


Despite being extensible... IT'S FASTER THAN MAKE OBJECT! because instead of looking up MAKE and looking up OBJECT! and building a frame and all that, it calls a native arity-1 intrinsically, with no frame at all! So you're paying for fewer word lookups and not even making a frame the resulting function (if you are using an intrinsic, which CONSTRUCT is, and presumably whatever other default maker would be too). :racing_car:

But crucially here...you should always have fallbacks for doing the creation without needing to use the lexical form. So you can use FENCE! creatively, however you like... but still have MAKE OBJECT! (or whatever) passed blocks to get the behavior if you need it.

And if there's only hookpoints for BLOCK!, FENCE! and GROUP!... but not their variations.. you'll always have quoted '[blocks] and $(groups) etc to fall back on if necessary.

Wild Example #1 : Progressive Parsing

Let's say you wanted to do a parse, but not all at once... rather continuing it a little piece at a time with handling code.

data: [
    The Sharp Gray @Fork "Quantum Leaped" Over The Lazy @Red
]

f: lambda [rule [block!]] [
    parse data [accept [rule, elide data: <here>]]
]

The short (meaningless) name gets things about as brief as you can get in "historical" code:

designer: f [some word!, one]
assert [designer = @Fork]

occurrence: f [text!, elide 'Over]
assert [occurrence = "Quantum Leaped"]

other: f [collect [
    keep one, keep ('Intellectually) keep spread across to <end>
]]
assert [other = [The Intellectually Lazy @Red]] 

But what if you wanted to do it so that a FENCE! was an implicit call to the parse steps?

fence!-EVAL: lambda [rule [fence!]] [
    rule: as block! rule
    parse data [accept [rule, elide data: <here>]]
]

Then your calls could look like this:

designer: {some word!, one}
assert [designer = @Fork]

occurrence: {text!, elide 'Over}
assert [occurrence = "Quantum Leaped"]

other: {collect [
    keep one, keep ('Intellectually) keep spread across to <end>
]}
assert [other = [The Intellectually Lazy @Red]] 

Yes, it works!

:open_mouth:

Is It THAT Different Than Using A Function? YES.

The difference is significant, and more than just cosmetic.

We've seen arguments against "more parts" before... e.g. saying that GROUP! is not necessary if you have BLOCK! (or vice-versa) because you can always split your intent up into multiple tokens. But it's very Turing Tar-Pit to say "oh it's all the same"... because it is not the same. :pouting_cat:

When the FENCE! can encode itself in one value vs. needing a word-and-a-value, you get better compositional properties. You have smoother meta-analysis when building higher level things that the fences are composed into (vs. WORD!+BLOCK!)

I see mountains of potential here.

:mountain: :man_climbing:

2 Likes

That method makes multiple calls to PARSE, updating the position of a data variable as it goes.

I'll just point out you could do this with a YIELDER (GENERATOR that takes parameters on each call). That avoids having to update DATA's position...the PARSE just gets suspended on the stack but with the parse state preserving the current position, all as part of a single operation:

fence!-EVAL: yielder [rule [fence!]] [
    rule: as block! rule
    parse data [opt some [let ^result: rule (yield ^result)]]
]

And if you use the ACTION! combinator you can avoid the temporary variable:

fence!-EVAL: yielder [rule [fence!]] [
    rule: as block! rule
    parse data [opt some /yield rule]]
]
1 Like

This reminds me strongly of RebindableSyntax in Haskell. It’s not used very much, because Haskellers prefer other means for extending the language, plus variable shadowing is frowned upon. In Ren-C, of course, those aren’t problems — and, besides, this feels like the natural conclusion of dialecting as a paradigm.

1 Like

That does indeed look to be very much in the same spirit:

  • "So the RebindableSyntax extension causes the following pieces of built-in syntax to refer to whatever is in scope, not the Prelude versions"

  • "An integer literal 368 means fromInteger (368::Integer), rather than Prelude.fromInteger (368::Integer)".

(Perhaps similar enough that calling it "Rebindable Syntax" might be a good idea. I hate making up new names for things that already have serviceable names.)

So I called the FENCE! hook "fence!-EVAL" based on the idea that you wouldn't want to redefine the natives that perform the actions. But given that you can call LIB/XXX versions, maybe you do want to redefine the natives?

e.g. perhaps it should have been CONSTRUCT instead. "Want to redefine FENCE!'s behavior? Redefine CONSTRUCT."

If that pattern extended, quoted things would run UNQUOTE, I guess?

>> unquote: func [x] [reduce [lib/unquote x]]

>> 'x
== [x]

>> '''{a}
== [''{a}]

GROUP!s could run EVALUATE. EVAL might be rethought as an alias for EVALUATE, e.g. look up whatever EVALUATE is defined as in the current context and run that, vs. a hard reference to LIB.EVALUATE. Although you'd want LIB/EVAL to run LIB/EVALUATE, I guess the aliasing would need to take where it was being run from into account.

BLOCK!s could run... uh... I dunno. What does the [:fox:] say?

Er, Nevermind... xxx!-EVAL is Better

Not knowing what the BLOCK!'s corresponding native is kind of drives that home.

And think I like that fence!-EVAL stands out enough to make people go "something weird is going on here", and you don't have to corrupt the meanings of natives to get it. You can call CONSTRUCT from inside your fence!-EVAL instead of having to call LIB/CONSTRUCT.

But I got on this track when crossed my mind that the hooking could call regular looking functions... this was particularly because I'm thinking about being able to redefine SET and GET.

>> set-EVAL: func [var ^value] [print ["var is" mold var "value is" value]]

>> x: 1000 + 20
var is x value is '1020

But it's probably for the best you have to say something like set-EVAL instead of requiring you to override set. Again: probably the first thing you're going to be reaching for in implementing your SET customization is... SET.

It does seem that if you buy into Rebol's odd bargain enough to use the language in the first place, then this shouldn't raise eyebrows. You'd be doing these alterations of behavior anyway in a dialect. So why not make it easier to bind a block to some altered evaluators and just run it, vs. accomplishing the same goal with FOR-EACH or PARSE and having to worry about all the binding manually?

Not fit for all purposes. But when every invocation of something like IF is making a function call... well, you might as well be getting something for your money.

2 Likes

I was contemplating the idea of having things like FIND pick up on your operating idea of equality:

>> find [a b c] 'B
== /~null~/  ; antiform

>> eval [
       let equal?: lib.lax-equal?/

       find [a b c] 'B
   ]
== [b c]

That's appealing--in terms of letting people make their own choices.

...BUT... it doesn't work if you call a function that wraps find. Imagine something that searches 2 lists:

Let's say you're using some library and it defines FIND2:

find2: lambda [list1 list2 thing] [  ; <-- gets EQUAL? from this block's binding
    any [
        find list1 thing
        find list2 thing
    ]
]

If you use that FIND2, then it will use whatever notion of binding was on the block it received for its body.

But this isn't any kind of profound or novel problem...and it's not unique to this language. It's not like having an argument to pass to FIND gets you any further.

If you call somebody who calls something else, the something else won't get your parameterization unless they explicitly account for it in some way.

Though in Ren-C, if you do something like specialize FIND, then it will still fundamentally be a call to find... and (for better or worse) that specialization would still use the callsite's notion of EQUAL?

We could make it possible to say "using caller's notion of..." in functions... hand waving a bit but something like:

find2: lambda [list1 list2 thing <caller-environment> env] [
    let equal?: env.equal?/
    any [
        find list1 thing
        find list2 thing
    ]
]

Anyway... it's still a new idea to be able to do this kind of thing at all, so I don't really know what the applications will be.

1 Like

As a sort of proof of the "words aren't what we want" aspect, I changed FENCE! to default to WRAP and not CONSTRUCT...

I can still easily imagine being in some context where you want it to act like construct!

Weird Invasive Thought... what about {}: construct/

It's wacky. How would you get the current hook? Trailing slash?

{}: construct/

assert [construct/ = {}/]

This "just use the symbols" idea could go arbitrarily nuts, I guess:

>> @: func [x [none? any-element?]] [print ["DUMPING:" mold reify x]

>> @(a b c)
DUMPING: (a b c)

>> @
DUMPING: ~()~

>> x
DUMPING: x

Where this concerns me a bit is just the accidental formation of reduced cases having this heavily consequential outcome.

But... eh? It's a little hard to reject once you see it.

>> {x: 10 y: 20}
== [x: 10 y: 20]  ; bound

>> {}: func [f] [print ["Length is" length of f]]

>> {x: 10 y: 20}
Length is 4

>> {}: construct/

>> {x: 10 y: 20}
== &[object! [
    x: 10
    y: 20
]]

I'm Kind of Smitten

>> foo: proc [x] [
       let (): proc [group] [print to block! group]
       ("seriously, folks...why not?")
       ("by the way, X is" x)
     ]

>> foo 1000 + 20
seriously, folks...why not?
by the way, X is 1020

>> (300 + 4)  ; alteration to () is only in scope of foo
== 304

It has what feels like are problems, because it's invading the base case of things that might be synthesized on accident.

But are they really problems, when measured against the "HOLY CRAP, that is POWER, and that is FLUENT" aspect?

(cc: @bradrn for vibe check. Amazing lexical initiative, or silly/broken?)

:thinking:

When I put down the crack pipe :face_with_spiral_eyes: I realize this would be too dangerous/random to slipstream into regular assignments (which could be produced by programmatic code).

So maybe the actual realization here is that whatever syntax is used...it really needs to be semiotic and use the delimiters/sigils directly, and not be too much worse than such fluent assignments would appear.

Why LET and not something like (say) HOOK that acted like it but only accepted reduced case entities...leaving ordinary assignment out of it? Maybe HOOK is implicitly a LET?

>> foo: proc [x] [
       hook (): proc [group] [print to block! group]
       ("seriously, folks...why not?")
       ("by the way, X is" x)
     ]

>> foo 1000 + 20 
seriously, folks...why not?
by the way, X is 1020

I'm hand-waving a bit here about the LET vs. override. But my point is you can probably get in striking distance of the "aesthetic" I'm suggesting, without compromising rank-and-file assignment.

And dumb as this may seem: HOOK could defeat the things you wouldn't think should work:

>> hook ': proc [quoted] [print ["DUMPING:" mold unquote quoted]]

>> '(a b c)
DUMPING: (a b c)

This completely ignores the fact that ': is a quoted colon, not a "Set-Apostrophe". But it could just be the "picture" of what hook takes, how it encodes the desire to override quote.

(I'm not saying this is a great idea, I'm just saying that doors open when you are willing to use a literal slot as "looks like" vs. demanding it follow the ordinary rules of assignments.)

Or Maybe It's A Form Of Dialected Function Call?

>> hook:(): proc [group] [print join to block! group "!!!"]

>> ("Hello World")
Hello World !!!

>> hook:': func [quoted] [print ["DUMPING:" mold unquote quoted]]

>> '(a b c)
DUMPING: (a b c)

I haven't thought of the idea of CHAIN! ending in blank yet as a dialected call, but, hey. It could be a way to make "SET-looking" things, and it solves the "how to get a quote mark there" problem.