Understanding FRAME! "Lensing"

Development Internals
1

Good news: an old issue is (seemingly) mostly addressed!

Among the various implications of this design improvement, you can AUGMENT a function with new fields that share the name of either locals or specialized values. The only names you cannot use in extending a function are those that are public parameters on the interface!

>> ap10: specialize append/ [value: 10]
>> ap10 [a b c]
== [a b c 10]

>> wow: adapt (augment ap10/ [:value [integer!]]) [insert series value]
>> wow:value [a b c] 20
== [20 a b c 10]

So what's going on here is that underneath the hood, the single FRAME! for this function call has two slots with the label value. But they're never in effect and visible at the same time. This is great news for composability of functions.

I'm going to try to explain here a little bit of how this works.

Every Function Has a "ParamList" FRAME!

Some time ago I penned the prophetic post: "Seeing all ACTION!s as Variadic FRAME! Makers". This set the stage for what ultimately became an implementation mechanism where the interface to all actions are defined by a FRAME!.

So if you write something like:

foo: func [return: [integer!] x [tag! text!] y [integer!] <local> z] [
    print ["internal foo view:" mold binding of $x]
    return 5
]

Inside of FOO there is a FRAME! that lays out a map of the parameters and locals. This is called the "ParamList". Internally, it looks something like this:

&[frame! [
    return: &[parameter! [integer!]]
    x: ~(&[parameter! [tag! text!]])~  ; bedrock
    y: ~(&[parameter! [integer!]])~  ; bedrock
    z: ()
]]

This isn't an "execution" frame for the function. X and Y don't hold legitimate values for a function invocation...they are holding BEDROCK_0 parameters. RETURN is a special slot known to FUNC which it will fill in with a function, so it looks specialized (e.g. not bedrock), but it holds the return type as a plain parameter!. Z is a local, so it holds the value that it will have when a frame is made. (more on that in a second)

So now let's try making an ordinary frame for the function:

>> f: make frame! foo/
== &[frame! [
    x: ()  ; dual
    y: ()  ; dual
]]

Okay, that's neat. It doesn't seem to have the RETURN or Z fields because we aren't supposed to be setting those. They are there--the memory is part of the frame, and part of what will actually be backing the variables when you EVAL the frame function. But they are hidden.

And it also doesn't seem to have the PARAMETER! values. But they are there...hidden "underneath" the veneer of a GHOST!. You can use special accessors to dig out the parameter:

>> get meta $f.x
== \~,~\  ; antiform (ghost!) "void"

>> get:dual $f.x
== &[parameter! [tag! text!]]

I put code inside the function to print out its internal view of that same frame. Let's try running and see what it says:

>> f.x: "Hello"

>> f.y: 1020

>> do f
internal foo view: #[frame! [
    return: ~&[frame! [^value :run]]~
    x: "Hello"
    y: 1020
    z: ()
]]

Hey, look at that. When we see the frame from inside the function, it has access to RETURN and Z. How does it know to hide the fields on the outside, but give access to them on the inside?

The answer is that each FRAME! Cell instance can optionally hold a "Lens". A Lens is itself is a ParamList. The Lens informs which of the fields are supposed to be visible.

Now, Let's SPECIALIZE It...

Let's make a new function SPFOO which fixes the value of Y.

spfoo: specialize foo/ [y: 304]

And now let's look at what its internal "fake" exemplar FRAME! looks like:

&[frame! [
    return: &[parameter! [integer!]]
    x: ~(&[parameter! [tag! text!]])~  ; bedrock
    y: 304
    z: ()
]]

Something you'll notice is that the type information for Y is now lost, and the slot where the type information would have been has been replaced by the specialized value. That's a nice little efficiency trick. (We can still check the type, because FOO's ParamList has it.)

Now if we make a frame for SPFOO, the only thing it will let us set is X:

>> f: make frame! spfoo/
== #[frame! [
    x: ()  ; dual
]]

What if We Were to ADAPT the Specialization?

So this raises an interesting question about the "inside" and "outside" view of things.

At an interface level, I would argue that it should not usually be possible to tell the difference between SPFOO and any other function that takes a single parameter X.

So what happens if we ADAPT the SPFOO function and get access to the frame on the inside?

adspfoo: adapt spfoo/ [
    print ["inside adaptation:" mold binding of $x]
]

>> adspfoo "What happens?"
inside adaptation: make frame! [
    x: "What happens?"
]
internal foo view: make frame! [
    return: ~#[frame! [^atom :run]]~
    x: "What happens?"
    y: 304
    z: ()
]

Ta-da. ADAPT only saw a function with an X parameter, and none of the other details are exposed to it. Its view of the frame only sees X. But it's all the same frame... memory is being reused, just the access to it is controlled.

Pretty slick, huh? Anyway, I'm sure there are bugs but the groundwork is there. Please experiment and let me know if anything seems to be counterintuitive.

(Note that when you're inside the ADAPT, you don't have access to RETURN. It's not part of the interface, and we want you to be able to ADAPT functions that don't have RETURN. If you need greater control, use ENCLOSE.)

2 Likes
2

The following might not ultimately turn out to be a great idea. But it's an idea I'm giving a shot to.

I'm trying to make an option available for easily pushing parameters down through the stack, which is "frame tunneling".

So that's to say you can capture the view of a function at a level where certain variables are visible, and pass that frame down to a lower level that is expecting it.

For instance, let's make a function that you can optionally pass a frame of an augmented function to:

lower: func [x :augmented [frame!]] [
    print "running lower"
    compose [x (if augmented [spread reduce [augmented.y augmented.z]])]
]

And then, let's make a higher level wrapper that adds more arguments:

higher: adapt (augment lower/ [y z]) [
    print "running higher"
    augmented: binding of $y
]

So what you get is:

>> lower 10
running lower
== [10]

>> higher 10 20 30
running higher
running lower
== [10 20 30]

I wanted to mention this issue, because now that we have this rather effective "functions are black boxes to the outside" mechanic, it's throwing a wrench into some hacks that were able to get away with seeing things they shouldn't have been able to see. So more formal methods of exchanging information between higher layers and lower layers of purposefully collaborating functions are needed.

1 Like
3

So this is the "simple" version of the explanation. The actual truth is more complicated due to optimizations...and complicated to the point where I need to talk it out to make sure I understand it.

Running an ACTION! Requires Two Parts

  • Part #1 is a ParamList, that describes the arguments and layout of the FRAME!

    • it's basically an OBJECT! that contains some specially marked PARAMETER! cells that indicates they are unspecialized

    • other cells are the preloaded values...these can be either specializations of things that were once function arguments, or things that never were arguments (like locals). These can be regular PARAMETER! and that is fine.

  • Part #2 is a Details, that holds the function's C Dispatcher and an array interpreted by that dispatcher.

    • For instance: a FUNC makes a Details with a pointer to the Func_Dispatcher() C function, as well as the BLOCK! of code to execute. All FUNC use the same dispatcher, but the Details hold different blocks.

The Parts Are Split As An Optimization

At one point in time, every ACTION! pointed to a Details*.

So if you were to specialize a function, that would make a new ParamList with the specialized values. Then it would make a new Details which used the Specializer_Dispatcher(), and pointed at the old ACTION! to run.

I sidestepped this by realizing that the new ParamList itself contained the entirety of what the action needed to do differently. Hence, the ACTION! could just point to that (so long as the ParamList pointed to its "previous paramlist" in a linked-list kind of way).

This gave rise to the Phase*... which is a pointer that can be to either a Details* or a ParamList*

So specialization is relatively cheap--by creating only a new ParamList, but no Details.

Un-Lensed FRAME! (hence also ACTION!) Can Hold Symbols

It's very nice to be able to have frames and actions cache a symbol, so we get better information than just "anonymous" out of things like APPLY:

apply append/ [[a b c] [d e] dup: 2]

All APPLY gets there is a FRAME!. So the word "APPEND" would be lost if we didn't store it in the Cell somewhere.

So in the Cell slot where a Lens* can go, we can put a Symbol*... and consider this to be an "un-lensed" frame.

Being un-lensed means "only the inputs are visible to an enumeration". An un-lensed frame is not an executing frame, so you shouldn't expect to see any locals or other implementation details.

ParamList*-Lensed FRAME!

If you use a ParamList* to Lens a frame, then that means the fields you want visible are the input fields of the ParamList.

ADAPT uses ParamList-lensed frames when running the PRELUDE code. For example:

 >> foo: func [x] {local: 1000 print [x + local]}

 >> foo-plus-10: adapt foo/ [x: x + 10]  ; should not see LOCAL

 >> foo-plus-10 10
 == 1020

That LOCAL should not be seen by FOO-PLUS-10. Since ADAPT creates no new ParamList, the only ParamList-lens available is Foo's... and it contains [x local]... but we know that X is an unspecialized argument hence it should be visible.

Self-Lensed FRAME!

Many places in the system have full Cells, which have a lens. But for efficiency internally, most functions put themselves into the binding chain directly... e.g. the VarList itself is the binding chain element, linked in.

What we'd like to have in this case would be some signal that all the fields are visible. But we already know that ParamList-lensed FRAME!s are taken for meaning just inputs. And we have only a VarList, but no details... so what now?

This gives rise to the exception case... If a VarList is used as the lens and it's the same as the VarList being lensed, that means make everything visible according to the archetypal ParamList stored in the 0 Slot.

(I don't expect that to make sense to anyone reading--I'm putting it here just to remind myself how it works. But basically this means that it's the "inner" parameter list; if you've stacked on a bunch of ADAPTs and AUGMENTs and SPECIALIZEs onto a native, only the final phase will use this mode... where the APPEND instantiation itself uses its own VarList as the binding chain element, but the [0] element of that VarList has been tweaked down to where it refers to the APPEND phase itself. So asking that [0] cell will get you a view on the variables that has no knowledge of all the phases above.)

Walking Through The Mean AUGMENT

Let's look at this guy up close:

APPEND is an action specified by a Details*, which points to a ParamList* for the native.

So the FRAME! cell for APPEND holds a Phase that's a Details*, and a symbol for APPEND in the place where a Lens would be (for a running frame).

AP10 is a specialization, which creates a new ParamList... but has no unique Details of its own.

So the FRAME! cell for AP10 holds a Phase that's a ParamList*, and a symbol for AP10 in the place where a Lens would be.

AUGMENT creates an augmented ParamList, but like specialization it creates no new Details. This new ParamList "seals" the old VALUE (by setting a flag on it in that slot of its copy) and adds a new VALUE as an unspecialized PARAMETER!.

WOW is an adaptation, which needs to create a new Details to hold the body of [insert series value], and an Adapter_Dispatcher() that knows how to run it.

So the FRAME! cell for WOW holds a Phase that's a Details*, and a symbol for WOW in the place where a Lens would be.

When WOW runs, it needs to evaluate [insert series value] under a binding to a FRAME!...and that frame has to be Lensed. Remember that the underlying data of the frame has two VALUE slots.

But it needs a Lens that says "you're privileged only to see the inputs of WOW." If AP10 has locals, you don't want those to be visible. That Lens is the ParamList of the augmentation.

Note: There are some pretty impressive tests that show this all working!