Refinement Arguments at Head of Args List, Not Tail

[hostilefork]

In the corpus of code we have so far, it seems to me that when a refinement adds an argument to a function that it would be preferable if that argument would become the first parameter... not tacked onto the end.

Some cases might not be completely obvious one way or another:

>> append [a b c] [d e]
== [a b c [d e]]

>> append:dup [a b c] [d e] 2  ; old way
== [a b c [d e] [d e]] 

>> append:dup 2 [a b c] [d e]  ; new idea
== [a b c [d e] [d e]]

I think it's better if it's first, but it's not earth-shattering.

But in other cases it seems very much an improvement. Consider the positioning of the argument to FAIL:BLAME...

foo: func [arg thing] [
    if arg < 0 [
        fail:blame [
           "Here is some long error message:" @thing
           "Whatever..."
        ] $arg
    ]
]

foo: func [arg thing] [
    if arg < 0 [
        fail:blame $arg [
           "Here is some long error message:" @thing
           "Whatever..."
        ]
    ]
]

Or an argument to COMPOSE giving a pattern to use:

compose:pattern [
    some bunch of {{code that}} <spans>
    #multiple lines
    [and could go on for pages]
] ${{}}  ; afterthought...

compose:pattern ${{}} [  ; forethought
    some bunch of {{code that}} <spans>
    #multiple {{lines}}
    [and could go on for pages]
]

This goes along with some Haskell philosophy I cited in Parameter Order in Rebol:

"It's common practice in Haskell to order function parameters so that parameters which "configure" an operation come first, and the "main thing being operated on" comes last. This is often counter intuitive coming from other languages, since it tends to mean you end up passing the "least important" information first. It's especially jarring coming from OO where the "main" argument is usually the object on which the method is being invoked, occurring so early in in the call that it's out of the parameter list entirely!"

These refinements typically seem to be configuring, as if they are changing the function itself, and belong at the head.

e.g. above, the function you're conceptually applying is (compose:pattern ${{}})

History Didn't Do It This Way, With Some Reasons

Refinements are typically listed at the end of the function spec.

From an implementation standpoint, that's also where their "slots" are in the argument list.

This means that as you are walking the argument list and fulfilling arguments from the callsite, if refinements were used you would have to skip over the "normal" arguments in a first pass, and then come back and fill them later.

Historical Redbols only had to be worried about the order of usage of refinements... if you used them out of order from the declaration, a second pass would be needed. But using them in order would not require it.

This isn't a problem for Ren-C...it's designed for generic parameter reordering (refinements or otherwise) and it has an efficient way to beeline back to slots it skipped on a second pass.

So really the only issue is the mismatch between the visual order in the spec (which may be exposed mechanically by fixed orders of enumeration of FRAME! keys and values), compared with the gathering behavior. But the disconnect of that order has always been there, with foo/refine1/refine2 vs. foo/refine2/refine1 in Redbol... the callsite order may not match the frame order.

Is It Worth Changing?

The competing (complementary) idea of CHAIN! dialecting offers something that's likely even more compelling:

compose:pattern ${{}} [  ; better than today...
    some bunch of {{code that}} <spans>
    #multiple {{lines}}
    [and could go on for pages]
]

compose:${{}} [  ; ...but this surpasses even that
    some bunch of {{code that}} <spans>
    #multiple {{lines}}
    [and could go on for pages]
]

Though it's kind of up in the air if and when that's going to get attacked, and how well it will work (it may run afoul of problems in binding, etc.)

My instincts tell me that it's worth changing. In practice, refinements that take arguments are not super common... but when they do happen, being up front seems to make the most sense.

[bradrn]

A thought… might it be possible to do something like this?

append:dup:2 [a b c] [d e]

Basically, this would mean that refinements don’t add new arguments per se — instead such cases can be unified with CHAIN! dialecting.

(I feel certain that we’ve discussed something like this before, but I can’t seem to find it with a quick search.)

[hostilefork]

Using another CHAIN! element was intended to let you drop the refinement altogether, to accomplish briefer things like:

append:2 [a b c] [d e]

But I was thinking TUPLE! could do this more generically:

append:dup.2 [a b c] [d e]

Even so, if you used a variable, it would probably have to be decorated:

append:dup.(n + 1) [a b c] [d e]

append:dup.@n [a b c] [d e]

The relatively "noiseless" injection to the head of the args list feels like a better answer to me:

append:dup 2 [a b c] [d e]

append:dup n [a b c] [d e]

[bradrn]

@hostilefork

But I was thinking TUPLE! could do this more generically:

append:dup.2 [a b c] [d e]

Actually, this was what I wrote first… but then I remembered that CHAIN! binds tighter than TUPLE!, so this wouldn’t make sense. Or am I misremembering? If this can be done, I think it looks better than using CHAIN!.

[hostilefork]

TUPLE! is the lowest (giving the pleasing hierarchy in terms of "heft"...lighter things are tighter).

One of the problems with putting arguments into the chain/tuples is that binding gets tricky. There's some open issues with that, but if you embed arguments into a sequence then you're tying their binding up in the binding of that sequence. It's still a ways out.

I think I still would prefer the "configuring" refinements to put their arguments earlier, it really seems to be preferable in practice.

>> join:with '+ fence! ['a 'b 'c]
== {a + b + c}

[hostilefork]

@hostilefork

In the corpus of code we have so far, it seems to me that when a refinement adds an argument to a function that it would be preferable if that argument would become the first parameter... not tacked onto the end.

As a counterexample... there's DO:ARGS

do %some-script.r

do:args %some-script.r ["hello" "world"]

That particular case has a possible workaround, that since DO is no longer used to evaluate blocks, the args could go in a block as part of a "do spec"

do [%some-script.r, args: ["hello" "world"]]

I don't know, but I thought I'd point out the first example I'd noticed where injecting the argument first wasn't ideal.

Of course, these cases could do what we have to do with configuring refinements that aren't where we want them today, which is to use APPLY

do // [%some-script.r :args ["hello" "world"]]

(Note: I'm still wondering if APPLY should use SET-WORD!s instead of ":refinements", there are tradeoffs for each...)

[hostilefork]

This question of refinement ordering is another one of those "really fundamental things that can't be skipped over in a minimum viable product, even a demo".

So it's been a real source of stress.

As is often the case, sitting back and erasing preconceptions, and asking questions from first principles, may be the way to solve it...

Why Do Refinements Exist?

e.g. why would you make APPEND with a :DUP and :PART option instead of writing separate APPEND and APPEND-DUP and APPEND-DUP-PART functions?

In many languages, you would write APPEND-DUP-PART as the "maximally configurable" core function with a bunch of arguments. Then you'd make specializations like APPEND-DUP call the core function with a default for the PART, and probably APPEND in turn by calling APPEND-DUP with a default DUP of 1, etc.

Advantages that come to mind of the refinement approach over this are:

1. You don't have to remember the order of the optional parameters, you just have to remember their names, and you use them in any order (so long as you make your usage order match).

2. One entry point for documentation, and that entry point is the "natural" name of the function where you have discoverability of what the options are.

3. No need to write out the awkward cascade of specializations.

So at least three good things. But we've seen one rather key weakness:

The design chosen as most palatable for the default usage of the function name without optional parameterization dictates the parameter order, in a way that the refinements cannot override.

When you have individual functions like APPEND-DUP and APPEND-DUP-PART as their own entry points, those aren't beholden to the order that is most natural for APPEND. They can pick what makes the most sense for that "configuration".

And as we've noticed, optional or extra "configuring" parameters frequently make sense to appear earlier rather than later in the invocation. Though perhaps not always.

The historical refinement approach anchors the mandatory parameters in place, and configurations can only be added at the end.

APPLY as the Escape Hatch

It makes a big difference to have modern APPLY and its shorthand of //. This lets you pick whatever order of parameters makes the most sense to you.

I name COMPOSE as a good example because it shows that the "configuration" information is something that has a big impact on understanding the meaning of the operation as a whole:

compose:pattern [
    some bunch of {{code that}} <spans>
    #multiple {{lines}}
    [and could go on for pages]
] '{{}} 

So redoing that with APPLY, you can push the pattern up front:

compose // [
    pattern: '{{}}
    [
        some bunch of {{code that}} <spans>
        #multiple {{lines}}
        [and could go on for pages]
    ]
]

It's not terrible... it does resolve the tension between the desires of the default meaning of the word COMPOSE, and the need to see the configuring parameter before seeing the data operated on.

Using this shorthand you're paying for: // [ ] and a level of indentation, with some need to understand what // means. As opposed to:

apply compose/ [
    pattern: '{{}}
    [
        some bunch of {{code that}} <spans>
        #multiple {{lines}}
        [and could go on for pages]
    ]
]

But, still there's something unsatisfying about saying this is what you have to do.

You Could Fallback To... Making More Functions

The Non-Refinement world says that if the parameter-order needs of your "configured version" are in contention with your "default", make another function:

compose-pattern '{{}} [
    some bunch of {{code that}} <spans>
    #multiple {{lines}}
    [and could go on for pages]
]

And you can try some other naming ideas (I thought of COMPOSE2, to mean, "arity-2 compose", and it's briefer):

compose2 '{{}} [
    some bunch of {{code that}} <spans>
    #multiple {{lines}}
    [and could go on for pages]
]

You've lost your discoverability (HELP COMPOSE won't tell you about COMPOSE-PATTERN or COMPOSE2, the way HELP COMPOSE could find COMPOSE:PATTERN).

But when compared with COMPOSE:PATTERN in a refinements-take-their-args-first world, would COMPOSE2 still win out in usage?

compose:pattern '{{}} [
    some bunch of {{code that}} <spans>
    #multiple {{lines}}
    [and could go on for pages]
]

COMPOSE2 might still be something some people would choose (I doubt everyone). But at least you could just say compose2: compose:pattern/ and get the right behavior.

Positional Refinements: Best Of Both Worlds?

One idea might be that the position of a refinement in the parameter list actually makes a difference. This is something Ren-C can do since "refinements are their own arguments":

 compose: native [
     :pattern [...]
     template [...]
     :deep [...]
 ]

The idea that "refinement order matters" would mean that your callsite should probably be forced in line with that ordering. But is that so bad, considering you're forced in line with non-optional parameters already?

Let's say APPEND put :DUP before the main arguments, but :PART after the series parameter that it is constraining... (I don't know if this is desirable, let's just say it did)

>> append:dup:part 3 [a b c] [d e f] 2
== [a b c [d e] [d e] [d e]]

Now say you gave the order wrong:

>> append:part:dup 3 [a b c] [d e f] 2
** PANIC: Incorrect optional parameter order (should be :DUP:PART)

This then turns the tables, and says that if you want an arbitrary order of parameters that doesn't match the spec you should use APPLY.

Bridges The Non-Refinement World To The Refinement One

This effectively means that you are automatically getting the permutations of functions written for you:

APPEND
APPEND-DUP
APPEND-DUP-PART
APPEND-PART

But only those where the parameters follow the order in the definition.

It still lets you add the refinements after the fact, and you shouldn't get bitten by any callsites that mix and match refinements that haven't been used together before.

From an implementation point of view, this is a vast improvement. Right now, there's bend-over-backwards mechanics to figure out how to align an ordering of refinement usage that doesn't match the frame definition. That code is non-trivial, and removing it simplifies things.

Honestly I Think, I've Just Solved It

Looks like a win to me.

Q: Should non-argument-bearing refinements be subject to interface order as well?

That's tough, but I kind of feel like the answer is "maybe so", if we want the mechanics to be simplest. Though it could probably still be simple as I can think of ways to handle parameterless refinements without requiring multiple passes. I guess I'd have to see how oppressive it winds up being to make it part of the order rule.

[hostilefork]

@hostilefork

In many languages, you would write APPEND-DUP-PART as the "maximally configurable" core function with a bunch of arguments. Then you'd make specializations like APPEND-DUP call that function with a default, and probably APPEND in turn by calling APPEND-DUP with a default DUP of 1, etc.

Advantages that come to mind of the refinement approach over this are:

1. You don't have to remember the order of the optional parameters, you just have to remember their names, and you use them in any order (so long as you make your usage order match).
2. One entry point for documentation, and that entry point is the "natural" name of the function where you have discoverability of what the options are.
3. No need to write out the awkward cascade of specializations.

So at least three good things.

Sacrificing (1) might seem harsh...

But there was also resistance to getting rid of multi-argument refinements.

And I think what we learned in general from that was that multi-argument refinements were basically irrelevant. When that was dropped, the system could move ahead by leaps and bounds in terms of FRAME! mechanics and function specialization/composition.

Along those same lines, I think fretting over the sacred cows of overlong refinement-based function call syntax is a big distraction compared to what empowering dialects has to bring to the table.

e.g. it is rather rare to invoke a function with more than one refinement. And if you really are using multiple refinements (with arguments), the more likely you're dealing with a call that's going to be spanning multiple lines. That's where the added boilerplate of APPLY is a benefit and not a drawback... which will also give you the opportunity to pick your ordering.

I'd actually imagine that code quality would improve if there was a rule saying "only one refinement can be used in a CHAIN!-based call, otherwise you have to use APPLY". Not that such a rule needs to be made, but if you only use one refinement unless you apply you worry less about the order.

This is kind of like saying that if you have lots of refinements that might be used together, you're probably doing something that would be better exposed as a dialect. But APPLY is just a way of slipping into that dialected mode.

@hostilefork

It still lets you add the refinements after the fact, and you shouldn't get bitten by any callsites that mix and match refinements that haven't been used together before.

This is a crucial bit. You can add optional parameters to a function and it won't disrupt existing calls. Anything that lined up before will keep lining up.

[hostilefork]

@hostilefork

APPLY as the Escape Hatch

It makes a big difference to have modern APPLY and its shorthand of //. This lets you pick whatever order of parameters makes the most sense to you.

It occurs to me that historical Rebol APPLY had this backwards. It was APPLY that made you read the precise function spec to line up the refinements:

Function refinements can also be passed in the order they are specified by the arguments spec block. For example, we can see:

>> ? append
USAGE:
    APPEND series value /part length /only /dup count

So in this example we use the /dup refinement:

data: [456]
apply :append [data 1 none none none true 3]
probe data
[456 1 1 1]

Note that the refinement itself must be set to true.

Turning this on its head, means you're forced into awareness of the order of interface only when you're not using APPLY.

I think this makes more sense: that APPLY is the "sophisticated dialect" with advanced processing that has to do more figuring and "multiple passes" to map your intent to the frame, while a plain call in the evaluator is the "blunt and simple" logic for fastest and most aesthetic calls.