Re-imagining DO/NEXT as EVAL:STEP

Development Internals
1

NOTE: The nature of DO is to deal with whole scripts. We know do/next %foo.r doesn't make any sense, and in fact having DO take a BLOCK! of Rebol code should be a dialect about how to load and bind what you pass it... not expressions like do [1 + 2].

So in modern Ren-C, array evaluation is handled by a primitive called EVAL. Hence the /NEXT refinement has been removed from DO, and for a transitional period DO will not take BLOCK! at all... until all such references are gone. At which point it will take a dialected LOAD spec, probably aligning closely with what IMPORT takes.

Historically, DO/NEXT Took A Variable To Store The New Position

This is how Red and R3-Alpha handle DO/NEXT:

>> result: do/next [1 + 2 10 + 20] 'pos
== 3

>> pos
== [10 + 20]

It was an R3-Alpha-ism, considered more convenient than how Rebol2 gave you a block of both the result and the new position...which you had to pick apart:

rebol2>> do/next [1 + 2 10 + 20]
== [3 [10 + 20]]

(Notice that couldn't work at all in Ren-C, because evaluation can produce antiforms, and antiforms can't be put in blocks.)

First Twist: EVAL:STEP of [] Returns A NULL Position

If you try to step with :STEP over a BLOCK! like [1 + 2 10 + 20], then there are EXACTLY TWO steps with meaningful results of 3 and 30.

So if you're going to be doing the evaluations in a WHILE loop, you want the EVAL:STEP position result to return success twice, and then have a third call that returns null to signal the looping is done.

This gives you the possibly surprising (or not?) result that EVAL:STEP [] doesn't take a step and doesn't synthesize VOID, even though EVAL [] is VOID. It's a terminal condition. So if you're trying to take steps and generate an overall accumulated result, you have to seed your result with VOID... and then EVAL:STEP [] will tell you there was nothing to do and you return your seeded result.

Make sense?

Rebol2, Red, and R3-Alpha all require you to check for the TAIL? of the block as your terminal condition. Because DO/NEXT on a tail position just produces an UNSET! and another tail position.

rebol2>> do/next [10 + 20]
== [30 []]

rebol2>> do/next []
== [unset []]

rebol2>> do/next []
 == [unset []]

That's quite a lot more awkward to handle for a terminal condition. In fact it forces you to check for TAIL? on the block you're evaluating before the first call to DO/NEXT (because seeing the tail afterward won't tell you if the previous step synthesized a valid UNSET!).

R3-Alpha and Red didn't change this, and still make you check for TAIL? before you take steps:

r3-alpha/red>> do/next [10 + 20] 'pos
== 30

r3-alpha/red>> pos
== []

r3-alpha/red>> do/next [] 'pos
; no console result here means unset

r3-alpha/red>> pos
== []

r3-alpha/red>> do/next [] 'pos
; no console result here means unset

r3-alpha/red>> pos
== []

Still very awkward, and unclear why they did this instead of making the POS be #[none].

Second Twist: Ren-C Can Do Multi-Returns!

Now consider EVAL:STEP turning the return result into a parameter pack, where you get both the evaluation product and the new position!

>> block: [1 + 2 10 + 20]
== [1 + 2 10 + 20]

>> pos: eval:step block  ; don't have to heed both returns
== [10 + 20]

>> [pos result]: eval:step pos  ; but you can heed both returns
== []

>> result
== 30

>> [pos result]: eval:step pos
== ~null~  ; anti

>> result
** Error: result is unset  ; <- "true unset", because POS was null

Why Did I Make Position The Primary Return Result?

  1. It Makes It Easier to Loop Through an Evaluation - There are some situations where EVAL:STEP doesn't care about the value synthesized, but pretty much no cases where you don't care about the new position. Being able to conditionally test if the returned position reached the end of a loop is super convenient.

    block: [1 + 2 10 + 20]

while [[block result]: eval:step block] [
    print ["Step result was:" mold result]
]

  2. Avoids Ambiguity When EVAL Result Is Itself A Multi-Return - Imagine the following kind of confusion if we made the evaluation product the first result instead of the second:

    >> block: [1 + 2 comment "I don't care about this"]

>> result: eval:step block  ; I just want the first thing!
== 3  ; great, I didn't want that position anyway

>> block: [pack [<left> <right>] comment "I don't care about this"]

>> [left right]: eval:step block  ; just want to unpack that first thing
== <left>  ; great, just what I expected

>> right
== [comment "I don't care about this"]  ; whaaa? I wanted <right>!

    Encountering problems with this in the past has made me back off from using multi-returns in places they seemed like they would be perfect. But what I now realize is you simply don't want your primary return result of a multi-return to be something that can itself be a multi-return... unless you really know what you are doing.

    If you intend to do something with the evaluation product and want to be truly general, you of course have to be using ^META conventions:

    [pos ^result]: eval:step pos

    Whether you need to do that or not depends on what you are doing. Why are you stepping through arrays one step at a time, anyway? Usually intermediate results are discarded. What is it precisely you are looking for? (Again on my point of why making the position the primary result makes sense... usually you aren't looking at the result at all, you're a dialect and looking at what you advance to at the next position.)

LGTM :+1:

3 Likes
3

UPDATE 2025: This used to have to be:

>> [pos :result]: eval:step pos
== ~null~  ; anti

>> result
== ~null~  ; antiform <- conflated with result of null

But I updated it in light of the new attitude regarding things like VOID assignment

It's still an option to mark the argument as optional via leading-colon, and get null. But if you choose not to mark it, then your variable becomes "true unset" (which cannot be the product of an evaluation).

I think the "true unset" variables strike a balance, giving the desired guardrail while keeping the source looking nice and neat!

It might seem "more dangerously permissive" to not panic by default when you're unpacking a pack with too few values. BUT I think when you look at it holistically, it's probably safer in the long run. Because rather than force you to knee-jerk add a leading colon just to say "yes, I know it might be too few items, make a null if it is"... it's keeping you from conflating a lack of a value with synthesizing an "in-band" null... and I imagine that could wind up being more important, because it catches unintended uses of a meaningless state.

Additionally... consider that a lot of EVAL:STEP calls probably want to be generalized, and process ^META results. If you had to say something was both optional and ^META:

[pos :^result]: eval:step pos

If you can avoid that, so much the better:

[pos ^result]: eval:step pos

Improvements, a little bit at a time...

1 Like
4

I was looking at Oldes R3 which has almost no instances of DO/NEXT (and in fact, Oldes himself doesn't seem to use it in any modules):

https://github.com/search?q=owner%3AOldes%20"do%2Fnext"&type=code

But it has an instance in the REWORD Mezzanine function, inherited from R3-Alpha:

while [not tail? values] [
    w: first+ values  ; Keywords are not evaluated
    set/any 'v do/next values 'values
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

At first I thought "well, it's not that much of an improvement, but... it is one, so I'll take it..."

while [not tail? values] [
    w: first+ values  ; Keywords are not evaluated
    values: eval/step values 'v
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

...BUT WAIT...the loop is doing some kind of "pre-work" at each step. :thinking:

You'd have to have some way of testing for either a block at the tail -OR- a none/null.

Red, R3-Alpha, and Oldes R3 have EMPTY?

>> empty? [a b c]
== #(false)

>> empty? []
== #(true)

>> empty? tail [a b c]
== #(true)

>> empty? none
== #(true)

Note that EMPTY? isn't a question distinct from TAIL? on series, in the sense that it doesn't convey "I'm at the tail of a series that isn't technically empty, if you skipped backwards..."

Rebol2 had a narrower type spec for empty (series! port! bitset!) and didn't take NONE!.

Should Ren-C allow EMPTY? on NULL :red_question_mark:

One of the concepts about NULL is that you don't want to be passing it to things on accident, believing you have something in your hand when you don't.

Should voids be empty, while EMPTY? on null be an error, so you have to OPT it?

while [not empty? opt values] [
    w: first+ values
    values: eval/step values 'v
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

Or if you ask if something is EMPTY? are you expressly wishing to conflate nullness with "an actually empty thing"?

while [not empty? values] [
    w: first+ values
    values: eval/step values 'v
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

As I point out, Rebol2 had a different opinion.

Also, should there be a SOME? test as the anti-empty?

while [some? values] [
    w: first+ values
    values: eval/step values 'v
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

In any case, seeing the situation in practice made me realize that there has to be some way of testing for tail-or-null, in situations where you're iterating over positions in a block and some operations might null the position while others would not.

But the overall point remains: if empty blocks are used as the terminal state, you can't tell if a value was synthesized or not by looking at the state of the block after a step. It's too late. You always have to check for the tail first. And Ren-C has an additional issue with COMMA! that complicates it:

>> eval:step [1 + 2 , , ,]
; first in pack of length 2
== [, , ,]

>> eval:step [, , ,]
== \~null~\  ; antiform (keyword!)

Checking the TAIL? before the evaluation wouldn't tell you that no value was synthesized from those commas. The only way you know is that null was returned.

:face_with_spiral_eyes: