Re-imagining DO/NEXT as EVAL:STEP

Development Internals
1

NOTE: The nature of DO is to deal with whole scripts. We know do/next %foo.r doesn't make any sense, and in fact having DO take a BLOCK! of Rebol code should be a dialect about how to load and bind what you pass it... not expressions like do [1 + 2].

So in modern Ren-C, array evaluation is handled by a primitive called EVAL. Hence the /NEXT refinement has been removed from DO, and for a transitional period DO will not take BLOCK! at all... until all such references are gone. At which point it will take a dialected LOAD spec, probably aligning closely with what IMPORT takes.

Historically, DO/NEXT Took A Variable To Store The New Position

This is how Red and R3-Alpha handle DO/NEXT:

>> result: do/next [1 + 2 10 + 20] 'pos
== 3

>> pos
== [10 + 20]

It was an R3-Alpha-ism, considered more convenient than how Rebol2 gave you a block of both the result and the new position...which you had to pick apart:

rebol2>> do/next [1 + 2 10 + 20]
== [3 [10 + 20]]

(Notice that returning a BLOCK! from a generic evaluation in Ren-C would require speaking in lifted values, because evaluation can produce antiforms, and antiforms can't be put in blocks directly. So we have an additional reason to avoid blocks for this kind of thing.)

First Twist: EVAL:STEP of [] Returns A NULL Position

If you try to step with :STEP over a BLOCK! like [1 + 2 10 + 20], then there are EXACTLY TWO steps with meaningful results of 3 and 30.

So if you're going to be doing the evaluations in a WHILE loop, you want the EVAL:STEP position result to return success twice, and then have a third call that returns null to signal the looping is done.

This gives you the possibly surprising (or not?) result that EVAL:STEP [] doesn't make a step happen, and doesn't synthesize VOID, even though EVAL [] is VOID. It's a terminal condition. So if you're trying to take steps and generate an overall accumulated result, you have to seed your result with VOID... and then EVAL:STEP [] will tell you there was nothing to do and you return your seeded result.

Make sense?

Rebol2, Red, and R3-Alpha all require you to check for the TAIL? of the block as your terminal condition. Because DO/NEXT on a tail position just produces an UNSET! and another tail position.

rebol2>> do/next [10 + 20]
== [30 []]

rebol2>> do/next []
== [unset []]

rebol2>> do/next []
 == [unset []]

That's quite a lot more awkward to handle for a terminal condition. In fact it forces you to check for TAIL? on the block you're evaluating before the first call to DO/NEXT (because seeing the tail afterward won't tell you if the previous step synthesized a valid UNSET!).

R3-Alpha and Red didn't change this, and still make you check for TAIL? before you take steps:

r3-alpha/red>> do/next [10 + 20] 'pos
== 30

r3-alpha/red>> pos
== []

r3-alpha/red>> do/next [] 'pos
; no console result here means unset

r3-alpha/red>> pos
== []

r3-alpha/red>> do/next [] 'pos
; no console result here means unset

r3-alpha/red>> pos
== []

Still very awkward, and unclear why they did this instead of making the POS be #[none].

Second Twist: Ren-C Can Do Multi-Returns!

Now consider EVAL:STEP turning the return result into a parameter pack, where you get both the evaluation product and the new position!

>> block: [1 + 2 10 + 20]
== [1 + 2 10 + 20]

>> pos: eval:step block  ; don't have to heed both returns
== [10 + 20]

>> [pos result]: eval:step pos  ; but you can heed both returns
== []

>> result
== 30

>> [pos result]: eval:step pos
== ~null~  ; anti

>> result
** Error: result is unset  ; <- because too few items in multi-return pack

Why Did I Make Position The Primary Return Result?

  1. It Makes It Easier to Loop Through an Evaluation - There are some situations where EVAL:STEP doesn't care about the value synthesized, but pretty much no cases where you don't care about the new position. Being able to conditionally test if the returned position reached the end of a loop is super convenient.

    block: [1 + 2 10 + 20]

while [[block result]: eval:step block] [
    print ["Step result was:" mold result]
]

  2. Avoids Ambiguity When EVAL Result Is Itself A Multi-Return - Imagine the following kind of confusion if we made the evaluation product the first result instead of the second:

    >> block: [1 + 2 comment "I don't care about this"]

>> result: eval:step block  ; I just want the first thing!
== 3  ; great, I didn't want that position anyway

>> block: [pack [<left> <right>] comment "I don't care about this"]

>> [left right]: eval:step block  ; just want to unpack that first thing
== <left>  ; great, just what I expected

>> right
== [comment "I don't care about this"]  ; whaaa? I wanted <right>!

    That actually isn't what happens, because a PACK! with a PACK! in its first slot is considered "undecayable", so you'd get an error. You have to do a nested unpack:

    [[left right]]: eval:step block

    Or get the pack as-is via ^META assignment

    [^left-and-right]: eval:step block

    Encountering problems with this in the past has made me back off from using multi-returns in places they seemed like they would be perfect. But what I now realize is you simply can avoid making the primary return result of a multi-return something that can itself be a multi-return..

    If you intend to do something with the evaluation product and want to be truly general, you of course have to be using ^META conventions:

    [pos ^result]: eval:step pos

    Whether you need to do that or not depends on what you are doing. Why are you stepping through arrays one step at a time, anyway? Usually intermediate results are discarded. What is it precisely you are looking for? (Again on my point of why making the position the primary result makes sense... usually you aren't looking at the result at all, you're a dialect and looking at what you advance to at the next position.)

LGTM :+1:

3 Likes
Should EVALUATE:STEP bomb on an error?
Blocks vs Variadics as Dialect Formats
Multiply-Unpacking Multi-Returns
Meet the REEVALUATE (REEVAL) native
3

UPDATE 2025: This used to have to be:

>> [pos :result]: eval:step pos
== ~null~  ; anti

>> result
== ~null~  ; antiform <- conflated with result of null

But I updated it in light of the new attitude regarding things like VOID assignment

It's still an option to mark the argument as optional via leading-colon, and get null. But if you choose not to mark it, then your variable becomes unset.

I think the unset variables strike a balance, giving the desired guardrail while keeping the source looking nice and neat!

It might seem "more dangerously permissive" to not panic by default when you're unpacking a pack with too few values. BUT I think when you look at it holistically, it's probably safer in the long run. Because rather than force you to knee-jerk add a leading colon just to say "yes, I know it might be too few items, make a null if it is"... it's keeping you from conflating a lack of a value with synthesizing an "in-band" null... and I imagine that could wind up being more important, because it catches unintended uses of a meaningless state.

Additionally... consider that a lot of EVAL:STEP calls probably want to be generalized, and process ^META results. If you had to say something was both optional and ^META:

[pos :^result]: eval:step pos

If you can avoid that, so much the better:

[pos ^result]: eval:step pos

Improvements, a little bit at a time...

1 Like
4

I was looking at Oldes R3 which has almost no instances of DO/NEXT (and in fact, Oldes himself doesn't seem to use it in any modules):

https://github.com/search?q=owner%3AOldes%20"do%2Fnext"&type=code

But it has an instance in the REWORD Mezzanine function, inherited from R3-Alpha:

while [not tail? values] [
    w: first+ values  ; Keywords are not evaluated
    set/any 'v do/next values 'values
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

At first I thought "well, it's not that much of an improvement, but... it is one, so I'll take it..."

while [not tail? values] [
    w: first+ values  ; Keywords are not evaluated
    values: eval/step values 'v
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

...BUT WAIT...the loop is doing some kind of "pre-work" at each step. :thinking:

You'd have to have some way of testing for either a block at the tail -OR- a none/null.

Red, R3-Alpha, and Oldes R3 have EMPTY?

>> empty? [a b c]
== #(false)

>> empty? []
== #(true)

>> empty? tail [a b c]
== #(true)

>> empty? none
== #(true)

Note that EMPTY? isn't a question distinct from TAIL? on series, in the sense that it doesn't convey "I'm at the tail of a series that isn't technically empty, if you skipped backwards..."

Rebol2 had a narrower type spec for empty (series! port! bitset!) and didn't take NONE!.

Should Ren-C allow EMPTY? on NULL :red_question_mark:

One of the concepts about NULL is that you don't want to be passing it to things on accident, believing you have something in your hand when you don't.

Should voids be empty, while EMPTY? on null be an error, so you have to OPT it?

while [not empty? opt values] [
    w: first+ values
    values: eval/step values 'v
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

Or if you ask if something is EMPTY? are you expressly wishing to conflate nullness with "an actually empty thing"?

while [not empty? values] [
    w: first+ values
    values: eval/step values 'v
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

As I point out, Rebol2 had a different opinion.

Also, should there be a SOME? test as the anti-empty?

while [some? values] [
    w: first+ values
    values: eval/step values 'v
    if any [set-word? :w lit-word? :w] [w: to word! :w]
    ...
]

In any case, seeing the situation in practice made me realize that there has to be some way of testing for tail-or-null, in situations where you're iterating over positions in a block and some operations might null the position while others would not.

But the overall point remains: if empty blocks are used as the terminal state, you can't tell if a value was synthesized or not by looking at the state of the block after a step. It's too late. You always have to check for the tail first. And Ren-C has an additional issue with COMMA! that complicates it:

>> eval:step [1 + 2 , , ,]
; first in pack of length 2
== [, , ,]

>> eval:step [, , ,]
== \~null~\  ; antiform (keyword!)

Checking the TAIL? before the evaluation wouldn't tell you that no value was synthesized from those commas. The only way you know is that null was returned.

:face_with_spiral_eyes:

5

I Think MORE? Is A Better Term

I've also been reasoning that we should repurpose AT to give you the current value of an iterator (sort of a "PICK ITER 1" synonym).

So MORE? would tell you that AT would work without giving you an error, and also that NEXT would work without giving you an error.

I think that returning an error state (FAILURE! or "hot potato") that you could turn into NULL with a TRY is better than just having NEXT produce NULL directly. But that may be superfluous. AT certainly can't return NULL because I think that iterators may iterate over values and NULL could be in band with that.

Anyway... I think what this may point to is that if you want "nice" behavior with MORE? and you don't have a series to iterate... you should use NONE (an empty splice) and not NULL.

NULL-permissiveness is not something we want to get into here. And VOID-permissiveness is no longer the right thing either (we have COND/CONDITIONAL if you want to get null answers out of things when the input is null).

This all looks to be tying up nicely. :ribbon:

6

Let me rephrase what this is here as a hybrid iteration and evaluation scenario.

You're walking a block. Sometimes you're calling the evaluator and getting the next position after an evaluation. Sometimes you're doing literal inspection yourself.

I'm finding a few footholds here that I like. I like MORE? as the negation of TAIL? as a test for whether an iterator has more information.

But I also am biased toward the idea that MORE? does not accept nulls. Generally I do not like functions taking nulls and treating them like nothing happened.

We know that when no more values are available, it's best that iterators return a tail position--that is still itself an "iterator". It's more useful than a state that holds nothing: it can be reversed, for instance (assuming the iterator is reversible).

Unfortunately it's not a "falsey" null, because tail iterators are "things". Still, it seems like the tail of the array may be a better "standardized" end of evaluation.

Given that MORE? doesn't take null, if you're in a position where you don't have a series handy and want to force a terminal condition in an upcoming MORE? test, you could use NONE (empty splice) and be a little less misleading than using something like a random empty block.

So... all that said...

If I were to seek holistic consistency here, it makes it seem like EVAL:STEP should do the same thing that NEXT would do.

That means if NEXT gives a FAILURE!, or panics, or returns null... whatever it does, EVAL:STEP should do that too.

Today NEXT returns a FAILURE!. But the bad thing about that is that if your stepping action does something else and returns a result (as EVAL:STEP does) then you would be conflating a FAILURE! coming from the evaluation with a failure of being at the end of the step.

For that matter, you could wind up conflating a panic from the user code with a panic in the step. (Less of an issue, because you could also pass an invalid refinement or argument to EVAL, and so panic conflation is a fact of life.)

I think this means that NEXT of a tail position must either panic or return null, or none.

Historical Rebol (and Red) leaves it at the tail, which I think is a really bad idea:

rebol2>> pos: [a]
== [a]

rebol2>> next pos
== []

rebol2>> next next pos
== []

rebol2>> next next next pos
== []

rebol2>> back next next next pos
== [a]  ; <-- I think this is an empiricaly bad result for that

If we returned NONE we could give you something that doesn't trigger an error from MORE? and such (giving you a falsey result), but that wouldn't give the illusion of reversibility.

>> pos: [a]
== [a]

>> next pos
== []

>> next next pos
== \~[]~\  ; antiform (splice!) "none"

>> next next next pos
== \~[]~\  ; antiform (splice!) "none"

>> more? next next next pos
== \~null~\  ; antiform

>> back next next next pos
** PANIC...  ; I think this is a better answer than [a]

This is really just a long and convoluted brainstorm that boils down to not wanting to make MORE? and TAIL? and such NULL-tolerant.

If the iteration domain is tuned to not speak in terms of nulls for failed operations, but rather NONE, then it can preserve some of the "good" properties of being able to repeatedly advance a tail, without generating bad results and without risking arbitrary nulls from having this effect.

So You Probably Have to Use MORE? In EVAL:STEP Loops

block: [1 + 2 10 + 20]

while [more? [block result]: eval:step block] [
    print ["Step result was:" mold result]
]

I think the only other option is to make things like TAIL? and MORE? null-tolerant, and I'm not really feeling that's a good idea for the health of NULL overall.

But I will have to think about this... more.

The way to write this that looks better is:

block: [1 + 2 10 + 20]

while [more? block] [
    [block result]: eval:step block
    print ["Step result was:" mold result]
]

And this is how you would have to write it if EVAL:STEP errored on tail positions.

If it's tolerant then you get that "no result" evaluation, which is misleading.

Maybe eliminating the misleading thing entirely is the wiser path.

7

However... if we disallowed EVAL:STEP on tails and PANIC'd, we could reformulate this:

block: [1 + 2 10 + 20]

while [more? block] [
    result: eval:step $block
    print ["Step result was:" mold result]
]

We could also say that EVAL:STEP on a tail returned a state (like a ~(done)~ hot potato) which was a conflation, and then it's your job to think about whether that conflation bothered you.

block: [1 + 2 10 + 20]

while [not done? result: eval:step $block] [
    print ["Step result was:" mold result]
]

My bias is ot think that creates more risk than is worth it, because it's an easy mistake to think that your step produced a value when you're at the tail. So you wind up having a bunch of people writing code in this pattern that misbehaves when a step actually produces a ~(done)~ hot potato.

Anyway, passing the variable might be the better EVAL:STEP concept.