Iterating "NEW-LINE" Markers (and BLANK! Lines)

Development Iterators
1

There's a pretty good mechanical idea here where the system knows the difference between a BLANK! that's an actual "blank line" for rendering purposes, and one where it's a "comma".

But if you are running something like a FOR-EACH or a PARSE as a user, is there an easy way for you to tell?

I've proposed "new-line-marker aware iteration":

Iterators are still being contemplated, but the idea is that @iter would be "dereference iterator".

This raises a bit of a question about iterators and NULL. I feel like it would be awfully nice to be able to just say:

while [@iter] [
    ...
    iter: next iter  (or `next $iter`)
]

But in the model I've discussed, iterators can return null as an in-band value, so you have to test them for DONE?...which is a "hot potato".

until [done? @iter] [
    ...
    iter: next iter  (or `next $iter`)
]

The idea with iterators is that it's cheap to get the current value, so you don't need to store it in a temporary value.

Anyway, let's imagine:

>> block: [
      a b 


      c d
   ]

Then:

iter: each-special block  ; or whatever it's called

until [done? @iter] [
    if not @iter [                             ; e.g. newline marker
        next $iter                             ; advance iterator
        if blank? @iter [                      ; maybe blank on own line?
             next $iter                        ; advance iterator
             if not @iter [                    ; another newline marker                     
                 ... handle blank line...
             ]
         ]
    ] 
    ...
    iter: next iter  (or `next $iter`)
]

That's rather complicated. :frowning: You're doing a PARSE-like task on invisible format bits revealed through an iterator.

The design premise--a single BLANK! type rendered by context as either comma or blank lines--pushes the burden away from code that doesn't care, and onto code that does. That seems like a wise choice, but a bit burdensome when you do care.

Though it could be trivial in PARSE. It could handle this for you with a combinator:

 parse block ['a 'b repeat 2 blank!-line 'c 'd]

There could similarly be a BLANK!-COMMA if you wanted to match blanks not on their own line.

I think I'm just having a little bit of a mental disconnect with the idea of manually writing code to recognize the pattern a b a being a peer level of difficulty as recognizing a blank line.

But the rationale is that the reason it is difficult is precisely because most code doesn't want to care and hence introducing multiple kinds of blanks isn't a good idea... kind of like why putting the newline markers themselves in source isn't a good idea.

2 Likes
Too Cool To Not Take Advantage Of: BLANK! Lines
2

I've suggested that TAG! combinators would generally match at a position but not move the parse position...and that could come in handy here.

 parse block ['a 'b repeat 2 [<noneline> blank! <noneline>] 'c 'd]

NONELINE is the best name I've come up with, and the idea of an empty SPLICE! that contains a newline marker being a way to append a newline marker is a weird concept. But it should be noted that today the newline markers are disregarded by parse matches, so by default trying to match NONELINE would need some kind of "newline-significant-match" switch. :frowning:

The idea that the <noneline> combinator would not advance the input would actually make this a viable way to recognize a blank line, regardless of the processing before and after.

blank!-line: [<noneline> blank! <noneline>]

But a dedicated BLANK!-LINE combinator would be faster, and probably worth having.

A BLANK!-COMMA combinator is a little weirder... it's any BLANK! that isn't a BLANK!-LINE

blank!-comma: [not ahead blank!-line blank!]

Again, faster to just write that as a quick and dirty combinator.

Weird Though This Is, I Think It's Right

It does mean simple code will go around turning blank lines into commas:

block: [
    a b

    c d
]

>> map-each item block [item]
== [a b, c d]

But you'd be losing your newlines generally?

block: [
    a b  
    c d
]

>> map-each item block [item]
== [a b c d]

Given that, I feel like it's a feature and not a bug that when the new-line markers go away, your BLANK! "lines" degrade into BLANK! "commas".

That kind of closes the case for me--it makes the unification seem pretty solid.

1 Like
4

How about making it possible to give arguments on iterator creation?

Making it possible to say you're only interested "normal" values, or only values and blank lines, blanks in a line, but not blank lines. This way you only need to pay for the values you really are interested in.

1 Like
5

I vaguely recollect PARSE/ALL when you had to explicitly ask to see whitespace in strings. The filtering was found to be problematic. Though maybe there's a more solid foundation for building opt-in filtering logic these days to where it could be the right answer.

...especially because I'm increasingly thinking fully reified NEW-LINE'd BLANK!s is probably the only sensible choice.

Let's consider a world where the following is not legal, due to newline significance and ELSE not being able to get its left hand side across a newline:

Then let's say someone comes up with a special meaning for ELSE with "nothing" on its left. (as there's meanings for RETURN with "nothing" on its right).

If you copy a block cell-by-cell (without special handling) you lose the newline. So you can change the semantics via what looks like a normal copy.

I think this points to the need for a BLANK! in cases like the above.

At minimum, we'd need to guarantee that if you copy the above cell-by-cell without tending to formatting what you wind up with is:

if condition [
   bunch of code here
], else [
   bunch of code here
]

This would imply that a BLANK! with a new-line marker on it, followed by content on the same line doesn't render anything:

if condition [
   bunch of code here
]  ; <-- there's a blank! at the end of this line...
else [  ; <-- ...or maybe it's at the start of this line
   bunch of code here
]

But if we're doing that, why not just preserve the new-line flag by making it not a formatting flag at all, but just the payload of a BLANK!.

That gets us back to this situation if you don't want newlines:

 block: [ \
     a b c \
     d e f \
     g h i \
 ]

That sucks, so the alternative could be a notation for saying "don't scan with blanks"

>> block: \[
       a b c
       d e f
       g h i
   ]\
== [a b c d e f g h i]

Maybe inside the bracket would make more sense when used with quasiforms, given that outer backslashes are used to represent antiforms at the moment...though that might not be a great idea anymore. (maybe vertical bars?)

>> splice: ~[\
       a b c
       d e f
       g h i
   \]~
== |~[a b c d e f g h i]~|  ; antiform (splice!)

Or Accept The BLANK!s, Solve It With Iterators

In the "solve it with iterators" worldview, you would write what you wanted as source:

 block: [ ; <-- blank! here
     a b c  ; <-- blank! here
     d e f  ; <-- blank! here
     g h i  ; <-- blank! here
 ]

That block would have 4 more cells in it than we are used to.

But at least it would be honest.

And as @IngoHohmann says, maybe it's just your pick to say whether "I want to experience this block without the blanks" when you use an iterator.

We might be able to get away from the uncomfortable terminology of NEW-LINE by asking the inverse of blanks... "are you comma-like?" If you're not comma-like, you're new-line like.

Definitely Different, Possibly Inevitable?

The hiddenness of the new-line flag makes it weird and hard to handle, when I think the reality is:

  • people care about it (and always have)
  • the evaluator semantics are irreversibly starting to care about it

It costs slightly more to have cells for it (4 platform pointers a newline, vs being free.) And it also means that the evaluator pays a little more to process it... although that's mitigated somewhat by the fact that you can just test the same byte as you test for anything else vs. having to bit-mask and look for a flag.

I'm not certain about this yet, but reifying newlines as BLANK! seems like it could be the way forward.

1 Like