I love GENERATORS and YIELDERS, however they have a dark side...
If you are enumerating something with a generator or yielder, and you don't call it to exhaustion, it never cleans up... leaving latent locks on series, and just clogging up memory.
History and experience with serious languages like C++ and Rust have shown that we don't really have better a better answer for default lifetime control than scope.
So let's try a thought experiment.
>> foo: func [data [block!]] [
let g: make-generator [
for-each 'item data [print "Inside!" yield item + 1]
]
return reduce [g g]
]
>> foo [10 20 30 40]
Inside!
Inside!
== [11 21]
Imagine if at the moment of returning from FOO, the incomplete generator G should be destroyed, and its locks released.
OTOH, if you have a module-level variable that's a generator, we can't automatically get rid of it. And setting it to NULL wouldn't be enough to get it synchronously GC'd. You would have to FREE it.
So Many Questions...
There's actually one important mechanism that the system has now, which is the ability to FREE basically anything and have stray references crash the GC.
So if LET wants to, it can FREE whatever it holds when the scope exits.
But now we enter a situation where not just exiting from scope, but a new assignment would have to free the generator too:
foo: func [data [block!]] [
let g: make-generator [
for-each 'item data [print "Inside!" yield item + 1]
]
let data: reduce [g g]
g: 1020
return data
]
Overwriting G with 1020 would synchronously FREE the previous contents of G.
Clearly not all variables being overwritten should free them. If you passed G to another function, and that function made a couple of calls and then overwrote the argument... it shouldn't in the general case mean FOO lost G... unless it did a transfer of ownership.
So maybe this needs to be conveyed with a new concept, like C++'s unique_ptr
.
Let's say the concept is UNIQUE:
let g: unique make-generator [
for-each 'item data [print "Inside!" yield item + 1]
]
When you say that something is unique, you're saying that if that variable slot gets overwritten for any reason... it should free the value. This needs to include cases of exiting scope.
This may have potential, and it may be able to be implemented using the same mechanics as accessor. In other words, it could be usermode... UNIQUE would be an infix function that specifically sets up the left hand side as a slot that runs code on overwrite of the variable.
What About Indefinite Word Lifetime?
This has always been a bit of a thorn:
What happens to FUNCTION! arguments and locals when the call ends?
If LETs and function args are getting wiped clean on function exit... (sometimes clearing out UNIQUE or other accessors, other times having no effect)... then returning code bound to those LETs would not be usable.
There's never really been a satisfying answer to indefinite lifetime. But it's almost like you want to do the opposite of UNIQUE, to tell a variable "hey, I need you to live". Some kind of UNSCOPE operation, or SURVIVE or something.
>> f: func [x] [return [x]]
>> b: f 10
== [x]
>> reduce b
!! PANIC: X is trash ; or whatever, wiped out by function exit
vs.
>> f: func [x] [survive $x, return [x]]
>> b: f 10
== [x]
>> reduce b
[10]
So what SURVIVE would do is stop the system from setting X to trash on exit of F.
This starts to put a lot of little invisible bits on things, which makes me a bit uneasy. But there's no in-band way to do this. The values themselves can't encode information about their lifetimes (e.g. some UNIQUE! antiform which is a box around a value... you'd have to unbox the value every time you used it).
Though... quick devil's advocacy detour ...it is the case that quasiform actions can be accessed and run via
/^g
, and so you might say "What if meta actions were immune to freeing, but normal ones were not." That's kind of dumb, but it does point to the idea that there might be some way of encoding the "don't free me automatically on exiting functions" state in a value itself, if you were willing to use a special decoration to refer to that value to undo whatever you did to make it denote indefinite lifetime.
Maybe this is "meta-fencing" values?
>> f: func [x] [x: generator [yield 10], return [x]]
>> b: f 10
== [x]
>> reduce b
!! PANIC: X is trash ; or whatever, wiped out by function exit
vs.
>> f: func [x] [^{x}: generator [yield 10], return [^{x}]]
>> b: f 10
== [^{x}]
>> reduce b
[10]
The thought here would be something like:
>> ^{x}: 10
== 10
>> x
== {'10}
>> ^{x}
== 10
Then say that these fences keepalive things that would otherwise be wiped out on function exit. Notably you could put quasiforms of generators and such in these.
Not a good idea, I was just playing devil's advocate when I said the only way to talk about modifying a variable's lifetime was with out of band bits...and needed to justify that statement.
Could UNIQUE And SURVIVE Cover Enough Common Cases?
It's hard to say.
But GENERATOR and YIELDER definitely present a challenge, because cleaning them up requires running finalization code. That finalization code might panic. If that happens at an arbitrary moment in time, you'll just get a random error popping out of nowhere.
So in a sense, a running GENERATOR and YIELDER pretty much can't be GC'd--in effect, they hold references to themselves while they're still running. That's among the many reasons why it's important to clean them up intentionally.
They're not the only such entities with this problem. User objects that hold onto resources are in the same category, they need to be shut down.
It seems to me that something needs to be tried, I just wanted to look at the landscape a bit and see if there was anything new in the picture.