I've given lip service to the importance of a stepwise debugger for a very long time.
But it's a hard problem--in particular because dialecting means there's no fixed "source language". I showed a methodology of attack in the Visual UPARSE Debugger but it didn't have the ability to switch into a kind of "assembly view" where you could start stepping into the usermode evaluator code that implemented it...which was the aspirational goal.
The crystallization of various pieces of the system is at a point where it doesn't make sense to keep going without making some progress here. So I thought I'd take a step back and start attacking the problems blocking the next level of demo.
The Trampoline Is The Centerpiece
One thing about switching to Stackless is that it brought about a Trampoline-Based Architecture.
This means that when an evaluation needs to be performed from within (most) natives, instead of going directly through a C function call that always puts you perpetually deeper into a machine stack... you instead "push a continuation (request)" and make a C return from the function that wants the evaluation.
Once the evaluation result is ready, the C code is re-entered with the answer. So each native that wants evaluations on its behalf becomes a little state-machine that has to remember why it asked, and what it's going to do next.
The Trampoline is the logical hook for a generic debugger. It means that the debugger code is not itself forced to be deep within an interpreter stack with many frames over its head. And evaluation requests are a meaningful granularity at which to get a hook.
Does "No Return to Trampoline" Mean "No Debug Step"?
There are some places in the system that don't return to the trampoline, but invoke a nested trampoline.
In particular this happens in natives based on the external API. If you are in such a native and you write:
int sum = rebUnboxInteger("1000 + 20");
printf("The sum was %d\n", sum);
The trampoline that called that native doesn't see the +, because an all new trampoline is started.
I think the answer is that this is just not visible to any debugger of the trampoline above them, and effectively makes such natives a black box.
If you want to step into them, they have to be written in the continuation style.
Let's say there's a native-local variable called sum, that starts out null:
if (rebNot("sum")) ; e.g. sum is null
return rebContinue("sum: 1000 + 20");
printf("The sum was %d\n", rebUnboxInteger("sum"));
Your code would wind up looking more like that. The local variables aren't reset on each continuation, so the sum being assigned becomes the signal that the function is being continued. You could set this up with more states.
In such a situation you wouldn't see the sum fetches in the debugger, just the sum: 1000 + 20 evaluation.
It may be that if you start off your whole evaluation with one big API call, that the debugger could exist inside that... but I don't think there's any "cross-trampoline debugging". You write your natives cooperatively through continuations or you don't get debugging.
The "Executor" Reflects Out The Stack Level
Stack levels are actually called Level inside the C code. Each Level has what's called an Executor.
When you think about the kinds of things you'd expect to see in a stack dump tool--you might want to be able to see things like "if it's a function invocation, what's the name of the function". This means something has to know how to turn the "Describe this Level" into digging into the part of the memory that holds the function name. I believe this is dispatched on a "per-Executor" level.
There's executors like the Action_Executor() which knows how to run functions (including natives), the Stepper_Executor() which knows how to run individual evaluation steps, and the Evaluator_Executor() which runs sequential evaluation steps (and does things like handle the vanishing of void steps).
I deliberately broke the Stepper_Executor() and the Evaluator_Executor() into separate Levels because I wanted to have a debugging granularity of single steps. This means that conceptually a stepper Level* can comes into existence with an identity and be able to run.... producing a result, and being "complete", that represents a single step.
Does A Level Have To "Complete" To Get A Step?
I hadn't really thought about it--but I'd been making an implicit assumption that within one Level, a continuation isn't the granularity of a debug step. I was assuming a debug "step" happens when a result is synthesized... no matter how many internal states a Level goes through.
That was the rationale between the Stepper/Evaluator split; if it weren't the case that spawning a Level's existence was the "API" for exposing a step, you'd need some other hook to say what "buttons" there were to push on a Level.
But how well does this idea hold up with other examples? Let's take the example of COMPOSE. When a COMPOSE starts it's just an ordinary Action_Executor(). But then it breaks into levels that run a Composer_Executor() which recursively breaks down each layer of the COMPOSE (only one if it's not COMPOSE:DEEP.
If you're stepping into a COMPOSE, what granularity do you expect? e.g.:
compose:deep [a b (1 + 2) [c [d (3 + 4) e [f g]]]]
Let's say "I step in" to that. The implementation creates a Composer_Executor() for the outermost block, which then looks for GROUP!s and spawns an Evaluator_Executor() when it finds one. It seems to me that at minimum the "step in" should put you cued up to the (1 + 2) before that is executed, so I'd expect at minimum to see an "execution point" there:
compose:deep [a b (1 + 2) [c [d (3 + 4) e [f g]]]]
-^-
There is probably some granularity that should let me step through such that I can evaluate and say "1 is 1", and then maybe "begin +", and then "2 is 2" and then a step telling me that one plus 2 yielded 3. And of course there should also be a granularity that lets me just step over the whole expression.
Less obvious is whether the composer should be offering some kind of "internal" state to tell us "I started a new Level." e.g. do we expect to be able to get execution points like:
compose:deep [a b (1 + 2) [c [d (3 + 4) e [f g]]]]
-^-
compose:deep [a b (1 + 2) [c [d (3 + 4) e [f g]]]]
-^-
compose:deep [a b (1 + 2) [c [d (3 + 4) e [f g]]]]
-^-
compose:deep [a b (1 + 2) [c [d (3 + 4) e [f g]]]]
-^-
If it tells you when it enters these Levels, it should also tell you when it exits them and what the result is... although the "result" is something that makes sense only in the language of COMPOSE (including possibly "no substitution sites, don't make a copy").
At its extreme, you could imagine it not just being Levels of recursion that where you got COMPOSE to reveal its thoughts. It could advertise every array element it looked at to tell us "I'm not doing anything here, because it's doesn't match the pattern of what I'm substituting".
...Exposing These States Is Not Free (Though Shortcuts Exist)
Every time you return to the Trampoline from an Executor, there's cost--even if it just calls you right back. You have to process the state to get you back to what you were going to do before the yield.
One possibility is to make yielding conditional on debugging. This is actually something that I've been building in, even without a functioning debugger--sometimes Level creation is skipped altogether if a result can be cheaply evaluated (most notably with Intrinsics, but other places as well). I have it sporadically throw in a non-optimized call in checked builds just to prove the debugging scenario works.
But COMPOSE is a good talking point just to ask about how many of these "step points" should be exposed. It doesn't immediately seem useful to offer a step at every element in a composed block--though it's hard to say no one would want to.
At least as far as the current implementation goes, you'd get what seems like useful debug granularity if the trampoline offered access to just the events of a Level beginning, and a Level ending with a result.
This is kind of like what the visual PARSE debugger does... it receives a FRAME! and runs an operation on it. This operation can include just skipping it, or turning off notifications and doing an evaluation and getting a result, or evaluating it with a notification received on the next evaluation.
Hence I think what one would expect would be that you'd have the opportunity to know about each Composer_Executor() Level in the COMPOSE that got pushed... with the opportunity to do similar things (although being able to arbitrarily skip an executor is not something that the system is built for at this time...maybe I should reconsider that).
More Granularity Questions
There's interesting questions about an expression like:
x: y: z: add (add 1 2) 3
-^-
So let's imagine you're positioned right there at the start, with some visual indication you are at the X.
What do you think should happen if you press:
- Step In?
- Step Over?
- --other buttons you imagine?--
One might imagine a very fine granularity, in which you get just a move to the Y: assignment
x: y: z: add (add 1 2) 3
-^-
If you expect that, then one probably also expects a "stack trace" tool to show a stack level for the X: assignment above you.
But as I've said, these kinds of full reifications make optimizations tough.
I've Been Focusing On Simplification
What became clear to me as I looked at the situation was that I had too many distinct control signals, and too many specialized Xxx_Executor(). These needed to be pared back--especially if each new Executor() has to be stylized to answer questions about the Level* to which it is attached.
So that's what I'm looking at right now. I want to build the "Hello World" of modern debug stepping, and ideally I want that demo to be able to show some amount of coherence when crossing between UPARSE frames and ordinary function calls and things like COMPOSE.
It's seriously challenging to attack this--and maybe with all the other innovations in the system I might have to say that interactive debugging to the degree I want it is a bridge too far. But I want to at least bring myself to make the effort.