Decoupling FFI from LIBRARY!

Domains FFI
1

I've been so gung-ho about finally coming up with a good implementation of the Extension Type Mechanism that I've sort of changed my tune about whether it's worth it to keep around extensions like IMAGE!, VECTOR!, the FFI, or even GOB! and the serial port extension...

But in any case, LIBRARY! is now the best it's been:

>> winsock: make library! %/C/Windows/System32/wsock32.dll
== #[library! %/C/Windows/System32/wsock32.dll]

>> pick winsock "gethostbyname"
== #[handle!]

>> pick winsock "gethostbynickname"
** Error: Couldn't find "gethostbynickname"
      in #[library! %/C/Windows/System32/wsock32.dll]

>> try pick winsock "gethostbynickname"
== ~null~  ; anti

>> close winsock
== #[library! {closed} %/C/Windows/System32/wsock32.dll]

FFI Has A Dependency on LIBRARY!... But...

So the FFI extension is designed to let you interface with C functions from Rebol code.

You've got a C function in some DLL, the FFI lets you wrap that up in something you can call as a Rebol function and it translates the Rebol parameters (e.g. TEXT!) into something the C can take (e.g. char*).

The way Shixin did it, you pass the FFI a LIBRARY! and a TEXT! of a function name:

libgtk: make library! %libgtk-3.so

gtk-init: make-routine libgtk "gtk_init" [
    argc [pointer]
    argv [pointer]
]

But what if MAKE-ROUTINE just took a CFunction HANDLE! ?

libgtk: make library! %libgtk-3.so

gtk-init: make-routine (pick libgtk "gtk_init") [
    argc [pointer]
    argv [pointer]
]

We could make this even better, by letting LIBRARY! accept WORD! as well as TEXT!, and using a TUPLE!-based PICK:

libgtk: make library! %libgtk-3.so

gtk-init: make-routine libgtk.gtk_init [
    argc [pointer]
    argv [pointer]
]

This makes the FFI generalize, to however you get your CFunction.

Maybe you compiled a function in-memory with the TCC extension and want to test it from Rebol?

Maybe you are using a different extension than LIBRARY! for getting DLLs.

Anyway, I was working on trying to resolve dependencies between modules when I realized that this particular dependency isn't necessary!

2 Likes
2

Well, I did find one aspect where knowing the library is useful.

It stored the library from which the function handle originated, and then when went to dispatch the function it checked to see if the library was closed. It would give you an error then, when you called the function, instead of crashing.

But this seems to call for a more general mechanic, whereby the pointers themselves handed out by a library maintain a connection to the library, and go bad when the library is unloaded.

This could be done by having the Cell which a LIBRARY gives back contain a Stub, which is in a linked list with all the stubs that the library hands out. When the library is unloaded, it walks this list and makes all the Stubs go bad. This makes pointers handed out by LIBRARY! slightly more expensive, but there was cost to storing the library in the FFI functions as well.

It would also let you get more information out of a handle if you wanted it, of showing what library it was in.

3

Decoupling FFI from LIBRARY!

This raises another question about what should be coupled or uncoupled...

Should FFI depend on the VECTOR! extension?

The original FFI from Rebol2's paid version (Rebol/Command) didn't seem to support arrays at all:

REBOL External Library Interface

When Shixin implemented support for them, the result you get back is a block, e.g. for C if you had:

struct example {
    uint8_t some_array[1000];
};

If you made an instance of that struct, and then picked instance.some_array you'd get back a BLOCK! of 1000 integers. This would use Cells, so on a 64-bit system:

1000 * (4 platform pointers per cell) * (8 bytes per platform pointer)
-> 32,000 bytes to represent an array of 1000 bytes

But it's worse than that...

Dealing With "Sub-Cell Addressing"

I've previously mentioned a nasty property that PICK mechanics have, which is that because it moves in steps...what if you wrote:

instance.some_array.500  ; pick the 500th byte out of the 1000 byte array

If this goes stepwise and fabricates that block of 1000 INTEGER!, and then picks just the 500th one out of it, then it wastefully created a 32,000 byte BLOCK! for no reason.

The historical "solution" to this problem was that in such cases, STRUCT! would hijack the path picking process itself. Instead of the (instance.some_array) step producing something from which the 500th element would be picked, it would actually "look ahead" and see the 500 coming, and take over the processing.

I've pushed back against that idea, because irregularizing the picking process causes more damage than it is worth. I'd sooner have you write instance.[some_array 500] or similar.

BUT if VECTOR! was willing to act as an interface to in-memory data that it did not own, it could solve this problem. instance.some_array could fabricate a VECTOR! as a light wrapper over memory that could accept the reads and the writes.

Weak VECTOR! Coupling: Only need if you use C arrays?

I think it can be designed in such a way that if you don't have VECTOR! loaded, you can still do basic FFI. Maybe even be able to represent and proxy data containing arrays. But if you want to PICK and POKE out of arrays, then you need VECTOR!.

Or maybe plain old binary values could be used as a last resort, if you don't have vector...and you can mess with the bytes yourself.

I definitely like the idea of the FFI extension being able to work without VECTOR!. But the properties of using BLOCK! for arrays are so bad regarding the subcell addressing issues that I think doing that with VECTOR! makes more sense.