Internals

The project allows multiple OS threads to call asio::io_context::run(), so lua VMs can jump from one thread to another freely, but they will always refer to the same asio::io_context and each will be protected by its own ASIO strand.

You must specify a lua module name to run in the new VM, not a function. The module will be loaded and run in the new VM.

The only way for two different lua VMs to communicate is message passing. The channels are given when you instantiate the extra VMs. The channels accept a range of different values and will deep-copy them. You can also send references to IO objects, but the original references will be rendered unusable (their metatables are unset). Do pay attention to not let objects that have pending operations to be sent over (EBUSY, but do create an error code just for that).

Nor synchronization primitives (such as mutex) nor fiber handles can be sent over the channels and by implication can’t be used to synchronize (or send cancellation requests to) fibers running in different lua VMs.

You can also send a channel over a channel. This will only send the channel “address” over and will allow complex routing among the lua VMs. If you send a channel’s rx-end, the other side will receive a tx-channel anyway. On the C++-side, we need to implement a MPSC strand-based channel.

These characteristics should be enough to implement actor patterns. And it is not the job of emilua to enforce good patterns on applications. The patterns can be configured purely in the lua side of coding.

Leaving the actor model aside for a moment, it’s now easy to have threads with work-stealing (e.g. 8 lua VMs sharing the same asio::io_context running on 4 threads) so you don’t have to worry about load-balancing.

When you issue some IO operation (including chan:receive()), the calling fiber will suspend, but other fibers from the same lua VM are allowed to kick in (cooperative multitasking). Fibers can share state with each other safely (and free from contention problems) as-if the program was single-threaded.

You can use the fiber handle just like you’d use a thread handle. There is join(), detach() and interrupt().

All sync primitives obey some characteristics thanks to the restrictions we’ve laid out:

Given these conditions, it’s now easier to implement and reason about the C++ code.

Only the C++ code that suspended the fiber can resume it back. If the operation should be cancellable, the async op should set an interrupter before suspending the fiber. No other code from the runtime will wake this fiber up. Once the interrupter is called, it’ll be cleared automatically to prevent further complications on the async op implementation. The completion handler should also clear the interrupter to make sure it won’t be (wrongly) reused for other operations.

A good level of serialization can be done by exploring these properties and simplify the implementation a lot. For once, you know no other code will wake the fiber up, so you can just as well call io_obj.cancel() on the interrupter and map asio::error::operation_aborted to errc::interrupted on the completion handler. A single handler (and no other) will take care of waking the fiber. There is no race to deal with here or anything alike.

A lot of the boilerplate is handled already on the prologue/epilogue functions from vm_context.

Besides the common practices to create custom objects through userdata, Emilua (IO) objects will also:

Let’s begin with require().

require()'ing a module is also an async operation which will suspend the caller fiber. Every module has its own isolated environment (i.e. a new lua thread is created for every module and that thread’s environment is configured to use a separate lua table) sharing the same lua VM. The module’s entry point is an user-provided source code evaluated to prepare the environment with the names that should be exported to the caller fiber. But this preparatory step may not be immediately ready and may need to call other async operations. The rule we define to mark a module as loaded and ready is when its main fiber finishes (synchronization code similar to fiber:join()).

To further enforce a more manageable project layout, it is only allowed to import new modules from the main fiber. This may introduce a “slow” startup in some project layouts, but:

Modules evaluate only once and are cached. We never unload them. We keep a reference to their lua thread for as long as the lua VM is active.

Loading a module forms a loader-loaded relationship. This relationship builds a chain that must be checked when a new module is require()d (so we can for instance prevent cyclic imports). But each module will have its own environment. This means the C++ function that implements require() needs to check lua-hidden state associated with the caller lua function (not a global one). That’s the module system state per-module.

We can’t store the module system data directly at the thread environment because lua code can change the thread environment by calling setfenv(0, table).

We’ve already gone through the trickiest parts and added the most important restrictions to the table (no lua-related pun intended), so the remaining rules should be quick’n’easy to catch.

When you initiate an async operation, the C++ side will copy the lua_State* to handle the completion (or cancellation) later. However, any LUA_ERRMEM will trigger an emilua-call to lua_close() and L may then be invalid when we later try to resume it. So the completion handler need to check whether the vm is still valid before accessing it and this is the purpose of the vm_context structure (also protected by the same strand as the vm).

As long as lua code is executing, there is a current fiber and this property stays unchanged for as long as control doesn’t return to host.

This property is mostly transparent to lua code. Which is to say that the programmer is aware of this property, but there isn’t a tangible object that it can track back to this_fiber. This is mostly true, but there is a quite tangible this_fiber lua global object that the user can inspect — exposed at the beginning of the first thread execution.

However, this_fiber being a global is shared among all the fibers, so it can’t point to a single fiber. Instead, it will query which fiber is current and do operations on it.

C++ async ops will always store which fiber is current to know how to resume it back. And before a fiber is resumed, this info is stored at a know lua registry’s index so future async ops will get to know about it too. The reason why we can’t rely on the L argument passed to C functions registered at the VM and the current fiber needs to be remembered is because there will be a L that points to the wrong lua thread as soon as the user wraps some function in a coroutine.

This design works well because we don’t mix responsibilities of the scheduler with user code (as is the case for Fiber#resume in Ruby which would be better suited by a Fiber#spawn() that accepts post/dispatch execution policies and would avoid the (un-)parking unsound ideas altogether).

Some events are intrusive and will be generated even when no thread/fiber asked for them. The classical example are UNIX signals. A sighandler must be registered to handle them, but that begs the question: from which thread are these functions called? In the C world there are multiple answers:

Generally the need for asynchronous events spurs from bad design and should be avoided. However when integrating lua code to existing libraries we must deal with asynchronous events now and then. Emilua reserves a lua coroutine/thread for which no suspension is ever allowed and that will give the lua user a mix between SIGEV_SIGNAL and SIGEV_THREAD restrictions. From the handler the user can notify a condition variable to achieve friction-less handling from a different fiber similar to what SIGEV_KEVENT enables.

From the C++ side, one just needs to get the asynchronous event (lua) thread and rely on lua_pcall() (no need for complex lua_resume() handling, nor fiber APIs).

Lua code cannot recover from allocation failures. As an example (and single-VM only):

If the VM fails to allocate the closure passed to scope_cleanup_push(), my_mutex will be kept locked and the lua code inside that VM will be in an unrecoverable state. There’s no pattern or ordering to make resource management work here as allocation failures can happen almost anywhere and we then inherit some constraints and reasoning from preemptive scheduling. The only option (and this applies to any allocation failure reported by the lua VM when running arbitrary user code) is to terminate the VM from the C++-side.

When lua_close() is called, there is no guarantee pending operations will be canceled as they might hold strong references to the underlying IO object preventing its destructor from getting called. Therefore, the vm_context structure also holds an intrusive container of polymorphic elements which are destroyed after lua_close() is called and can be used to register cleanup code to avoid such leaks. If the operation finishes, the IO object is free to reclaim their own objects from this container and use them for other purposes.

lua_CFunction objects should never call lua_close(). If they detect LUA_ERRMEM all they have to do is to mark the flags field from vm_context and suspend the fiber. The host will take care of closing lua_State* and extra cleanup when it recovers control of the thread.

The other side of the coin is to detect LUA_ERRMEM. All interactions with the VM from the C API happens through the virtual stack, so naturally that’s the first concern. You must not push anything on the stack if there’s no extra free stack slot available. To check for such slot space, there’s lua_checkstack().

The usual C function signature is not enough to convey all the semantics required by the Lua C API. On the Functions and Types section from the manual, we verify the following information:

The 5.1’s signature for lua_checkstack() is:

That’s obviously bogus. If lua_checkstack() can throw on ENOMEM that means there is no possible safe interaction with the VM. That’s — plain and simple — a bug. This bug was fixed in Lua 5.2 when the signature changed to:

You don’t always need to call lua_checkstack() before doing anything thanks to at least LUA_MINSTACK free stack slots being guaranteed for you when the VM calls into your lua_CFunction objects. And here’s where things start to get tricky. Consider the following Lua code:

The underlying C function implementing spawn() is exposed to 3 different lua_State* handles:

If lua_error() is called on L, the stack for L will be in a completely deterministic state. Anything this lua_CFunction object pushed on the stack will be popped and the whole pcall()-chain on the state L will be respected too. However lua_error() might be called indirectly through other API functions. That’s the signature for lua_newtable():

As we’ve seen previously:

“Throw” here means sorts of a call to lua_error() (LUAI_THROW to be more accurate). That’s the pcall()-chain and each lua_State has its own (this property won’t change even if you compile the Lua VM as C++ code). This independent pcall()-chain for each lua_State is not a limitation from the C API, but an accurate model of the underlying machinery happening in Lua code itself. Consider the following snippet:

If c1 is suspended in the middle of pcall(), it retains this private pcall()-chain that doesn’t get mixed with pcall()-chains from other coroutines (i.e. the other lua_State* handles). Therefore the C API accurately maps the language behaviour on retaining a private pcall()-chain for each lua_State and we can’t expect any different behaviour here really. Lua documentation on the issue has been ironed out little-by-little throughout its releases. Lua 5.3 was the one to finally explicitly state the behaviour we just described:

In short, that means our spawn() implementation that is exposed to the {L, current fiber, new fiber} triple would throw to the wrong pcall()-chain if it calls lua_newtable(new_fiber). The solution is to use lua_xmove() when necessary and maintain rigorous discipline as to which C API functions are called on “foreign” lua_State* handles paying very special attention to their respective throw specifications. As for the discipline required, Rici Lake wrote a good summary on the lua-users wiki:

On a side note, Lua 5.2 added the following:

That’s actually behaviour that already existed on the version 5.1. An alternative panic function could just throw a C++ exception to implement this __attribute__((noreturn)) behaviour. However this hypothetical panic function is not an alternative solution to our problems due to the combination of the following facts:

That covers our policy when implementing lua_CFunction objects. In short, we cannot resort to Lua panics here and the only real solution is the rigorous discipline on C API usage mentioned earlier.

Now let’s talk about our policy for host code. The Lua suspending IO functions are implemented by querying which fiber is current and scheduling a lua_resume() on it as the callback for some Boost.Asio supported C++ async_*() function (plus a ton of other details properly documented elsewhere on this document such as strand handling and so on). The initiating function is called from the Lua VM, but the callback is not. The callback will act as the host.

Back to lua_resume(), this function itself doesn’t throw:

However the code that runs before lua_resume() might throw. This is the code that pushes the arguments to the coroutine. For instance, if a string is one of the coroutine parameters, you will have to use C API that might throw on ENOMEM:

It’s no use trying to call lua_pcall() to wrap lua_pushlstring() here. lua_state() now returns LUA_YIELD and that means you can’t use lua_pcall() on this lua_State* handle. You can’t create a new handle and use the lua_xmove() trick either as lua_newthread() itself can throw on ENOMEM:

Fear not, for here is the place where we can finally use a panic function to throw a custom C++ exception. There are only two caveats. The first one is related to LuaJIT having such tight integration with native exceptions that it makes (almost) no distinction between lua_pcall() and C++ catch frames^[2]. The net result is that you can use C++'s catch-all blocks and then no panic function will ever be involved (by now you must be feeling that we just travelled to the farthest candy shop in the kingdom just to make a full-turn just one block away from destination when we changed our minds and decided to go on the neighbour’s candy shop). Despite the lack of a real panic function throwing our own exceptions, I’ll still use the same previous terminology (i.e. panic-triggered exceptions).

The second caveat is a little charming race to avoid. The completion handler doing the host job is executed through the strand that protects the VM. If we let the exception escape the completion handler, another thread might try to use the VM before we have the chance to close it. In other words, the following approach has a race and thus is not used:

Therefore, it is responsibility from the completion handler to handle the panic-triggered exception (sorry about the boilerplate on your side, but that’s the way it is).

That is enough to cover the policy for host code and finally finish the LUA_ERRMEM discussion too.

The biggest challenge to cross-VM resource management are the multi-strand sync primitives (i.e. the channels). They have to execute code that jumps from one strand to another to finish their jobs. If the associated execution context already finished, then they would be stuck forever. The solution is for them to keep the execution context busy through a work guard.

However some rules are needed to make this work:

If the tx-channels are not closed, they will prevent execution contexts that are no longer necessary from being destroyed. But that’s the best we can do. We could periodically call the GC to free unused channels, but so will lua code anyway and there’s nothing left for us to do on the C++ side. A good practice for lua code would be to add the following chunk at the beginning of the fiber who’s gonna process the actor messages:

Extra rules for channels management:

C++ exceptions must not be used to propagate errors across lua/C++ frames. However, lua errors may simply trigger stack unwinding (the code makes heavy use of setjmp()) and we do depend on RAII to keep the code correct.

It is assumed that any call to lua_error() will behave as-if it throws a C++ exception (thus triggering our destructors). We require some support from the luaJIT VM for this. Specifically, we can’t rely on the “no interoperability” category from their “exception” section on the “extensions” page because the following restriction:

To make matters worse, the feature we do depend on only appears in the the “full interoperability” category:

A different approach would be to implement an exception mechanism in terms of coroutines (although it’d add to code complexity):

But this path would be a dead-end as native lua errors would still be reported through lua_error(). For luaJIT, lua_error() plays well with our code because:

Wasn’t for this guarantee, the project would be monstrous. To understand why this guarantee is important, let’s unravel the fundamental pattern for fibers support. We always implicitly wrap every user code inside a lua coroutine:

So async operations can suspend the calling fiber and resume them later.

But user_fn might very well contain a pcall() and execute our suspending async function inside it:

The exception mechanism should not block our ability to suspend fibers. When our own native code calls lua_yield() to suspend a fiber, the suspension mechanism should be able to cross the pcall() barrier.

To wrap all up so far, the standard lua exception mechanism is used to report errors. The only difference is that emilua will lua_error() a structured error object inspired by std::error_code for our own errors.

Things would get a little tricky on the following point that we raised previously though:

Imagine we have some code like the following:

If an object of this type has its destructor called on lua_error()-triggered stack unwinding, it means we’re manipulating the lua_State* (luaL_unref(L) in this example) on stack unwinding (i.e. outside of a lua-catch block which would be just after a pcall() return). If the VM is not in a safe state for manipulations at this moment (this scenario just doesn’t happen if you stick with plain C which is the target lua was developed for) then we’re screwed. Luckily, the VM can handle such situations just fine as it is hinted on the luaJIT documentation:

This guarantee is promised again (although this version of the promise is read-only) in their “extensions” page (and again only at the full interoperability category):

The final piece for our puzzle is related to async ops converting std::error_code into lua exceptions (i.e. lua_error()). The completion handler for async ops is not called in a lua context, so they cannot just call lua_error() and hope the correct context will catch the exception (there’s no API similar to resume_with() from Boost.Context). They need to return control to the native code that suspended the fiber so it can throw a lua exception before control returns to lua code.

This guarantee used to exist on luaJIT 1.x (which included Coco):

The lack of allocated C stacks brings more complications to the implementation that will be discussed later. lua_yieldk() from Lua 5.2 would be enough for us (and cheaper!), but we don’t have that either.

Yet another option would be to set an one-time hook to be called immediately just before resuming the lua coroutine, but it’d present challenges in the future if we ever add debugging support, so it is avoided.

And the solution Emilua get away with is wrapping the C function inside a lua function. The C function returns a 2-tuple. If the first argument is not nil, the lua function itself will take care of use it to raise an error.

Let’s jump straight to a topic that gives some sense of continuity to the previous section. The pcall() barrier is not the only barrier that the user can insert to prevent lua_yield() from suspending the fiber. The user might very well just wrap calls using coroutine.create():

The problem is solved by exposing a different coroutine module — a small shim over the original one. This version inspects this_fiber's suspension reason (native code or lua code).

Conceptually, the implementation looks like this:

When an exception escapes the fiber stack, the hook registered with sys.set_uncaught_hook() is called. The default hook prints the stack trace to stderr and additionally terminates the VM if the exception escaped from the main fiber. If the custom hook itself fails, the default hook is then called anyway.

Scope handlers are properly popped and called after the hook returns control of the thread to the runtime.

The hook is only called for detached fibers. Therefore, a different behaviour can be chosen for each join()ed fiber. Also, if the fiber isn’t explicitly detach()ed, the hook action will be deferred until some GC round.

There isn’t a pcall block around the whole program. lua_resume is enough and it has the nice property of not unwinding the stack so it can be examined from the error handler. A new lua thread is created to execute the uncaught-hook while it has the chance to examine the unchanged error’ed call stack.

There are plenty of functions that have a lua closure as a parameter (e.g. pcall(), scope(), …​). If we blindly implement them in plain C, they will configure a non-leaf C stack frame which we cannot suspend.

To avoid the C stack frame in the middle of the call-stack altogether, we implement (parts of) these functions in lua, not C. The problem is then how to expose sensitive raw resources that the C functions would use. One of the goals is to not let these resources escape elsewhere.

A quick way to achieve it is by having a lua bootstrap function/chunk to create closures and later change their upvalues through C:

This approach is naive as luaJIT 2.x does not implement some lua functions (i.e. the sensitive raw resources that we want to keep private) as C functions and we cannot feed them as upvalues for the imported bytecode. For instance, we have this behaviour for pcall():

Therefore the lua bytecode won’t be a closure with uninitialized upvalues per se, but a function that receives the private resources and returns the needed closure. It is an extra step on startup, but at least we save some cycles by compiling the bytecode with stripped debug info in the project build stage.

A part of the process environment (e.g. UNIX signals) should be under complete control of the program and no external library should meddle with it. However, no protections will be provided to enforce this good practice.

New actors should inherit generic customization points for the GC (e.g. step count and period) and the JIT. They should also inherit allocator settings, but they must not be prevented from creating new actors with higher allocation quotas (unless of course the global pool is already at its limit).

We use some C functions found only on Lua 5.2+ and/or LuaJIT:

There are projects such as Kepler that offer a port of these functions to Lua 5.1.

luaJIT has a serious 2GB limit that has been fixed on forks. By default, the broken 64-bit addressing mode is hidden behind LUAJIT_ENABLE_GC64. Emilua might consider moving to moonjit if its author don’t try to part away from the lua 5.1 core and keep himself distant from 5.3+ syntactic explosion madness. I don’t like this C++-like culture expanding to lua or other languages (kudos to Go here for avoiding it).

The JIT parameters are also changed from the old defaults:

To defaults based on OpenResty findings:

A recent POSIX standard specified anemic per-thread and per-function locale support, but, aside from this anemic support, C uses the same locale globally for the whole process.

Meanwhile, C++ has somewhat usable support for multiple locales per process (and an extra global one that also affects the global C locale).

Functions such as perror() and strerror() will query LC_MESSAGES from the global C locale. However the sole function to query this attribute — setlocale() — is not thread-safe so we shouldn’t change the locale after the program starts and minimal initialization to the process state is done. Changing the global locale is highly unsafe and such API will not be exposed to Lua code.

The thread-safe C++ locales export functionality for LC_MESSAGES through the facet std::messages. This facet allows one to open system-defined message catalogs, and get translation messages for them. This facet exposes no equivalent for the query setlocale(LC_MESSAGES, NULL). Even if we query it at the beginning of the program and try to attach a new custom facet to the global locale object, this will create a nameless locale. Unnamed global C++ locales will break LC_MESSAGES for the C ecosystem (e.g. perror() will no longer print localized messages). Therefore custom facets are out of question.

A direct call to setlocale(LC_MESSAGES, NULL) is avoided too because ISO C++ doesn’t define the macro LC_MESSAGES. To query the current LC_MESSAGES we just look for LC_MESSAGES in the current C++ locale’s name. This approach doesn’t interfere with the C ecosystem, and also paves the way for multiple per-process locales.

One can find the list of POSIX environment variables that affect the process' locale at https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_02. The format for these variables is defined as:

This format is compatible with RDF’s Turtle where LANGTAG is defined as:

And it matches the semantics for BCP47 definition:

The registry of subtags is maintained by IANA at https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry.

So LC_MESSAGES=pt_BR becomes Turtle’s "literal"@pt-BR (and at least the subtag is case sensitive).

For more information about C++ locales, the following links are relevant:

Always keep in mind:

Emilua software is complex. There should be no pursuit in indefinitely extending this base. Rather, we should search for stabilization and maturity (and also tooling around a solid base).

If you think there should be a nice lua library to handle IRC and what-not, by all means do write it, but write it as a separate lua library (or native plug-in), and compete against the free market of libraries. Do not submit a proposal to integrate it in the core. There are no batteries included. And there shall be no committee-driven development.

Likewise, we should be stuck in the current lua syntax (5.1 plus some extensions found in the beta branch of luaJIT 2.1^[3]) forever. If you want more syntax, use a transpiler.