Emscripten ABI compatibility checks for side modules?

[hoodmane]

There has been discussion with the CPython folks about trying to add wasm32-emscripten wheels to PyPI. For this, we would like to have some way to decide which wheels are compatible with which main modules. Looking around, I see there used to be EMSCRIPTEN_ABI_MAJOR and EMSCRIPTEN_ABI_MINOR but they were removed.

Suppose we compile a main module and a side module at separate times using possibly different versions of Emscripten. Is there any way to check whether they are compatible? Could such a feature be added? What will happen at load time if they are not compatible?

[sbc100]

As of now we don't have any guarantees like that. Would it be possible to treat the entire emscripten version as the ABI version for now? Or perhaps that would be too limiting? Perhaps if would work if you selected just a few specific emscripten releases to support?

Can you explain the user case a little more? Is is idea that folks who use the emscripten version of python would be able to download pre-built side modules? Is there some way to disable pre-built binaries and have the side module always compile from source instead? Or might these users not even have emscripten itself installed?

[hoodmane]

This is still in an exploratory phase, so we don't have a completely clear set of requirements (and other people involved might have slightly different ideas about what our requirements are).

Can you explain the user case a little more? Is is idea that folks who use the emscripten version of python would be able to download pre-built side modules?

Yeah exactly. The way Pyodide currently works is that we build a bunch of side modules with Python packages in tree with the Python interpreter.

The way the broader Python system works is that different people build wheels for different targets -- packages build and distribute their own wheels, wheels for Raspberry Pi are built by people other than the original author and distributed on https://www.piwheels.org/, etc. In order to try to make this work, wheels are tagged with some platform information which tries to indicate which systems they can run on.

The Python people suggest that in the long term we could add Emscripten as a supported wheel platform, but we would need to define the compatibility more precisely. Ideally we should discuss our plans with the Emscripten team before doing that.

Or might these users not even have emscripten itself installed?

End users probably don't have Emscripten and won't rebuild packages. Many people making applications based on Pyodide have a webdev background struggle with a complicated and unfamiliar toolchain like Emscripten, though if we can make the tools easy to install and use then more people will manage.

Would it be possible to treat the entire emscripten version as the ABI version for now? Or perhaps that would be too limiting? Perhaps if would work if you selected just a few specific emscripten releases to support?

Yeah, these are certainly options.

I guess we have a pretty different use case from typical Emscripten users who are porting fixed applications like games and don't have any need for side module dependency management.

[sbc100]

Don't get me wrong, as stable ABI for dyanmic linking is certainly something we would like to have in the future. It could be that this use case is compelling enough to try to make progress on it.

[akpmilot]

Hi,

The lack of clear ABI compatibility between Emscripten versions is very problematic for us. Our product is game middleware that is pluggable in various game engines like Unity, which supports the Web platform via Emscripten. I'll direct you to Unity's documentation page on external native plug-ins, which is what our product is:

https://docs.unity.cn/2023.3/Documentation/Manual/webgl-native-plugins-with-emscripten.html

In particular, this line in their documentation explains the problem clearly:

If you choose to build plug-in code in advance, you should use the Emscripten compiler toolchain. To ensure LLVM binary format compatibility, the version of Emscripten that’s used to compile the plug-ins must match the version of Emscripten that Unity uses.

Since we distribute pre-compiled binaries for our plug-in, we must provide binaries that are supported across all maintained Unity versions. Each Unity version uses a different version of Emscripten. Because ABI compatibility is not clearly defined between Emscripten versions, it is difficult for us to determine which version of Emscripten we should use to support Unity.

We got bit by this recently as we upgraded to a new minor version of Emscripten, 3.1.52 and then bumped into #20233 when integrating into Unity because they use 3.1.38.

I humbly suggest the Emscripten project revises its approach to versioning to more clearly advertise when ABI compatibility is broken.

[sbc100]

@akpmilot, in a world where we were able to detect ABI breakages and document them, presumably you would still need a process for updating your libraries to match unity's versions right?

With that process in place, can you not work with the current status quo which is basically that each emscripten release should be considered and ABI breaking release? i.e. is this just an issue or degree? You can publish new versions of your library for emscripten releases but you would rather not do it for all releases?