Add OpenMP kernels by mcbarton · Pull Request #389 · compiler-research/xeus-cpp

mcbarton · 2025-10-07T13:31:33Z

Description

Please include a summary of changes, motivation and context for this PR.

Fixes # (issue)

Type of change

Please tick all options which are relevant.

Bug fix
New feature
Added/removed dependencies
Required documentation updates

codecov-commenter · 2025-10-07T13:35:33Z

Codecov Report

❌ Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.51%. Comparing base (3002a8e) to head (41df734).

Files with missing lines	Patch %	Lines
src/xinterpreter.cpp	92.30%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #389      +/-   ##
==========================================
+ Coverage   81.25%   81.51%   +0.26%     
==========================================
  Files          21       21              
  Lines         864      871       +7     
  Branches       78       80       +2     
==========================================
+ Hits          702      710       +8     
+ Misses        162      161       -1

Files with missing lines	Coverage Δ
src/xinterpreter.cpp	`91.61% <92.30%> (+0.99%)`	⬆️

Files with missing lines	Coverage Δ
src/xinterpreter.cpp	`91.61% <92.30%> (+0.99%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mcbarton · 2025-10-09T08:57:09Z

Things done so far

Add c++ + openmp kernels (includes kernel definition json files added, and cmake change)
Updated documentation so that LD_LIBRARY_PATH is defined
Add example c++ + openmp notebooks which have been tested locally to work (taken from xeus-clang-repl)

List of things still to do

Add c + openmp kernels
Update ci so LD_LIBRARY_PATH is defined and openmp kernels can be tested
Add some tests for the openmp kernels

vgvassilev · 2025-10-09T10:58:34Z

We should probably have a openmp specific ci without having to rely on the ld_library_path..

mcbarton · 2025-10-09T11:02:30Z

We should probably have a openmp specific ci without having to rely on the ld_library_path..

@vgvassilev Having an openmp specific ci will not avoid the need for ld_library_path. To avoid ld_library_path you would need to rework the cmake and the kernelspec (which makes reference to this variable) , and the example notebooks would need to know the path where you stored the openmp library. Using ld_library_path like xeus-clang-repl did simplifies this PR.

vgvassilev · 2025-10-09T15:55:44Z

We should probably have a openmp specific ci without having to rely on the ld_library_path..

@vgvassilev Having an openmp specific ci will not avoid the need for ld_library_path. To avoid ld_library_path you would need to rework the cmake and the kernelspec (which makes reference to this variable) , and the example notebooks would need to know the path where you stored the openmp library. Using ld_library_path like xeus-clang-repl did simplifies this PR.

Okay, let's see if that's compatible with the vision of @SylvainCorlay which he expressed recently.

mcbarton · 2025-10-09T19:23:50Z

test/test_xcpp_kernel.py

+        #include <omp.h>
+        #include <iostream>
+        #include <clang/Interpreter/CppInterOp.h>
+        Cpp::LoadLibrary("libomp");


Cpp::LoadLibrary("libomp"); works in the notebooks, but doesn't appear to be working in the python kernel tests. I believe this because the tests fail with the same error that I get if I remove this line from the notebook (see https://github.com/compiler-research/xeus-cpp/actions/runs/18385849945/job/52383968040?pr=389#step:13:1015 for error). @vgvassilev @anutosh491 Any idea what may be going on?

My only guess is that the kernelspecs have the flag -fopenmp=libomp , but the python output has the fopenmp flag without the =libomp (see https://github.com/compiler-research/xeus-cpp/actions/runs/18385849945/job/52383968040?pr=389#step:13:922 ). I don't know where to change it so that it has -fopenmp=libomp too though.

Can you call the Cpp::LoadLibrary("libomp") in a separate declare operation?

I don't know if you was suggesting splitting this code into 2 executions, one which loads the library, and one which executes the openmp code, but that has worked, and we now have a passing test. Its a simple OpenMP test, but hopefully it will suffice.

mcbarton · 2025-10-10T08:00:37Z

@vgvassilev @alexander-penev @Vipul-Cariappa @anutosh491 @SylvainCorlay I now consider this PR ready for review. It adds c and c++ kernels that have openmp. It includes example notebooks for the c++ + openmp kernels taken from xeus-clang-repl which were able to run when I tried locally. It also has one simple openmp test for the jupyter kernel test tests. I wasn't exactly sure what would suffice as an automatic test, but the test clearly demonstrates the kernel is executing OpenMP code.

Vipul-Cariappa

According to these changes; The user after selecting a omp kernel, still needs to include CppInterOp.h and call Cpp::LoadLibrary(...). The user should not be expected to know the internals and do this.
When we launch a kernel that enables OpenMP, it should automatically load all necessary stuff. You need to parse the compiler arguments, check if it contains -fopenmp, if yes, call LoadLibrary.
Also, can you share the contents of kernel.json files that gets installed.

Stretch goal (may not be possible): Can we get it to work, without modifying the LD_LIBRARY_PATH env var?

mcbarton · 2025-10-14T20:45:26Z

I think I have an idea of how to avoiding needing ld library path. My new method should work on the Windows once we have that platform working too. I just need to test it out, and plan to test it out sometime in the next few days.

I'm not so convinced that making the user have to include CppInterOp.h, and calling cpp::LoadLibrary(...) is as big deal as you make out to be. It would also let the user know to use other (non openmp) shared libraries in their notebooks. The example notebooks are there for a reason. I kept this PR the way xeus-clang-repl did openmp kernels. Despite this I will look into working out to have the kernel automatically do this if it notices -fopenmp in the kernel arguments.

CMakeLists.txt

Vipul-Cariappa

I too believe the CMake changes may not be required. At least the way it is implemented now. But we can track it in an issue, and fix it later.
Reasoning: OpenMP is not used by xeus-cpp itself (at least at build time), it is just an optional runtime dependency to the interpreter instance. Nowhere in xeus-cpp code we have #include <omp.h> ...

Otherwise, LGTM!

SylvainCorlay · 2026-02-04T09:36:22Z

On the packaging end, we will probably split the conda recipe in multiple outputs (so that installing xeus-cpp does not include the OpenMP kernelspecs unless e.g. xeus-cpp-openmp is installed).

I would also like to highlight that the Jupyter Enhancement Proposal was specifically created to address the C++ kernel use case. It has undergone extensive reviews, and our team produced a reference implementation covering kernel, server, and frontend components. The JEP is currently in the final voting phase.

Given that the official standardization process is nearly complete, it is disappointing to rush a workaround while the long-term solution is so close to being finalized and was developed specifically with the C++ kernel in mind.

vgvassilev · 2026-02-04T09:55:25Z

On the packaging end, we will probably split the conda recipe in multiple outputs (so that installing xeus-cpp does not include the OpenMP kernelspecs unless e.g. xeus-cpp-openmp is installed).

I would also like to highlight that the Jupyter Enhancement Proposal was specifically created to address the C++ kernel use case. It has undergone extensive reviews, and our team produced a reference implementation covering kernel, server, and frontend components. The JEP is currently in the final voting phase.

Given that the official standardization process is nearly complete, it is disappointing to rush a workaround while the long-term solution is so close to being finalized and was developed specifically with the C++ kernel in mind.

Do we have an ETA for this?

github-actions

clang-tidy made some suggestions

github-actions · 2026-02-11T10:34:58Z

src/xinterpreter.cpp


 void* createInterpreter(const Args &ExtraArgs = {}) {
  Args ClangArgs = {/*"-xc++"*/"-v"};
+  std::string resource_dir;


warning: no header providing "std::string" is directly included [misc-include-cleaner]

src/xinterpreter.cpp:23:

- #ifndef EMSCRIPTEN + #include <string> + #ifndef EMSCRIPTEN

github-actions · 2026-02-11T10:34:58Z

src/xinterpreter.cpp

  if (std::find_if(ExtraArgs.begin(), ExtraArgs.end(), [](const std::string& s) {
    return s == "-resource-dir";}) == ExtraArgs.end()) {
-    std::string resource_dir = Cpp::DetectResourceDir();
+    resource_dir = Cpp::DetectResourceDir();


warning: no header providing "CppImpl::DetectResourceDir" is directly included [misc-include-cleaner]

src/xinterpreter.cpp:20:

- #include <algorithm> + #include <CppInterOp/CppInterOp.h> + #include <algorithm>

github-actions

clang-tidy made some suggestions

github-actions · 2026-02-20T08:59:46Z

src/xinterpreter.cpp


 void* createInterpreter(const Args &ExtraArgs = {}) {
  Args ClangArgs = {/*"-xc++"*/"-v"};
+  std::string resource_dir;


warning: no header providing "std::string" is directly included [misc-include-cleaner]

src/xinterpreter.cpp:23:

- #ifndef __EMSCRIPTEN__ + #include <string> + #ifndef __EMSCRIPTEN__

vgvassilev · 2026-03-09T19:01:30Z

On the packaging end, we will probably split the conda recipe in multiple outputs (so that installing xeus-cpp does not include the OpenMP kernelspecs unless e.g. xeus-cpp-openmp is installed).
I would also like to highlight that the Jupyter Enhancement Proposal was specifically created to address the C++ kernel use case. It has undergone extensive reviews, and our team produced a reference implementation covering kernel, server, and frontend components. The JEP is currently in the final voting phase.
Given that the official standardization process is nearly complete, it is disappointing to rush a workaround while the long-term solution is so close to being finalized and was developed specifically with the C++ kernel in mind.

Do we have an ETA for this?

@SylvainCorlay it’s been a couple of weeks can we move forward with this or the alternative is ready?

mcbarton · 2026-03-09T20:26:59Z

Just a note in case it is brought up, the ci failing here doesn't mean we are not in a position to go. We are just waiting on a CppInterOp release to make it go green.

JohanMabille · 2026-03-10T07:46:04Z

On the packaging end, we will probably split the conda recipe in multiple outputs (so that installing xeus-cpp does not include the OpenMP kernelspecs unless e.g. xeus-cpp-openmp is installed).
I would also like to highlight that the Jupyter Enhancement Proposal was specifically created to address the C++ kernel use case. It has undergone extensive reviews, and our team produced a reference implementation covering kernel, server, and frontend components. The JEP is currently in the final voting phase.
Given that the official standardization process is nearly complete, it is disappointing to rush a workaround while the long-term solution is so close to being finalized and was developed specifically with the C++ kernel in mind.

Do we have an ETA for this?

@SylvainCorlay it’s been a couple of weeks can we move forward with this or the alternative is ready?

Let's move forward with this and refactor when the alternative is ready.
We can split the packages on conda-forge to avoid too many kernels for users who simply want vanilla C++.

anutosh491

Quick question : Can we just start with 1 OMP kernel instead of starting with 6 ?

Why maintain 6 when we are planning to refactor into 1 paramterized kernel spec !

Looks like a permutation and combination of language * versions * omp to me.

Even if there's time before the parameterized kernel spec lands, I think its just too much to maintain :\

So if we land a cuda kernel as was being discussed recently on discord, we again land 6 more kernel ?

github-actions

clang-tidy made some suggestions

github-actions · 2026-03-12T09:38:37Z

src/xinterpreter.cpp


 void* createInterpreter(const Args &ExtraArgs = {}) {
  Args ClangArgs = {/*"-xc++"*/"-v"};
+  std::string resource_dir;


warning: no header providing "std::string" is directly included [misc-include-cleaner]

src/xinterpreter.cpp:25:

- #ifndef __EMSCRIPTEN__ + #include <string> + #ifndef __EMSCRIPTEN__

github-actions · 2026-03-12T09:38:37Z

src/xinterpreter.cpp

-    } else {
-        std::cerr << "Failed to detect the resource-dir\n";
-    }
+      resource_dir = Cpp::DetectResourceDir();


warning: no header providing "CppImpl::DetectResourceDir" is directly included [misc-include-cleaner]

src/xinterpreter.cpp:9:

- #include <regex> + #include <CppInterOp/CppInterOp.h> + #include <regex>

…tory

mcbarton force-pushed the Add-OpenMP-kernel branch 3 times, most recently from e4af2d6 to 3824a2f Compare October 9, 2025 07:23

vgvassilev requested review from Vipul-Cariappa, alexander-penev and anutosh491 October 9, 2025 07:37

mcbarton force-pushed the Add-OpenMP-kernel branch from 7dda968 to cc7f3c2 Compare October 9, 2025 08:55

mcbarton force-pushed the Add-OpenMP-kernel branch from 6c4d0f4 to dbe5005 Compare October 9, 2025 10:54

mcbarton force-pushed the Add-OpenMP-kernel branch 3 times, most recently from 45a4ee5 to 571023c Compare October 9, 2025 12:43

vgvassilev requested a review from SylvainCorlay October 9, 2025 15:55

mcbarton force-pushed the Add-OpenMP-kernel branch from 1feccb1 to 1b1b9cd Compare October 9, 2025 18:44

mcbarton commented Oct 9, 2025

View reviewed changes

mcbarton force-pushed the Add-OpenMP-kernel branch 2 times, most recently from 5d1ad18 to 5a2630d Compare October 10, 2025 07:47

Vipul-Cariappa requested changes Oct 14, 2025

View reviewed changes

mcbarton force-pushed the Add-OpenMP-kernel branch 3 times, most recently from 2dac7b4 to a41ed55 Compare October 15, 2025 12:34

mcbarton commented Oct 15, 2025

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

mcbarton requested review from Vipul-Cariappa and vgvassilev October 15, 2025 17:59

Vipul-Cariappa approved these changes Feb 4, 2026

View reviewed changes

mcbarton force-pushed the Add-OpenMP-kernel branch 4 times, most recently from a889c1b to f7552c9 Compare February 4, 2026 13:00

github-actions bot reviewed Feb 11, 2026

View reviewed changes

mcbarton force-pushed the Add-OpenMP-kernel branch from b055d81 to 15d9486 Compare February 11, 2026 12:02

mcbarton force-pushed the Add-OpenMP-kernel branch 2 times, most recently from bcbe68f to 52721c2 Compare February 20, 2026 08:53

github-actions bot reviewed Feb 20, 2026

View reviewed changes

mcbarton force-pushed the Add-OpenMP-kernel branch 2 times, most recently from 2827a20 to 41df734 Compare February 24, 2026 15:23

mcbarton force-pushed the Add-OpenMP-kernel branch from 41df734 to cf57334 Compare March 9, 2026 18:35

mcbarton force-pushed the Add-OpenMP-kernel branch from e9efb75 to 93e375a Compare March 9, 2026 19:55

anutosh491 reviewed Mar 11, 2026

View reviewed changes

mcbarton force-pushed the Add-OpenMP-kernel branch from 93e375a to ec6207b Compare March 12, 2026 09:30

github-actions bot reviewed Mar 12, 2026

View reviewed changes

mcbarton added 7 commits March 13, 2026 14:09

Modify the conda environment to add openmp library and resource direc…

2b51bd8

…tory

Add OpenMP kernel configs

8ea4eba

Configure kernels in cmake

df8823b

Load libomp library automatically if -fopenmp used

73d5ee1

Add example notebooks

422b74d

Add test for C++ + OpenMP kernels

2e2bae6

Add kernel_protocol_version to openmp kernelspecs (xeus 6.0 requirement)

6debb9e

mcbarton force-pushed the Add-OpenMP-kernel branch from ec6207b to 6debb9e Compare March 13, 2026 14:10

Conversation

mcbarton commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Uh oh!

codecov-commenter commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mcbarton commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vgvassilev commented Oct 9, 2025

Uh oh!

mcbarton commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vgvassilev commented Oct 9, 2025

Uh oh!

mcbarton Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mcbarton Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

vgvassilev Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

mcbarton Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mcbarton commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Vipul-Cariappa left a comment

Choose a reason for hiding this comment

Uh oh!

mcbarton commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Vipul-Cariappa left a comment

Choose a reason for hiding this comment

Uh oh!

SylvainCorlay commented Feb 4, 2026

Uh oh!

vgvassilev commented Feb 4, 2026

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

vgvassilev commented Mar 9, 2026

Uh oh!

mcbarton commented Mar 9, 2026

Uh oh!

JohanMabille commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anutosh491 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 12, 2026

mcbarton commented Oct 7, 2025 •

edited

Loading

codecov-commenter commented Oct 7, 2025 •

edited

Loading

mcbarton commented Oct 9, 2025 •

edited

Loading

mcbarton commented Oct 9, 2025 •

edited

Loading

mcbarton Oct 9, 2025 •

edited

Loading

mcbarton Oct 10, 2025 •

edited

Loading

mcbarton commented Oct 10, 2025 •

edited

Loading

mcbarton commented Oct 14, 2025 •

edited

Loading

JohanMabille commented Mar 10, 2026 •

edited

Loading