Syntax highlighting breaks when trying to Show whitespace and tab characters for C/C++ files (v3.3.1+)

[nehalsharmap4]

Summary
When enabling the “Show whitespace and tab characters” option, syntax highlighting stops working correctly for .c and .cc files.
This issue starts appearing in v3.3.1 and persists in higher versions. It works fine in v2.4.1.

Steps to Reproduce

1. Use react-diff-view v3.3.1 or later
2. Load a diff containing .c or .cc files.
3. Enable Show whitespace and tab characters.
4. Observe that syntax highlighting disappears or becomes inconsistent.

Expected Behavior
Syntax highlighting should remain functional even when whitespace and tab characters are shown.

Actual Behavior
When the whitespace view is enabled, the new tokenization merges whitespace into syntax tokens, breaking syntax highlighting for .c and .cc files.

Notes

• Not reproducible in v2.4.1.
• The regression is likely related to the new tokenization system introduced in v3.3.1.
• Simpler file types (like .js, .py) are unaffected.
• Consistently reproducible on Chrome (latest).

Environment

• react-diff-view: v3.3.1 and above
• Browser: Chrome (latest)
• Affected file types: .c, .cc
• OS: macOS / Windows

Suggested Action
Review the whitespace tokenization logic introduced in v3.3.1.
Consider separating whitespace visualization from syntax token merging to restore proper highlighting for C/C++ grammars.

[nehalsharmap4]

Hi @otakustay @verheyenkoen @benallfree @bodinsamuel ,
Just checking in for an update on this issue.
Please let me know if you need any additional details from my end.

[otakustay]

Did you mean that you are programmingly use react-diff-view and try to enabel whitespace highlight via markWord enhancer?

Can you provide a code example for how you render the diff view, or a screenshot on our demo site?

[nehalsharmap4]

Yes,
Below is the code snippet how I am using it

const options = {
        highlight: syntaxHighlight,
        refractor,
        enhancers: [
            markEdits(hunks),
            ...(showWhiteSpace ? [
                markWord("\r\n", "carriage-return-newline"),
                markWord("\r", "carriage-return"),
                markWord("\n", "newline", ""),
                markWord("\t", "tab", ""),
                markWord(" ", "space", "")] : []),
        ],
        ...(language ? { language } : {}),
    };
    try {
        return tokenize(hunks, options);
    } catch (ex) {
        return undefined;
    }

[nehalsharmap4]

Here you can see that whitespace and tab characters are visible for .h files, but not for .c files.
Also, syntax highlighting seems to be disabled for the .c file type.

I’ve attached a screenshot for reference.

[otakustay]

Sorry but still I'm not able to reproduce this, here is my reproduction code:

import 'core-js/stable';
import {createRoot} from 'react-dom/client';
import {StrictMode} from 'react';
import {Diff, Hunk, markEdits, markWord, parseDiff, tokenize} from 'react-diff-view';
import 'react-diff-view/style/index.css';
import refractor from 'refractor';
import {formatLines, diffLines} from 'unidiff';
import './index.css';

console.log(refractor.listLanguages());

const a = 'void headers();';
const b = 'void headers();\n    void headers();'
const diffText = formatLines(diffLines(a, b), {context: 3});
const [file] = parseDiff(diffText);

const options = {
  highlight: true,
  refractor,
  language: 'clike',
  enhancers: [
      markEdits(file.hunks),
      markWord("\r\n", "carriage-return-newline"),
      markWord("\r", "carriage-return"),
      markWord("\n", "newline", ""),
      markWord("\t", "tab", ""),
      markWord(" ", "space", ""),
  ],
};
const tokens = tokenize(file.hunks, options);


function App() {
  return (
    <Diff tokens={tokens} hunks={file.hunks}>
      {hunks => hunks.map(hunk => <Hunk key={hunk.content} hunk={hunk} />)}
    </Diff>
  )
}

const root = createRoot(document.body.appendChild(document.createElement('div')));
root.render(
    <StrictMode>
        <App />
    </StrictMode>
);

And css:

.diff-code-mark-space::before {
  content: ".";
}

I see whitespace is repalced by a "." charactor

I'd like to have a more reproducible case