Flaky test fails intermittently in GitHub Action

[johnjaylward]

Test case passes on some runs and fails on others: org.json.junit.JSONObjectTest.issue654StackOverflowInputWellFormed

• Test Run: https://github.com/stleary/JSON-java/actions/runs/6644866305/job/18054932906#step:5:79
• Code Link:

  JSON-java/src/test/java/org/json/junit/JSONObjectTest.java

  Lines 3668 to 3676 in b5b9f63

  	@Test(expected = JSONException.class)
  	public void issue654StackOverflowInputWellFormed() {
  	//String input = new String(java.util.Base64.getDecoder().decode(base64Bytes));
  	final InputStream resourceAsStream = JSONObjectTest.class.getClassLoader().getResourceAsStream("Issue654WellFormedObject.json");
  	JSONTokener tokener = new JSONTokener(resourceAsStream);
  	JSONObject json_input = new JSONObject(tokener);
  	assertNotNull(json_input);
  	fail("Excepected Exception.");
  	}

Appears to be due to differences in resources available on each run. The exception expected is a "JSONException", but that's because it is wrapping a "StackOverflowError". StackOverflowError is going to be dependent on the environment and may not hit consistently.

[rudrajyotib]

I think this test case should be removed, as it never guarantees the behavior. To identify the max depth supported, we can test through configuration. And environment resource related runtime error expectation need not be tested.

[stleary]

Might be better to see if the test can be fixed, although if you see a chance to add a unit test, that is fine too.

[johnjaylward]

@stleary , I'm leaning towards just marking the test as a "skip/ignore" or removing it altogether. It seems like this test would need regular maintenance to ensure that it was consistently hitting stack overflows.

@Ignore("This test relies on system constraints and may not always pass.")
@Test(expected = JSONException.class) 
 public void issue654StackOverflowInputWellFormed() { 
     //String input = new String(java.util.Base64.getDecoder().decode(base64Bytes)); 
     final InputStream resourceAsStream = JSONObjectTest.class.getClassLoader().getResourceAsStream("Issue654WellFormedObject.json"); 
     JSONTokener tokener = new JSONTokener(resourceAsStream); 
     JSONObject json_input = new JSONObject(tokener); 
     assertNotNull(json_input); 
     fail("Excepected Exception."); 
 } 

[stleary]

@johnjaylward Fair enough, better to silence the test, than have it fail intermittently.

[rudrajyotib]

@johnjaylward - are you annotating the test with Ignore?