Test arrow without null (DEMO)

[alamb]

Testing apache/arrow-rs#1942

Use code on apache/arrow-rs#1943

Not meant for merging

[alamb]

Failed locally for me:

---- sql::expr::in_list_array stdout ----
thread 'sql::expr::in_list_array' panicked at 'called `Result::unwrap()` on an `Err` value: "Plan(\"Unsupported CAST from Utf8 to Null\") at Optimizing logical plan for 'SELECT\n            c1 IN ('a', 'c') AS utf8_in_true\n            ,c1 IN ('x', 'y') AS utf8_in_false\n            ,c1 NOT IN ('x', 'y') AS utf8_not_in_true\n            ,c1 NOT IN ('a', 'c') AS utf8_not_in_false\n            ,NULL IN ('a', 'c') AS utf8_in_null\n        FROM aggregate_test_100 WHERE c12 < 0.05': Projection: #aggregate_test_100.c1 IN ([Utf8(\"a\"), Utf8(\"c\")]) AS utf8_in_true, #aggregate_test_100.c1 IN ([Utf8(\"x\"), Utf8(\"y\")]) AS utf8_in_false, #aggregate_test_100.c1 NOT IN ([Utf8(\"x\"), Utf8(\"y\")]) AS utf8_not_in_true, #aggregate_test_100.c1 NOT IN ([Utf8(\"a\"), Utf8(\"c\")]) AS utf8_not_in_false, NULL IN ([Utf8(\"a\"), Utf8(\"c\")]) AS utf8_in_null\n  Filter: #aggregate_test_100.c12 < Float64(0.05)\n    TableScan: aggregate_test_100 projection=None"', datafusion/core/tests/sql/mod.rs:626:10
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: core::result::unwrap_failed
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1785:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1078:23
   4: sql_integration::sql::execute_to_batches::{{closure}}
             at ./tests/sql/mod.rs:623:16
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/future/mod.rs:91:19
   6: sql_integration::sql::expr::in_list_array::{{closure}}
             at ./tests/sql/expr.rs:914:47
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/future/mod.rs:91:19
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/future/future.rs:124:9
   9: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:507:48
  10: tokio::coop::with_budget::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/coop.rs:102:9
  11: std::thread::local::LocalKey<T>::try_with
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/thread/local.rs:442:16
  12: std::thread::local::LocalKey<T>::with
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/thread/local.rs:418:9
  13: tokio::coop::with_budget
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/coop.rs:95:5
  14: tokio::coop::budget
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/coop.rs:72:5
  15: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:507:25
  16: tokio::runtime::basic_scheduler::Context::enter
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:361:19
  17: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:506:36
  18: tokio::runtime::basic_scheduler::CoreGuard::enter::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:564:57
  19: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/macros/scoped_tls.rs:61:9
  20: tokio::runtime::basic_scheduler::CoreGuard::enter
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:564:27
  21: tokio::runtime::basic_scheduler::CoreGuard::block_on
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:497:9
  22: tokio::runtime::basic_scheduler::BasicScheduler::block_on
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:173:24
  23: tokio::runtime::Runtime::block_on
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/mod.rs:475:46
  24: sql_integration::sql::expr::in_list_array
             at ./tests/sql/expr.rs:929:5
  25: sql_integration::sql::expr::in_list_array::{{closure}}
             at ./tests/sql/expr.rs:904:7
  26: core::ops::function::FnOnce::call_once
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:227:5
  27: core::ops::function::FnOnce::call_once
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- sql::expr::test_in_list_scalar stdout ----
thread 'sql::expr::test_in_list_scalar' panicked at 'called `Result::unwrap()` on an `Err` value: "Plan(\"Unsupported CAST from Utf8 to Null\") at Optimizing logical plan for 'SELECT NULL IN ('a','b')': Projection: NULL IN ([Utf8(\"a\"), Utf8(\"b\")])\n  EmptyRelation"', datafusion/core/tests/sql/mod.rs:626:10
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: core::result::unwrap_failed
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1785:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1078:23
   4: sql_integration::sql::execute_to_batches::{{closure}}
             at ./tests/sql/mod.rs:623:16
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/future/mod.rs:91:19
   6: sql_integration::sql::execute::{{closure}}
             at ./tests/sql/mod.rs:650:45
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/future/mod.rs:91:19
   8: sql_integration::sql::expr::test_in_list_scalar::{{closure}}
             at ./tests/sql/expr.rs:984:5
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/future/mod.rs:91:19
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/future/future.rs:124:9
  11: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:507:48
  12: tokio::coop::with_budget::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/coop.rs:102:9
  13: std::thread::local::LocalKey<T>::try_with
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/thread/local.rs:442:16
  14: std::thread::local::LocalKey<T>::with
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/thread/local.rs:418:9
  15: tokio::coop::with_budget
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/coop.rs:95:5
  16: tokio::coop::budget
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/coop.rs:72:5
  17: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:507:25
  18: tokio::runtime::basic_scheduler::Context::enter
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:361:19
  19: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:506:36
  20: tokio::runtime::basic_scheduler::CoreGuard::enter::{{closure}}
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:564:57
  21: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/macros/scoped_tls.rs:61:9
  22: tokio::runtime::basic_scheduler::CoreGuard::enter
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:564:27
  23: tokio::runtime::basic_scheduler::CoreGuard::block_on
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:497:9
  24: tokio::runtime::basic_scheduler::BasicScheduler::block_on
             at /Users/alamb/.cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/tokio-1.18.2/src/runtime/basic_scheduler.rs:173:24
  25: tokio::runt
Compilation exited abnormally with code 101 at Fri Jun 24 14:40:50

[viirya]

The failed cases looks not like an attempt to really want to cast other datatype (Utf8) to Null. Looks like an incorrect type coercion rule for NULL IN ('a','b'). It tries to cast from utf8 to null, but seems it is more correct to cast from null to utf8.

[alamb]

The failed cases looks not like an attempt to really want to cast other datatype (Utf8) to Null. Looks like an incorrect type coercion rule for NULL IN ('a','b'). It tries to cast from utf8 to null, but seems it is more correct to cast from null to utf8.

I agree @viirya -- I'll try and find time to look at this later if no one else does. I have a few more things to deal with the fallout from upgrade to arrow 17 #2778

[viirya]

I agree @viirya -- I'll try and find time to look at this later if no one else does. I have a few more things to deal with the fallout from upgrade to arrow 17 #2778

I can take a look in this weekend.

[alamb]

For others -- proposed fix PR is in #2794