support data type coerced and decimal in INLIST expr #2755
Description
related issue:
- InList: Can't cast the list expr data type to value expr data type directly #2774
- InList Expr: expr and list values must can be converted to a same data type #2759
- Support
castto/fromNULLandDataType::Decimalarrow-rs#1921 - InList: Don't treat Null as UTF8(None) #2782
- InList: don't need to treat Null as UTF8 data type #2773
- support decimal in (NULL) #2800
- InList: fix bug for comparing with Null in the list using the set optimization #2809
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and why for this feature, in addition to the what)
When I apply the in list filter to the below table
❯ \d food
+---------------+--------------+------------+-------------+-----------------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------------+-------------+
| datafusion | public | food | a | Decimal(10, 5) | NO |
| datafusion | public | food | b | Decimal(20, 15) | NO |
| datafusion | public | food | c | Boolean | NO |
+---------------+--------------+------------+-------------+-----------------+-------------+
❯ select * from food where a in (1);
ArrowError(ExternalError(NotImplemented("InList does not support datatype Decimal(10, 5).")))
explain the query:
❯ explain select * from food where a in (1);
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: #food.a, #food.b, #food.c |
| | Filter: #food.a IN ([Int64(1)]) |
| | TableScan: food projection=Some([a, b, c]), partial_filters=[#food.a IN ([Int64(1)])] |
| physical_plan | ProjectionExec: expr=[a@0 as a, b@1 as b, c@2 as c] |
| | CoalesceBatchesExec: target_batch_size=4096 |
| | FilterExec: a@0 IN ([CastExpr { expr: Literal { value: Int64(1) }, cast_type: Decimal(10, 5), cast_options: CastOptions { safe: false } }]) |
| | RepartitionExec: partitioning=RoundRobinBatch(16) |
| | CsvExec: files=[/Users/kliu3/Documents/github/arrow-datafusion/datafusion-cli/target/debug/aggregate_simple.csv], has_header=false, limit=None, projection=[a, b, c] |
| | |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set. Query took 0.007 seconds.
We need convert the data type of expr and list to the common data type.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.