Enable setting default values for target_partitions and planning_concurrency

[nuno-faria]

Which issue does this PR close?

• None

Rationale for this change

The documentation for the datafusion.execution.target_partitions and datafusion.execution.planning_concurrency settings states that their default values are set to 0 (which in these cases maps to the number of cores). However, using 0 does not translate to their default values, but rather the literal 0, which is invalid in both cases.

In a previous version, setting datafusion.execution.target_partitions to 0 resulted in a single partition, while in the current version it results in a division by zero error. Likewise, setting datafusion.execution.planning_concurrency to 0 causes the system to hang.

With this change, setting datafusion.execution.target_partitions or datafusion.execution.planning_concurrency to 0 will result in the default value being assigned instead, making it coherent with the documentation.

What changes are included in this PR?

• datafusion/common/config - added the translation to the default values.
• datafusion/execution/config - updated the with_target_partitions method.
• sqllogictest/information_schema - added tests.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes -- when setting these configurations to 0 -- but it will match the existing documentation.

[ctsk]

Sensible changes, but I'm unsure of the implementation. I've had a quick look at the config_namespace! macro and see that there is a transform can be specified - Could that be used here?

[nuno-faria]

Sensible changes, but I'm unsure of the implementation. I've had a quick look at the config_namespace! macro and see that there is a transform can be specified - Could that be used here?

@ctsk Something like this?

pub target_partitions: usize, transform = |x: &str| if x == "0" ...

It appears to work well, however with_target_partitions must be changed to call set instead of assigning directly to execution.target_partitions.

Should transform = ... be a closure or a new standalone function?

[alamb]

Makes sense to me -- thank you @nuno-faria and @ctsk for getting the PR to a nicely mergable state

[alamb]

CI failure seems unrelated

• CI is failing on main: error: the lock file /__w/datafusion/datafusion/Cargo.lock needs to be updated #15724

[alamb]

Merged up from main to try and get a clean CI rnu

[alamb]

Thanks again @nuno-faria -- FYI this will be in the 48 release