Update Dockerfile to fix integration tests

[andygrove]

Which issue does this PR close?

Closes #1979

Rationale for this change

Fix integration tests

What changes are included in this PR?

Update Dockerfile to use new crates

Are there any user-facing changes?

No

[realno]

Thanks for fixing the tests 👍

[realno]

x-post from another PR:

I checked the pipeline definition the integration test is not part of it. I think at some point we should include it. @andygrove @alamb should we do that now? I guess one concern might be if we want it to block the whole DataFusion pipeline if this fails

[yjshen]

It seems I was unable to run the integration tests using this PR locally due to an arrow compilation error:

run:

dev/integration-tests.sh

Finally get:

error: could not compile `arrow`

Caused by:
  process didn't exit successfully: `rustc --crate-name arrow --edition=2021 /usr/local/cargo/registry/src/fd.fxlcf.dpdns.org-1ecc6299db9ec823/arrow-10.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C codegen-units=1 --cfg 'feature="comfy-table"' --cfg 'feature="csv"' --cfg 'feature="csv_crate"' --cfg 'feature="default"' --cfg 'feature="flatbuffers"' --cfg 'feature="ipc"' --cfg 'feature="prettyprint"' --cfg 'feature="rand"' --cfg 'feature="test_utils"' -C metadata=d397f8e267aeac42 -C extra-filename=-d397f8e267aeac42 --out-dir /tmp/ballista/target/release/deps -L dependency=/tmp/ballista/target/release/deps --extern bitflags=/tmp/ballista/target/release/deps/libbitflags-5182ee06b0261ea5.rmeta --extern chrono=/tmp/ballista/target/release/deps/libchrono-0093b5093d2d1ae8.rmeta --extern comfy_table=/tmp/ballista/target/release/deps/libcomfy_table-57c2aefe72826335.rmeta --extern csv_crate=/tmp/ballista/target/release/deps/libcsv-a9cbe2362b55a77e.rmeta --extern flatbuffers=/tmp/ballista/target/release/deps/libflatbuffers-5c4a4352ba21817e.rmeta --extern half=/tmp/ballista/target/release/deps/libhalf-d38738f000b3c670.rmeta --extern hex=/tmp/ballista/target/release/deps/libhex-7b34d2c291b6ed12.rmeta --extern indexmap=/tmp/ballista/target/release/deps/libindexmap-125fff6d7b2ad6d4.rmeta --extern lazy_static=/tmp/ballista/target/release/deps/liblazy_static-be01e6c7b419929c.rmeta --extern lexical_core=/tmp/ballista/target/release/deps/liblexical_core-86d4f438bc1a25c8.rmeta --extern multiversion=/tmp/ballista/target/release/deps/libmultiversion-133ec2520e3b23c5.rmeta --extern num=/tmp/ballista/target/release/deps/libnum-2f517b9ac25baa4d.rmeta --extern rand=/tmp/ballista/target/release/deps/librand-1ae75b019c023c85.rmeta --extern regex=/tmp/ballista/target/release/deps/libregex-e51999868ae7fc39.rmeta --extern serde=/tmp/ballista/target/release/deps/libserde-bb6f55394b459156.rmeta --extern serde_derive=/tmp/ballista/target/release/deps/libserde_derive-f7dad95eddccdc1b.so --extern serde_json=/tmp/ballista/target/release/deps/libserde_json-12e43b16bda7b1dc.rmeta --cap-lints allow` (signal: 9, SIGKILL: kill)
warning: build failed, waiting for other jobs to finish...
error: build failed
The command '/bin/sh -c cargo build $RELEASE_FLAG' returned a non-zero code: 101

[andygrove]

@realno We have an issue open for adding these tests to CI [1]. I remember there was an effort to implement this some time ago but there were some technical challenges, IIRC. It would be great if someone could have another go at this.

[1] apache/datafusion-ballista#24

[andygrove]

@yjshen Could you file a follow-on issue if you are still not able to compile, with some more detail about the error.