Antalya 26.1: Expose IcebergS3 partition_key and sorting_key in system.tables

[zvonand]

Supersedes #1095, #1098, #1211

Changelog category (leave one):

• Not for changelog (changelog entry is not required)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [fa81315]

[alsugiliazova]

PR #1432 CI Verification Report

Test Results

Stateless Tests — All Passed

Integration Tests — All Passed (except 1 cancelled)

Integration Tests (amd_tsan) — 2 Failures, Unrelated to PR

Failed tests (both unrelated to PR)

Test	Error	Related to PR?
test_restore_db_replica/test.py::test_query_after_restore_db_replica[alter table-no exists table-no restart]	Code: 60. UNKNOWN_TABLE — table does not exist during ALTER in DatabaseReplicated restore	No
test_restore_db_replica/test.py::test_query_after_restore_db_replica[alter table-with exists table-no restart]	Code: 60. UNKNOWN_TABLE — table does not exist during ALTER in DatabaseReplicated restore	No

Both failures are in test_restore_db_replica which tests DatabaseReplicated restore operations — unrelated to Iceberg or system.tables changes. Tests are flaky.

All Iceberg tests passed (33/33)

All 33 Iceberg-related tests (test_database_iceberg, test_storage_iceberg_no_spark) in shard 4/6 passed with OK status.

Other Checks — All Passed

---

PR Test Coverage

PR's own integration test — Passed

The PR includes a dedicated integration test test_system_tables_partition_sorting_keys in tests/integration/test_storage_iceberg_with_spark/test_system_iceberg_metadata.py that verifies partition_key and sorting_key are correctly exposed in system.tables for Iceberg tables. This test passed in all configurations where it ran:

Test	Backend	Result
test_system_iceberg_metadata	s3-1, s3-2, local-1, local-2, azure-1, azure-2	All OK
test_system_tables_partition_sorting_keys	s3	OK

(Verified in amd_tsan shard 3/6)

clickhouse-regression dedicated test — Available

The clickhouse-regression repository has a dedicated test that independently validates this functionality:

Test: iceberg/tests/iceberg_table_engine/system_tables_partition_sorting_keys.py

This test creates an Iceberg table with partitioning, reads it via the Iceberg table engine, and asserts that partition_key and sorting_key are correctly reported in system.tables.

[vzakaznikov]

Audit update for PR #1432 (Iceberg partition_key / sorting_key exposure in system.tables):

Confirmed defect:

• Medium: empty Iceberg tables can lose partition_key/sorting_key in system.tables
  • IcebergMetadata::partitionKey() and IcebergMetadata::sortingKey() both return nullopt when actual_data_snapshot is null.
  • For empty tables, getStateImpl() returns {nullptr, schema_id} when no current snapshot exists (or id < 0), even though table-level partition/sort definitions still exist in metadata.
  • Result: system.tables can show defaults/empty values for these keys on valid empty Iceberg tables.

Code anchors reviewed:

• src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
• src/Storages/System/StorageSystemTables.cpp

Suggested fix direction:

• Derive partition/sorting display from metadata/state snapshot even when there is no data snapshot, rather than gating on actual_data_snapshot.

Regression test direction:

• Add a system.tables assertion for an Iceberg table with partition/sort definitions before first insert/commit.

[alsugiliazova]

Issue opened after audit #1473