Antalya 26.1 forward-port: Hybrid tables

[filimonov]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

#1071
#1156
#1272

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [5882538]

[zvonand]

test_storage_delta/test.py::test_network_activity_with_system_tables is failing in other PRs

I will now pull the latest branch and merge w/o waiting for CI -- just to make sure settings changes history is not corrupted

[alsugiliazova]

PR #1442 CI Verification Report

MasterCI Failures

#	Check	Failed Test	Error	Flaky? (cidb 90d)	Related?
1	Stateless (amd_tsan, s3, parallel 2/2)	02911_backup_restore_keeper_map	ZooKeeper socket not connected	210 failures / 13 days	No
2	Stateless (arm_asan, azure, parallel)	—	Timeout (infrastructure)	N/A	No
3	Integration (amd_asan, 2/6)	test_storage_delta::test_network_activity_with_system_tables	assert 0 == 2 on scan state log count	81 failures / 19 days	No
4	Integration (amd_asan, 6/6)	test_multiple_disks::test_concurrent_alter_modify[mt]	multiprocessing.context.TimeoutError — concurrent ALTER task exceeded 120s timeout. Caused internal pytest exit (status 2). Test is about concurrent disk ALTER operations on MergeTree, unrelated to Hybrid/Distributed changes.	154 failures / 21 days	No
5	Integration (amd_binary, 2/5)	test_storage_delta::test_network_activity_with_system_tables	Same as #3	Same as #3	No
6	Integration (arm_binary, 2/4)	—	Dropped/cancelled	N/A	No
7	Integration (amd_tsan, 2/6)	test_storage_delta::test_network_activity_with_system_tables	Same as #3	Same as #3	No
8	Integration (amd_tsan, 4/6)	test_restore_db_replica::test_query_after_restore_db_replica	UNKNOWN_TABLE during ALTER	139 failures / 25 days	No
9	Integration (amd_tsan, 6/6)	test_distributed_index_analysis::test_primary_key	Bootstrap query failure	11 failures / 3 days	No

All MasterCI failures are known flaky tests confirmed via the CI database (play.clickhouse.com). None are related to the PR.

MasterCI Stateless Tests — All Passed

All 39 stateless test checks passed with 0 failures (except 1 flaky 02911 and 1 timeout). The PR's new tests (03643_hybrid, 03644_hybrid_auto_cast, 03644_hybrid_unqualified_table, 03645_hybrid_alias_columns, 03648_alias_marker_with_mergeable_state) passed across all configurations including amd_debug, amd_tsan, amd_msan, amd_ubsan, amd_asan, arm_binary, and coverage.

---

Regression Test Failures

Pre-existing failure verification

All regression failures were verified against the previous commit on antalya-26.1 (e2b6edba — before this PR was merged). Every failing suite was already failing before the PR:

Suite	Before PR (e2b6edba)	After PR (4da6d93b)	Conclusion
settings	failure	failure	Pre-existing
swarms	failure	failure	Pre-existing
iceberg_1	failure	failure	Pre-existing
iceberg_2	failure	failure	Pre-existing
parquet	failure	failure	Pre-existing
s3_export_partition	failure	failure	Pre-existing
s3_export_part	failure	failure	Pre-existing

Regression should pass after frontports of all antalya features.

Open Hybrid Issues

All open issues labeled hybrid in the Altinity/ClickHouse repository apply to this 26.1 build:

Issue	Title
#1422	IN is not working with hybrid tables
#1424	Cannot find column exception when using function on alias column in subquery
#1347	Hybrid table fails to load on ATTACH/server restart when segment table was dropped
#1336	Unexpected hint when one of segments in hybrid table dropped
#1335	Hybrid engine generates duplicate ALIAS expressions when ALIAS type is manually misspecified
#1209	Join fails with hybrid table
#1208	Hybrid table fails with subquery referencing itself (UNKNOWN_TABLE error)

[alsugiliazova]

PR #1442 audit

Possible defects

• High risk: Use-after-free risk in Hybrid identifier validation error path
  • Impact: Undefined behavior/crash risk while processing malformed or non-existent additional segment identifiers in CREATE/ATTACH TABLE ... ENGINE = Hybrid(...); can destabilize server process during DDL error handling.
  • Anchor: src/Storages/StorageDistributed.cpp / registerStorageHybrid identifier branch.
  • Trigger condition:
    1. additional segment uses unqualified identifier (enters AST replacement path),
    2. code replaces table_function_ast with a new ASTTableIdentifier,
    3. later validation throws (unknown db/table or related exception),
    4. catch block formats message through stale raw pointer ast_identifier.
  • Affected transition: T3 (identifier normalization -> stored AST update).
  • Why this is a defect: raw pointer lifetime is tied to the old intrusive AST object; replacing both shared owners can free old object before catch uses pointer, violating memory safety.
  • Smallest logical repro:
    1. CREATE TABLE ... ENGINE=Hybrid(remote(...), 1, missing_table, number > 0) with context that forces unqualified identifier qualification path.
    2. Ensure lookup/validation throws after AST replacement (e.g. missing table).
    3. Catch path dereferences stale ast_identifier.
  • Likely fix direction:
    • Capture identifier name into String identifier_name = ast_identifier->name(); before any AST replacement and use that immutable string in all diagnostics.
    • Or hold strong ASTPtr original_identifier = table_function_ast; until end of branch.
    • Avoid raw pointer reuse after table_function_ast reassignment.
  • Regression test direction:
    • Add stateless test with unqualified additional segment identifier that fails validation after qualification.
    • Assert deterministic BAD_ARGUMENTS error text and no crash/UB.
    • Add variant for unknown table and unknown database.
  • Affected subsystem / blast radius: Hybrid DDL registration path; affects CREATE/ATTACH reliability and potentially process stability on invalid input.
  • Evidence:

else if (const auto * ast_identifier = table_function_ast->as<ASTIdentifier>())
{
    // ... omitted ...
    if (storage_id.database_name.empty())
    {
        // ... omitted ...
        auto qualified_identifier = make_intrusive<ASTTableIdentifier>(storage_id.database_name, storage_id.table_name);
        qualified_identifier->alias = ast_identifier->alias;
        qualified_identifier->prefer_alias_to_column_name = ast_identifier->prefer_alias_to_column_name;
        table_function_ast = qualified_identifier;
        engine_args[i] = table_function_ast;
    }
    // ... omitted ...
    catch (const Exception & e)
    {
        throw Exception(ErrorCodes::BAD_ARGUMENTS,
            "Argument #{}: table '{}' validation failed: {}", i, ast_identifier->name(), e.message());
    }
    // ... omitted ...
    catch (const Exception & e)
    {
        throw Exception(ErrorCodes::BAD_ARGUMENTS,
            "Argument #{}: invalid table identifier '{}': {}", i, ast_identifier->name(), e.message());
    }
}

No additional Medium/Low defects are confirmed from static evidence in this audit.