Move deploy scripts from cloudberry-devops-release to main repo by leborchuk · Pull Request #1292 · apache/cloudberry

leborchuk · 2025-08-06T20:06:16Z

As was discussed in https://lists.apache.org/thread/p6otyrrnosg8fsbyr6ok7hl8wxpx4ss2 it will be great to have all scripts in one repo. We could change the code and the building scripts at the same time.

The good example is PAX, a big feature where we needed to make several commits to all repositories at the same time because the changes depended on each other.

Here I:

Moved all scripts from cloudberry-devops-release to cloudberry (still need to check I may missed the last commit)
Placed them in deploy directory. See it's empty now and could be re-used.
Added license info to all files missing it
Write a small instruction how to use scripts

leborchuk · 2025-08-06T20:06:34Z

@tuhaihe review pls

tuhaihe · 2025-08-07T07:24:54Z

Hey @leborchuk, thanks for working on this! It’s great to see the devops-release scripts being brought into the main repository — this will definitely help improve maintenance and visibility moving forward.

The following is my feedback for your reference:

Concern about Missing Commit History

We see that this is a complex work. It looks like the original commit history from the old repository has been lost during the migration. While it’s fine to move the code over, that would be better to keep the original commit history.

It will be a lot of noise for the main repo if we keep the full commit history. So I'd suggest squashing them into an initial import commit, which helps trace the origin of the scripts, like:

Initial import from cloudberry-devops-release 

(original commit <sha>)

xxxx
xxxx

For the tweaks to the old files, we can create new commits following the initial commits, which will make the changes obviously and the review work will be easier.
Dir reorganization

Now it's a good time to reorganzation files for a better management, also taking the future feature files into consideration.

Here is a new file tree:

apache/cloudberry/
├── .github/workflows/         # GitHub Actions
├── devops/                   # New: Unified DevOps directory
│   ├── README.md            # DevOps overview documentation
│   ├── build/               # Build-related files
│   │   ├── automation/      # Build automation scripts
│   │   │   └── cloudberry/
│   │   │       └── scripts/*
│   │   ├── packaging/       # Packaging-related files
│   │   │   ├── rpm/         # RPM packaging
│   │   │   │   ├── apache-cloudberry-db-incubating.spec
│   │   │   │   ├── apache-cloudberry-hll-incubating.spec
│   │   │   │   ├── apache-cloudberry-pgvector-incubating.spec
│   │   │   │   ├── cloudberry-dev-repo.spec
│   │   │   │   ├── build-rpm.sh
│   │   │   │   └── README.md
│   │   │   ├── deb/*        # Future DEB packaging (reserved)
│   │   │   └── README.md    # Packaging documentation
│   │   └── README.md        # Build documentation
│   ├── deploy/              # Deployment-related files
│   │   ├── docker/          # Docker images
│   │   │   ├── build/
│   │   │   │   ├── rocky8/
│   │   │   │   ├── rocky9/
│   │   │   │   └── ubuntu22.04/ # Future addition
│   │   │   ├── test/
│   │   │   │   ├── rocky8/
│   │   │   │   ├── rocky9/
│   │   │   │   └── ubuntu22.04/ # Future addition
│   │   │   └── README.md
│   │   └── README.md        # Deployment documentation
│   ├── release/             # Release management
│   │   ├── cloudberry-release.sh
│   │   └── README.md        # Release process documentation
│   └── tools/               # Utility scripts
│       ├── elf_rockylinux_dependency_analyzer.py
│       ├── elf_ubuntu_dependency_analyzer.py
│       ├── s3-repo-sync-and-sign.sh
│       └── README.md        # Tools documentation
├── .asf.yaml                # Remove the old one
├── README.md                # Remove the old one
├── CONTRIBUTING.md          # Remove the old one
├── LICENSE                  # Remove the old one

Directory Migration Mapping:

# Build automation
build_automation/                                    → devops/build/automation/
build_automation/cloudberry/                        → devops/build/automation/cloudberry/
build_automation/cloudberry/scripts/                → devops/build/automation/cloudberry/scripts/

# Packaging
packaging/                                          → devops/build/packaging/
packaging/rpm/                                      → devops/build/packaging/rpm/
packaging/rpm/el/SPECS/                            → devops/build/packaging/rpm/
packaging/rpm/repo-el/SPECS/                       → devops/build/packaging/rpm/

# Docker images
images/                                             → devops/deploy/
images/docker/                                      → devops/deploy/docker/
images/docker/cbdb/                                 → devops/deploy/docker/
images/docker/cbdb/build/                          → devops/deploy/docker/build/
images/docker/cbdb/test/                           → devops/deploy/docker/test/
images/docker/cbdb/build/rocky8/                   → devops/deploy/docker/build/rocky8/
images/docker/cbdb/build/rocky9/                   → devops/deploy/docker/build/rocky9/
images/docker/cbdb/test/rocky8/                    → devops/deploy/docker/test/rocky8/
images/docker/cbdb/test/rocky9/                    → devops/deploy/docker/test/rocky9/

# Script files
scripts/                                            → # Distributed to different directories
scripts/release/                                    → devops/release/
scripts/build-rpm.sh                               → devops/build/packaging/rpm/
scripts/elf_rockylinux_dependency_analyzer.py      → devops/tools/
scripts/elf_ubuntu_dependency_analyzer.py          → devops/tools/
scripts/s3-repo-sync-and-sign.sh                   → devops/tools/

Files migration mapping:

# Build automation scripts
build_automation/cloudberry/scripts/analyze_core_dumps.sh              → devops/build/automation/cloudberry/scripts/analyze_core_dumps.sh
build_automation/cloudberry/scripts/build-cloudberry.sh                → devops/build/automation/cloudberry/scripts/build-cloudberry.sh
build_automation/cloudberry/scripts/cloudberry-utils.sh                → devops/build/automation/cloudberry/scripts/cloudberry-utils.sh
build_automation/cloudberry/scripts/configure-cloudberry.sh            → devops/build/automation/cloudberry/scripts/configure-cloudberry.sh
build_automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh  → devops/build/automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh
build_automation/cloudberry/scripts/destroy-cloudberry-demo-cluster.sh → devops/build/automation/cloudberry/scripts/destroy-cloudberry-demo-cluster.sh
build_automation/cloudberry/scripts/parse-results.pl                   → devops/build/automation/cloudberry/scripts/parse-results.pl
build_automation/cloudberry/scripts/parse-test-results.sh              → devops/build/automation/cloudberry/scripts/parse-test-results.sh
build_automation/cloudberry/scripts/test-cloudberry.sh                 → devops/build/automation/cloudberry/scripts/test-cloudberry.sh
build_automation/cloudberry/scripts/unittest-cloudberry.sh             → devops/build/automation/cloudberry/scripts/unittest-cloudberry.sh

# RPM packaging files
packaging/rpm/el/SPECS/apache-cloudberry-db-incubating.spec           → devops/build/packaging/rpm/apache-cloudberry-db-incubating.spec
packaging/rpm/el/SPECS/apache-cloudberry-hll-incubating.spec          → devops/build/packaging/rpm/apache-cloudberry-hll-incubating.spec
packaging/rpm/el/SPECS/apache-cloudberry-pgvector-incubating.spec     → devops/build/packaging/rpm/apache-cloudberry-pgvector-incubating.spec
packaging/rpm/repo-el/SPECS/cloudberry-dev-repo.spec                  → devops/build/packaging/rpm/cloudberry-dev-repo.spec

# Script files
scripts/build-rpm.sh                                                  → devops/build/packaging/rpm/build-rpm.sh
scripts/release/cloudberry-release.sh                                 → devops/release/cloudberry-release.sh
scripts/elf_rockylinux_dependency_analyzer.py                         → devops/tools/elf_rockylinux_dependency_analyzer.py
scripts/elf_ubuntu_dependency_analyzer.py                             → devops/tools/elf_ubuntu_dependency_analyzer.py
scripts/s3-repo-sync-and-sign.sh                                      → devops/tools/s3-repo-sync-and-sign.sh

# Docker related files (keep internal structure unchanged)
images/docker/cbdb/build/rocky8/*                                      → devops/deploy/docker/build/rocky8/*
images/docker/cbdb/build/rocky9/*                                      → devops/deploy/docker/build/rocky9/*
images/docker/cbdb/test/rocky8/*                                       → devops/deploy/docker/test/rocky8/*
images/docker/cbdb/test/rocky9/*                                       → devops/deploy/docker/test/rocky9/*

# GitHub Actions workflows (file location unchanged, but need to update internal path references)
.github/workflows/docker-cbdb-build-containers.yml                     → .github/workflows/docker-cbdb-build-containers.yml
.github/workflows/docker-cbdb-test-containers.yml                      → .github/workflows/docker-cbdb-test-containers.yml

Note:

For the files under build/automation/cloudberry/scripts/*, if no more future plans for them, we can directly place them in build/automation/*, making them easier to access.
For the old deploy dir under apache/cloudberry, there are no files in it now. So we can make it as a subdir under devops. For the cloudberry-bootcamp/sandbox, I plan to migrate it to devops/sandbox.
For the new README.md files, there is no need to create them all at once in this PR. We can do that in the future.

Love to have your thoughts! @leborchuk @edespino

leborchuk · 2025-08-07T08:04:21Z

Thanks, I have been thinking about the directory structure and agree that it needs to be reorganized for better clarity. I will start working on the suggested structure, as it looks quite logical to me, let's try to implement it.

leborchuk · 2025-08-07T08:10:07Z

@tuhaihe, could we freeze the changes on https://github.com/apache/cloudberry-devops-release at the time of the migration? I believe we can get all the work done in two weeks, and then we can move the open PR to the main repository.

tuhaihe · 2025-08-07T08:48:18Z

@tuhaihe, could we freeze the changes on https://github.com/apache/cloudberry-devops-release at the time of the migration? I believe we can get all the work done in two weeks, and then we can move the open PR to the main repository.

Good catch! IMO, we probably won't see many changes during migration, and not sure if any urgent updates are needed. How about pinning a notice issue instead to keep everyone in the loop about the migration? Once the migration is done, we can request the ASF infra team to archive the repo.

leborchuk · 2025-08-15T15:14:23Z

Hey @leborchuk, thanks for working on this! It’s great to see the devops-release scripts being brought into the main repository — this will definitely help improve maintenance and visibility moving forward.

The following is my feedback for your reference:

Concern about Missing Commit History

We see that this is a complex work. It looks like the original commit history from the old repository has been lost during the migration. While it’s fine to move the code over, that would be better to keep the original commit history.

It will be a lot of noise for the main repo if we keep the full commit history. So I'd suggest squashing them into an initial import commit, which helps trace the origin of the scripts, like:

Initial import from cloudberry-devops-release 

(original commit <sha>)

xxxx
xxxx

For the tweaks to the old files, we can create new commits following the initial commits, which will make the changes obviously and the review work will be easier.
Dir reorganization

Now it's a good time to reorganzation files for a better management, also taking the future feature files into consideration.

Here is a new file tree:

apache/cloudberry/
├── .github/workflows/         # GitHub Actions
├── devops/                   # New: Unified DevOps directory
│   ├── README.md            # DevOps overview documentation
│   ├── build/               # Build-related files
│   │   ├── automation/      # Build automation scripts
│   │   │   └── cloudberry/
│   │   │       └── scripts/*
│   │   ├── packaging/       # Packaging-related files
│   │   │   ├── rpm/         # RPM packaging
│   │   │   │   ├── apache-cloudberry-db-incubating.spec
│   │   │   │   ├── apache-cloudberry-hll-incubating.spec
│   │   │   │   ├── apache-cloudberry-pgvector-incubating.spec
│   │   │   │   ├── cloudberry-dev-repo.spec
│   │   │   │   ├── build-rpm.sh
│   │   │   │   └── README.md
│   │   │   ├── deb/*        # Future DEB packaging (reserved)
│   │   │   └── README.md    # Packaging documentation
│   │   └── README.md        # Build documentation
│   ├── deploy/              # Deployment-related files
│   │   ├── docker/          # Docker images
│   │   │   ├── build/
│   │   │   │   ├── rocky8/
│   │   │   │   ├── rocky9/
│   │   │   │   └── ubuntu22.04/ # Future addition
│   │   │   ├── test/
│   │   │   │   ├── rocky8/
│   │   │   │   ├── rocky9/
│   │   │   │   └── ubuntu22.04/ # Future addition
│   │   │   └── README.md
│   │   └── README.md        # Deployment documentation
│   ├── release/             # Release management
│   │   ├── cloudberry-release.sh
│   │   └── README.md        # Release process documentation
│   └── tools/               # Utility scripts
│       ├── elf_rockylinux_dependency_analyzer.py
│       ├── elf_ubuntu_dependency_analyzer.py
│       ├── s3-repo-sync-and-sign.sh
│       └── README.md        # Tools documentation
├── .asf.yaml                # Remove the old one
├── README.md                # Remove the old one
├── CONTRIBUTING.md          # Remove the old one
├── LICENSE                  # Remove the old one

Directory Migration Mapping:

# Build automation
build_automation/                                    → devops/build/automation/
build_automation/cloudberry/                        → devops/build/automation/cloudberry/
build_automation/cloudberry/scripts/                → devops/build/automation/cloudberry/scripts/

# Packaging
packaging/                                          → devops/build/packaging/
packaging/rpm/                                      → devops/build/packaging/rpm/
packaging/rpm/el/SPECS/                            → devops/build/packaging/rpm/
packaging/rpm/repo-el/SPECS/                       → devops/build/packaging/rpm/

# Docker images
images/                                             → devops/deploy/
images/docker/                                      → devops/deploy/docker/
images/docker/cbdb/                                 → devops/deploy/docker/
images/docker/cbdb/build/                          → devops/deploy/docker/build/
images/docker/cbdb/test/                           → devops/deploy/docker/test/
images/docker/cbdb/build/rocky8/                   → devops/deploy/docker/build/rocky8/
images/docker/cbdb/build/rocky9/                   → devops/deploy/docker/build/rocky9/
images/docker/cbdb/test/rocky8/                    → devops/deploy/docker/test/rocky8/
images/docker/cbdb/test/rocky9/                    → devops/deploy/docker/test/rocky9/

# Script files
scripts/                                            → # Distributed to different directories
scripts/release/                                    → devops/release/
scripts/build-rpm.sh                               → devops/build/packaging/rpm/
scripts/elf_rockylinux_dependency_analyzer.py      → devops/tools/
scripts/elf_ubuntu_dependency_analyzer.py          → devops/tools/
scripts/s3-repo-sync-and-sign.sh                   → devops/tools/

Files migration mapping:

# Build automation scripts
build_automation/cloudberry/scripts/analyze_core_dumps.sh              → devops/build/automation/cloudberry/scripts/analyze_core_dumps.sh
build_automation/cloudberry/scripts/build-cloudberry.sh                → devops/build/automation/cloudberry/scripts/build-cloudberry.sh
build_automation/cloudberry/scripts/cloudberry-utils.sh                → devops/build/automation/cloudberry/scripts/cloudberry-utils.sh
build_automation/cloudberry/scripts/configure-cloudberry.sh            → devops/build/automation/cloudberry/scripts/configure-cloudberry.sh
build_automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh  → devops/build/automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh
build_automation/cloudberry/scripts/destroy-cloudberry-demo-cluster.sh → devops/build/automation/cloudberry/scripts/destroy-cloudberry-demo-cluster.sh
build_automation/cloudberry/scripts/parse-results.pl                   → devops/build/automation/cloudberry/scripts/parse-results.pl
build_automation/cloudberry/scripts/parse-test-results.sh              → devops/build/automation/cloudberry/scripts/parse-test-results.sh
build_automation/cloudberry/scripts/test-cloudberry.sh                 → devops/build/automation/cloudberry/scripts/test-cloudberry.sh
build_automation/cloudberry/scripts/unittest-cloudberry.sh             → devops/build/automation/cloudberry/scripts/unittest-cloudberry.sh

# RPM packaging files
packaging/rpm/el/SPECS/apache-cloudberry-db-incubating.spec           → devops/build/packaging/rpm/apache-cloudberry-db-incubating.spec
packaging/rpm/el/SPECS/apache-cloudberry-hll-incubating.spec          → devops/build/packaging/rpm/apache-cloudberry-hll-incubating.spec
packaging/rpm/el/SPECS/apache-cloudberry-pgvector-incubating.spec     → devops/build/packaging/rpm/apache-cloudberry-pgvector-incubating.spec
packaging/rpm/repo-el/SPECS/cloudberry-dev-repo.spec                  → devops/build/packaging/rpm/cloudberry-dev-repo.spec

# Script files
scripts/build-rpm.sh                                                  → devops/build/packaging/rpm/build-rpm.sh
scripts/release/cloudberry-release.sh                                 → devops/release/cloudberry-release.sh
scripts/elf_rockylinux_dependency_analyzer.py                         → devops/tools/elf_rockylinux_dependency_analyzer.py
scripts/elf_ubuntu_dependency_analyzer.py                             → devops/tools/elf_ubuntu_dependency_analyzer.py
scripts/s3-repo-sync-and-sign.sh                                      → devops/tools/s3-repo-sync-and-sign.sh

# Docker related files (keep internal structure unchanged)
images/docker/cbdb/build/rocky8/*                                      → devops/deploy/docker/build/rocky8/*
images/docker/cbdb/build/rocky9/*                                      → devops/deploy/docker/build/rocky9/*
images/docker/cbdb/test/rocky8/*                                       → devops/deploy/docker/test/rocky8/*
images/docker/cbdb/test/rocky9/*                                       → devops/deploy/docker/test/rocky9/*

# GitHub Actions workflows (file location unchanged, but need to update internal path references)
.github/workflows/docker-cbdb-build-containers.yml                     → .github/workflows/docker-cbdb-build-containers.yml
.github/workflows/docker-cbdb-test-containers.yml                      → .github/workflows/docker-cbdb-test-containers.yml

Note:

For the files under build/automation/cloudberry/scripts/*, if no more future plans for them, we can directly place them in build/automation/*, making them easier to access.
For the old deploy dir under apache/cloudberry, there are no files in it now. So we can make it as a subdir under devops. For the cloudberry-bootcamp/sandbox, I plan to migrate it to devops/sandbox.
For the new README.md files, there is no need to create them all at once in this PR. We can do that in the future.

Love to have your thoughts! @leborchuk @edespino

Hi, @tuhaihe
All done, could you launch the tests in order to check if everything is fine here in github actions? I do not have rights do it by myself.

Renames (just for check with previous version) I have done in the last commit
renamed: deploy/README.md -> devops/README.md renamed: deploy/build_automation/cloudberry/scripts/analyze_core_dumps.sh -> devops/build/automation/cloudberry/scripts/analyze_core_dumps.sh renamed: deploy/build_automation/cloudberry/scripts/build-cloudberry.sh -> devops/build/automation/cloudberry/scripts/build-cloudberry.sh renamed: deploy/build_automation/cloudberry/scripts/cloudberry-utils.sh -> devops/build/automation/cloudberry/scripts/cloudberry-utils.sh renamed: deploy/build_automation/cloudberry/scripts/configure-cloudberry.sh -> devops/build/automation/cloudberry/scripts/configure-cloudberry.sh renamed: deploy/build_automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh -> devops/build/automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh renamed: deploy/build_automation/cloudberry/scripts/destroy-cloudberry-demo-cluster.sh -> devops/build/automation/cloudberry/scripts/destroy-cloudberry-demo-cluster.sh renamed: deploy/build_automation/cloudberry/scripts/parse-results.pl -> devops/build/automation/cloudberry/scripts/parse-results.pl renamed: deploy/build_automation/cloudberry/scripts/parse-test-results.sh -> devops/build/automation/cloudberry/scripts/parse-test-results.sh renamed: deploy/build_automation/cloudberry/scripts/test-cloudberry.sh -> devops/build/automation/cloudberry/scripts/test-cloudberry.sh renamed: deploy/build_automation/cloudberry/scripts/unittest-cloudberry.sh -> devops/build/automation/cloudberry/scripts/unittest-cloudberry.sh renamed: deploy/packaging/rpm/el/SPECS/apache-cloudberry-db-incubating.spec -> devops/build/packaging/rpm/apache-cloudberry-db-incubating.spec renamed: deploy/packaging/rpm/el/SPECS/apache-cloudberry-hll-incubating.spec -> devops/build/packaging/rpm/apache-cloudberry-hll-incubating.spec renamed: deploy/packaging/rpm/el/SPECS/apache-cloudberry-pgvector-incubating.spec -> devops/build/packaging/rpm/apache-cloudberry-pgvector-incubating.spec renamed: deploy/scripts/build-rpm.sh -> devops/build/packaging/rpm/build-rpm.sh renamed: deploy/packaging/rpm/repo-el/SPECS/cloudberry-dev-repo.spec -> devops/build/packaging/rpm/cloudberry-dev-repo.spec renamed: deploy/images/docker/cbdb/build/rocky8/Dockerfile -> devops/deploy/docker/build/rocky8/Dockerfile renamed: deploy/images/docker/cbdb/build/rocky8/configs/90-cbdb-limits -> devops/deploy/docker/build/rocky8/configs/90-cbdb-limits renamed: deploy/images/docker/cbdb/build/rocky8/configs/gpinitsystem.conf -> devops/deploy/docker/build/rocky8/configs/gpinitsystem.conf renamed: deploy/images/docker/cbdb/build/rocky8/configs/init_system.sh -> devops/deploy/docker/build/rocky8/configs/init_system.sh renamed: deploy/images/docker/cbdb/build/rocky8/tests/requirements.txt -> devops/deploy/docker/build/rocky8/tests/requirements.txt renamed: deploy/images/docker/cbdb/build/rocky8/tests/testinfra/test_cloudberry_db_env.py -> devops/deploy/docker/build/rocky8/tests/testinfra/test_cloudberry_db_env.py renamed: deploy/images/docker/cbdb/build/rocky9/Dockerfile -> devops/deploy/docker/build/rocky9/Dockerfile renamed: deploy/images/docker/cbdb/build/rocky9/configs/90-cbdb-limits -> devops/deploy/docker/build/rocky9/configs/90-cbdb-limits renamed: deploy/images/docker/cbdb/build/rocky9/configs/gpinitsystem.conf -> devops/deploy/docker/build/rocky9/configs/gpinitsystem.conf renamed: deploy/images/docker/cbdb/build/rocky9/configs/init_system.sh -> devops/deploy/docker/build/rocky9/configs/init_system.sh renamed: deploy/images/docker/cbdb/build/rocky9/tests/requirements.txt -> devops/deploy/docker/build/rocky9/tests/requirements.txt renamed: deploy/images/docker/cbdb/build/rocky9/tests/testinfra/test_cloudberry_db_env.py -> devops/deploy/docker/build/rocky9/tests/testinfra/test_cloudberry_db_env.py renamed: deploy/images/docker/cbdb/test/rocky8/Dockerfile -> devops/deploy/docker/test/rocky8/Dockerfile renamed: deploy/images/docker/cbdb/test/rocky8/configs/90-cbdb-limits -> devops/deploy/docker/test/rocky8/configs/90-cbdb-limits renamed: deploy/images/docker/cbdb/test/rocky8/configs/gpinitsystem.conf -> devops/deploy/docker/test/rocky8/configs/gpinitsystem.conf renamed: deploy/images/docker/cbdb/test/rocky8/configs/init_system.sh -> devops/deploy/docker/test/rocky8/configs/init_system.sh renamed: deploy/images/docker/cbdb/test/rocky9/Dockerfile -> devops/deploy/docker/test/rocky9/Dockerfile renamed: deploy/images/docker/cbdb/test/rocky9/configs/90-cbdb-limits -> devops/deploy/docker/test/rocky9/configs/90-cbdb-limits renamed: deploy/images/docker/cbdb/test/rocky9/configs/gpinitsystem.conf -> devops/deploy/docker/test/rocky9/configs/gpinitsystem.conf renamed: deploy/images/docker/cbdb/test/rocky9/configs/init_system.sh -> devops/deploy/docker/test/rocky9/configs/init_system.sh renamed: deploy/scripts/release/cloudberry-release.sh -> devops/release/cloudberry-release.sh renamed: deploy/scripts/elf_rockylinux_dependency_analyzer.py -> devops/tools/elf_rockylinux_dependency_analyzer.py renamed: deploy/scripts/elf_ubuntu_dependency_analyzer.py -> devops/tools/elf_ubuntu_dependency_analyzer.py renamed: deploy/scripts/s3-repo-sync-and-sign.sh -> devops/tools/s3-repo-sync-and-sign.sh

leborchuk · 2025-08-18T08:52:18Z

All tests look good. Fixed tests/requirements.txt located in pom.xml since files have now been placed in the devops/deploy/docker/build folder.

tuhaihe · 2025-08-20T10:23:10Z

Hey @leborchuk sorry for the late reply. Amazing work!

I would suggest having an initial commit that squashes all the original commits into one. Then can apply for the new commits for changes following the initial commits, which will be more visual for the review. I mean, the way of cherry-picking commits from GP.

leborchuk · 2025-08-27T11:44:15Z

Sorry, I do not fully understand the right way to make an initial commit, but tried my best.

I moved files from cloudberry-devops-release (cannot cherry-pick because had conflicts) to main repo and created description mentioned all commits in old repo

And then created the second commit to reorganize files in a suggested new directory structure.

Please review it. @tuhaihe

tuhaihe · 2025-08-28T09:25:37Z

Hi @leborchuk, thanks for your continued work on this complex migration!

I see some places still need to be updated, but I’d prefer not to ask too much -- you have tried your best 🫶 . I will help update it later once your PR is merged.

Before we merge this PR, here are the things we need to update:

License header

As reported by the RAT workflow, the standard Apache license header needs to be added to the following files:

  devops/tools/elf_rockylinux_dependency_analyzer.py
  devops/tools/s3-repo-sync-and-sign.sh
  devops/tools/elf_ubuntu_dependency_analyzer.py
  devops/build/packaging/rpm/apache-cloudberry-pgvector-incubating.spec
  devops/build/packaging/rpm/apache-cloudberry-hll-incubating.spec
  devops/build/packaging/rpm/cloudberry-dev-repo.spec
  devops/build/packaging/rpm/apache-cloudberry-db-incubating.spec
  devops/build/packaging/rpm/build-rpm.sh

Can take the files in #1246 as the reference.

Once all the checks pass, please send a notice email to the Dev mailing list to inform the developers that this change will happen, and request one more approval from the committers.
I will help ask the infra team to archive the original apache/cloudberry-devops-release/ repo once this PR is merged.

Let's work further!

leborchuk · 2025-08-29T10:34:59Z

@tuhaihe Thanks for the support, let's keep going ) Could you re-launch all checks once again?

License header

fixed, I see rat-check works in my repo too and it green now https://github.com/leborchuk/cloudberrydb/actions/runs/17321280086

leborchuk · 2025-08-29T12:11:00Z

Once all the checks pass, please send a notice email to the Dev mailing list to inform the developers that this change will happen, and request one more approval from the committers.

done, we're waiting here for comments/approvals.

…in repo Changes here includes original commits ``` git log --pretty=format:"%H%x09%an%x09%ad%x09%s" 5c1a2ada9a93ab5f930aebd0018a7369fdf61930 Dianjin Wang Wed Jun 25 18:28:43 2025 +0800 Update gcc/g++ settings for PAX on RockyLinux 8 133a81303555dedc07c36ec16aa686367c47c774 Leonid Borchuk Wed Jul 16 15:58:52 2025 +0000 Rename greenplum_path to cloudberry-env eef2516b90bb7b9de2af95cc2a9df5b125533794 Dianjin Wang Sat Jun 7 08:51:09 2025 +0800 Update `dorny/paths-filter` version tag to commit e06dd830250ce89184b13a18aa6663ffcb56db4b Ed Espino Sun Jun 1 12:27:17 2025 -0700 Initial commit: Apache Cloudberry (Incubating) release script 384202893e571ce06a2224a116019c2ca9a3dce5 Dianjin Wang Wed Apr 30 16:09:51 2025 +0800 Add the PAX support in the configure 5081920c07096e9e2cf217ad1b7489ae8963a86a Ed Espino Tue Apr 1 03:31:45 2025 -0700 Add protobuf-devel to Dockerfiles for Rocky 8 and 9 builds (apache#14) 7a6549cedb84edad516b89ebbd73529b773aaf03 Jianghua Yang Thu Feb 27 22:31:07 2025 +0800 Add debuginfo package. 5965faabbf15965778ec09bca02960dbc1900a6a Ed Espino Thu Feb 13 01:28:47 2025 -0800 Add new Cloudberry dependency (apache#11) d50af03d7c04341ce86047aa098bfd8e6a914804 Ed Espino Sun Dec 15 23:27:47 2024 -0800 Add script to analyze core dumps with gdb (#10) 54bbb3d10ad642f98cf97418978319d6d030070c Ed Espino Sun Dec 15 21:39:20 2024 -0800 Adding packages (gdb and file) used in core file analysis. (apache#9) 9638c9e3c983e4d9ae7327517c3f88b0f8335614 Ed Espino Mon Dec 9 18:43:57 2024 -0800 Container - Multi arch support for Rocky 8 & 9 (apache#8) 5249d69825ec34c2363aa3de3391ddc790786ffe Ed Espino Wed Nov 27 02:09:46 2024 -0800 Enhance test result parsing for ignored tests (apache#7) f6fb4296c392b3cfaef0da45da282ca667af1025 Ed Espino Tue Nov 19 18:02:14 2024 -0800 fix: remove -e from set options in parse-test-results.sh (apache#6) 2c8302c2c2beb2682185cc9c54503342b0bb0351 Ed Espino Thu Nov 7 22:11:17 2024 -0800 build: add Apache Cloudberry build automation and test framework 6b8f8938196d55902eecf3506a9142811f97d633 Ed Espino Thu Nov 14 02:42:02 2024 -0800 Update to Apache Cloudberry (incubating) rpm name and add disclaimer 1a5852903579b11ec0bd12d3083cf4299250eb96 Ed Espino Thu Nov 14 10:44:10 2024 -0800 Update repository names for pushing to official Docker Hub repository 7f82c004c4a106832a2483e509ba152082561838 Ed Espino Thu Nov 14 00:13:43 2024 -0800 Add initial Dockerfiles, configs, and GitHub workflows for Cloudberry 72e5f06a7cdf6e807b255717ffe5681a122da474 Dianjin Wang Tue Nov 5 14:21:11 2024 +0800 Add asf-yaml and basic community files 5acab8e2c785c8036c92562017d06f666b42cfd5 Ed Espino Wed Sep 4 04:16:35 2024 -0700 Fix release names and paramerize pgvector version. 4ec026dea578f397512c3d8686082317338d6d2c Ed Espino Tue Sep 3 01:50:34 2024 -0700 Using Cloudberry pgvector 0.5.1 ed928320f640731efd33e6157a315a4d3100efb5 Ed Espino Mon Sep 2 23:50:39 2024 -0700 Change ownership of symlink. e8b22b0bffdb44a65528cbc862cfe8d6425302af Ed Espino Sun Sep 1 23:52:57 2024 -0700 Update elf scripts 7f295146bc3412894ecbb0027c10754e736fb7ed Ed Espino Fri Aug 30 16:23:38 2024 -0700 Change default installation directory. 1a571a8bdeac6eb1ccf4e7d81d7d89da7ae6b0e8 Ed Espino Fri Aug 30 16:10:44 2024 -0700 Change default installation directory. 8ed243cff42501843f86a8b70925a6b1e34c6681 Ed Espino Fri Aug 30 03:52:16 2024 -0700 Updates ab43f527c56b197f5c223672e92fb1d384f40858 Ed Espino Fri Aug 30 03:36:35 2024 -0700 Add hll and pgvector extensions a133b6ea1fe5b0cb8c7440218873023be990ddfa Ed Espino Wed Aug 28 22:57:20 2024 -0700 Remove Changelog 371e50dafd31282da78ac661aa7f9890f083b3b9 Ed Espino Wed Aug 28 22:56:26 2024 -0700 Add Group to spec file. 54be4f19ecc15ee477040e169cd7e66246c3034e Ed Espino Wed Aug 28 12:43:09 2024 -0700 Fix changing ownership to gpadmin. 357936d3efd9de69b913b736172270781ac0b6f1 Ed Espino Wed Aug 28 12:02:15 2024 -0700 Adjustment for shipping GO apps (e.g. gpbackup). 4f2cde6f1c97913152498a6cbb54fcc0d9785958 Ed Espino Mon Aug 26 23:29:03 2024 -0700 Update spec files 44b1425441c482d260f087d0702aef66f379e840 Ed Espino Mon Aug 26 22:57:05 2024 -0700 Update docs 57faae9ab7600efc3a4bfe0dd7a2b4eb2d82229f Ed Espino Mon Aug 26 22:51:41 2024 -0700 minor enhancements 44123c9a7a133b4625ddaa2a06fb0be0db77f6a2 Ed Espino Thu Aug 22 02:15:00 2024 -0700 Script update. 60a5884b33ef848aa6861bb9537f28b84ce6597a Ed Espino Wed Aug 21 22:28:07 2024 -0700 Clean up repo rpm making it noarch and making the repo entry dynamic e40c74a91ee4dcac727622fdfbef84e41e61969e Ed Espino Wed Aug 21 13:09:22 2024 -0700 Add repo tool 16b3dfe3ead0834bd83165b5559f3388f4e55e6e Ed Espino Wed Aug 21 00:45:53 2024 -0700 Fix description in repo RPM. feb096c5d8887dd6b29d382be9b37844169df1ca Ed Espino Wed Aug 21 00:42:28 2024 -0700 Fix relocation RPM feature by createing own prefix variable. fd5017365759572248fd07c39af67f83f3987295 Ed Espino Wed Aug 21 00:10:05 2024 -0700 In spec file, set version and release variables via script (build-rpm.sh). b9668e1b86cc631db50ae603cc63cc0ffa35a400 Ed Espino Tue Aug 20 11:44:05 2024 -0700 EL SPEC file consolidation. 904f2981e8d1217b57bc5490d31e4463399d4551 Ed Espino Sun Aug 18 00:45:32 2024 -0700 Rename spec file and add additional runtime dependencies. 89f65a6a0902bbe0fd8294c5ad0767419e53e608 Ed Espino Sat Aug 17 01:04:02 2024 -0700 Add RPM GPG KEY 91ea01f7f0b7ce9e5553d0861ec640b3392bacd2 Ed Espino Fri Aug 16 22:56:54 2024 -0700 Update Spec file. c362bdab21ef9e33e885ee839ad9cf8b009122bd Ed Espino Fri Aug 16 19:00:41 2024 -0700 Create repo RPM 9b02885090fb9c09dd03202616303afe9c7f93f5 Ed Espino Fri Aug 16 11:14:36 2024 -0700 feat: Add ELF dependency analyzer script f3a569e9620c1d643046e92d1387f104a3dbf8cd Ed Espino Fri Aug 16 10:30:55 2024 -0700 Initial EL9 spec file ```

Update documentation and configuration files to use consistent "Apache Cloudberry" naming instead of "Cloudberry Database". This change affects Docker files, RPM specs, README files, and configuration templates to maintain brand consistency across the project. Main changes include: - Updated Dockerfile comments and descriptions - Modified RPM package summaries and descriptions - Standardized product names in README documentation - Updated gpinitsystem configuration file headers - Reorganized pom.xml excludes for better structure - Update the README.me for a better Cloudberry build guide - Ajust the pom.xml for a better layout This commit are mostly used for improving the PR apache#1292 .

Update documentation and configuration files to use consistent "Apache Cloudberry" naming instead of "Cloudberry Database". This change affects Docker files, RPM specs, README files, and configuration templates to maintain brand consistency across the project. Main changes include: - Updated Dockerfile comments and descriptions - Modified RPM package summaries and descriptions - Standardized product names in README documentation - Updated gpinitsystem configuration file headers - Reorganized pom.xml excludes for better structure - Update the README.me for a better Cloudberry build guide - Ajust the pom.xml for a better layout This commit are mostly used for improving the PR #1292.

Update documentation and configuration files to use consistent "Apache Cloudberry" naming instead of "Cloudberry Database". This change affects Docker files, RPM specs, README files, and configuration templates to maintain brand consistency across the project. Main changes include: - Updated Dockerfile comments and descriptions - Modified RPM package summaries and descriptions - Standardized product names in README documentation - Updated gpinitsystem configuration file headers - Reorganized pom.xml excludes for better structure - Update the README.me for a better Cloudberry build guide - Ajust the pom.xml for a better layout This commit are mostly used for improving the PR apache#1292.

leborchuk mentioned this pull request Aug 6, 2025

Add scripts for create ubuntu jammy docker file apache/cloudberry-devops-release#13

Closed

tuhaihe requested review from edespino and tuhaihe August 7, 2025 03:58

leborchuk force-pushed the AddDeployScripts branch from b6a5886 to 535d5da Compare August 15, 2025 15:12

leborchuk force-pushed the AddDeployScripts branch from 535d5da to f637ea2 Compare August 18, 2025 08:50

leborchuk force-pushed the AddDeployScripts branch from f637ea2 to 331d799 Compare August 27, 2025 11:38

leborchuk force-pushed the AddDeployScripts branch from 331d799 to 208f386 Compare August 29, 2025 10:23

tuhaihe approved these changes Sep 1, 2025

View reviewed changes

tuhaihe force-pushed the AddDeployScripts branch from 208f386 to ec3d63d Compare September 8, 2025 02:20

xtangcode approved these changes Sep 10, 2025

View reviewed changes

Leonid Borchuk added 2 commits September 11, 2025 07:38

Reorganize files in main repo and add licence info to all files

753e4ee

tuhaihe force-pushed the AddDeployScripts branch from ec3d63d to 753e4ee Compare September 10, 2025 23:38

tuhaihe merged commit b1e46cb into apache:main Sep 11, 2025
27 checks passed

tuhaihe mentioned this pull request Sep 11, 2025

Doc: update Cloudberry naming in devops dir #1354

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move deploy scripts from cloudberry-devops-release to main repo#1292

Move deploy scripts from cloudberry-devops-release to main repo#1292
tuhaihe merged 2 commits intoapache:mainfrom
leborchuk:AddDeployScripts

leborchuk commented Aug 6, 2025

Uh oh!

leborchuk commented Aug 6, 2025

Uh oh!

tuhaihe commented Aug 7, 2025

Uh oh!

leborchuk commented Aug 7, 2025

Uh oh!

leborchuk commented Aug 7, 2025

Uh oh!

tuhaihe commented Aug 7, 2025

Uh oh!

leborchuk commented Aug 15, 2025 •

edited

Loading

Uh oh!

leborchuk commented Aug 18, 2025

Uh oh!

tuhaihe commented Aug 20, 2025

Uh oh!

leborchuk commented Aug 27, 2025

Uh oh!

tuhaihe commented Aug 28, 2025

Uh oh!

leborchuk commented Aug 29, 2025

Uh oh!

leborchuk commented Aug 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

leborchuk commented Aug 6, 2025

Uh oh!

leborchuk commented Aug 6, 2025

Uh oh!

tuhaihe commented Aug 7, 2025

Uh oh!

leborchuk commented Aug 7, 2025

Uh oh!

leborchuk commented Aug 7, 2025

Uh oh!

tuhaihe commented Aug 7, 2025

Uh oh!

leborchuk commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leborchuk commented Aug 18, 2025

Uh oh!

tuhaihe commented Aug 20, 2025

Uh oh!

leborchuk commented Aug 27, 2025

Uh oh!

tuhaihe commented Aug 28, 2025

Uh oh!

leborchuk commented Aug 29, 2025

Uh oh!

leborchuk commented Aug 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leborchuk commented Aug 15, 2025 •

edited

Loading