AWS profile support

[stats-dev]

Closes #2841

Rationale for this change

This PR adds explicit AWS profile support for both the Glue catalog client and
fsspec-based S3 FileIO.

While GlueCatalog already supports profile configuration, fsspec-based S3
operations did not propagate profile selection to the underlying
S3FileSystem or async AWS session. As a result, users had to rely on environment
variables or the default AWS profile, which makes it difficult to work with
multiple AWS configurations in parallel.

This change introduces two configuration properties:

• client.profile-name: a unified AWS profile for the catalog client and FileIO
• s3.profile-name: an AWS profile specifically for S3 FileIO

Profile resolution follows this precedence:

1. s3.profile-name
2. client.profile-name

This ensures consistent and explicit credential selection across catalog and
FileIO layers when using the fsspec backend.

Are these changes tested?

Yes. New unit tests were added to validate the profile propagation behavior.

• Glue Catalog

  • Verifies that boto3.Session(profile_name=...) is created when initializing
    GlueCatalog with client.profile-name.

• S3 FileIO (fsspec)

  • Verifies that client.profile-name or s3.profile-name results in the
    creation of an async AWS session with the correct profile, which is then
    passed to S3FileSystem.

The tests were run locally with:

pytest tests/catalog/test_glue_profile.py tests/io/test_fsspec_profile.py

Output would be:

==================== test session starts =====================
platform darwin -- Python 3.12.4, pytest-9.0.2, pluggy-1.6.0
rootdir: ${ROOTDIR}/iceberg-python
configfile: pyproject.toml
plugins: anyio-4.2.0, lazy-fixture-0.6.3, requests-mock-1.12.1
collected 3 items                                            
tests/catalog/test_glue_profile.py .                   [ 33%]
tests/io/test_fsspec_profile.py ..                     [100%]
===================== 3 passed in 1.02s ======================

Are there any user-facing changes?

Yes, this adds new configuration properties that users can set:

• client.profile-name: Sets the AWS profile for both the catalog client and FileIO (unified configuration).
• s3.profile-name: Sets the AWS profile specifically for S3 FileIO.

Example Usage:

catalog = GlueCatalog(
    "my_catalog",
    **{
        "client.profile-name": "my-aws-profile",
        # ... other config
    }
)

[kevinjqliu]

Thanks for the PR!
Left a comment about passing profile name.

[Copilot]

Pull request overview

This pull request adds AWS profile support for the Glue catalog client and fsspec-based S3 FileIO, addressing issue #2841. Users can now explicitly configure AWS profiles through client.profile-name (unified) and s3.profile-name (S3-specific) properties, with s3.profile-name taking precedence over client.profile-name for S3 operations. Similarly, glue.profile-name takes precedence over client.profile-name for Glue catalog operations.

Changes:

• Added AWS_PROFILE_NAME and S3_PROFILE_NAME configuration constants
• Extended GlueCatalog to support fallback from glue.profile-name to client.profile-name
• Implemented profile support in S3FileSystem by creating AioSession with the configured profile
• Added comprehensive unit tests validating profile propagation

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pyiceberg/io/init.py	Defines new constants AWS_PROFILE_NAME and S3_PROFILE_NAME for configuration properties
pyiceberg/catalog/glue.py	Updates GlueCatalog to fall back to client.profile-name when glue.profile-name is not set
pyiceberg/io/fsspec.py	Implements profile support by creating AioSession with the configured profile for S3FileSystem
tests/catalog/test_glue_profile.py	Adds test verifying GlueCatalog uses client.profile-name when provided
tests/io/test_fsspec_profile.py	Adds tests verifying S3FileIO uses both s3.profile-name and client.profile-name
tests/io/test_fsspec.py	Updates existing tests to include session=None parameter in S3FileSystem assertions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.