Merged
Conversation
osipovartem
requested review from
DanCodedThis,
Vedin,
YaroslavLitvinov,
eadgbear and
rampage644
Contributor
|
Instead of mocking the S3 client, we can explore using Bollard to start up a Minio docker container. |
Contributor
Author
This should be added as a part of integration tests, let's add unit tests first. We can add a separate workflow or mark some tests with specific tag to run them as integration. WDYT? |
Contributor
Fair. I just don't think that mocking an S3 client is necessary outside of Integration tests. Unit test should ideally be Memory only, then integration tests for File/S3-based tests, IMO |
rampage644
reviewed
Feb 18, 2025
Contributor
Author
That's why I added a mock to not use any real s3 client calls. This mock works in memory only and it allows us to cover create_storage_profile and validate_credentials endpoints by unit tests without any external calls |
DanCodedThis
approved these changes
Feb 19, 2025
rampage644
approved these changes
Feb 19, 2025
osipovartem
temporarily deployed
to
production
GitHub Actions
Inactive
eadgbear
pushed a commit
that referenced
this pull request
Feb 27, 2025
* Add CP service tests * Add create/delete session tests * Add create/delete session tests * Use converted S3 error
eadgbear
pushed a commit
that referenced
this pull request
Mar 6, 2025
* Add CP service tests * Add create/delete session tests * Add create/delete session tests * Use converted S3 error
eadgbear
added a commit
that referenced
this pull request
Mar 7, 2025
* Initial metastore crate * Metastore implementation * Upgrading Slatedb to 0.4, adding snapshot tests * Checkpoint commit * Checkpoint before crate re-org * Fix timestamp displaying format (#279) * fix timestamp format * Revert timestamp change * in login request use database_name instead of warehouse (#277) * Adding missing licenses + license check (#278) * Adding missing licenses + license check * Fixing license config path * Fix IPC arrow format issue (#280) * Fix IPC arrow format issue * Fix IPC arrow format issue * Fix comments * Control plane tests (batch 3) (#274) * Add CP service tests * Add create/delete session tests * Add create/delete session tests * Use converted S3 error * Move DBT_SERIALIZATION_FORMAT to cli (#286) * Move dbt_ser_fmt to cli * Rename the field * Update query runner with session id (#285) * Issue 115 Arrow support: replace uints by ints in resulted RecordBatch (#287) * replace uints by ints in resulted RecordBatch * cargo clippy --workspace --tests * rebase & fixes * Set warehouse location based on storage profile path (#288) * map float types to real logical_type (#289) * map float types to real logical_type * separate sessions tests to avoid blocking issues * unit test for floats datatype maped into real logical type * Snowflake timestamp_from_parts udf (#297) * Inital version * snowflake timestamp_from_parts udf * Fix clippy * Fix example typo * Support negative values * Snowflake time_from_parts udf (#298) * Inital version * snowflake timestamp_from_parts udf * snowflake time_from_parts udf * snowflake time_from_parts udf * Fix clippy * Fix clippy * Fix example typo * Support negative values * Merge * snowflake date_from_parts udf (#299) * Inital version * snowflake timestamp_from_parts udf * snowflake time_from_parts udf * snowflake time_from_parts udf * snowflake date_from_parts udf * Fix clippy * Implement extraction of the corresponding date part from a date or timestamp (#302) * Implement extracts the corresponding date part from a date or timestamp * Add units for visit expressions * Add geospatial udfs (#304) * Fix object store for custom endpoint (#295) AWS S3 builder uses dedicated setting to allow http. Parse endpoint and enable if it isn't https. * Unified query, execution, catalog * Adding temporary tables to metastore * Custom DataFusion Catalog implementation (WIP) * Adding execution User Session type * Checkpoint * Catalog implementation for IB * Crate re-org complete, writing tests * Changing metastore crate to use iceberg_rust types * Metastore crate cleanup * Create table query fixed * Fixing copy_into query * Finished updating query type * Cleaning up service type * Wrapping up changes to the execution crate * Bumping Datafusion git ref * Fixing up after package reorg * Fixed DBT handler * Removing old crates * Disabling some UI handlers and models until new metastore integration is done * Disabling extra OpenAPI endpoints and some unit tests while test are fixed * Deleting manifest files that snuck into the repo (my bad) * Final compilation and clippy fixes * `cargo fix` * `cargo fmt` * Adding license blocks to new files * Adding missing licenses + license check (#278) * Adding missing licenses + license check * Fixing license config path * Fix IPC arrow format issue (#280) * Fix IPC arrow format issue * Fix IPC arrow format issue * Fix comments * Control plane tests (batch 3) (#274) * Add CP service tests * Add create/delete session tests * Add create/delete session tests * Use converted S3 error * Move DBT_SERIALIZATION_FORMAT to cli (#286) * Move dbt_ser_fmt to cli * Rename the field * Snowflake timestamp_from_parts udf (#297) * Inital version * snowflake timestamp_from_parts udf * Fix clippy * Fix example typo * Support negative values * Implement extraction of the corresponding date part from a date or timestamp (#302) * Implement extracts the corresponding date part from a date or timestamp * Add units for visit expressions * Add geospatial udfs (#304) * patch chrono 0.4.40 as it breaks build (#310) * Added support for `day` and other datetime keywords in DF (#294) * start here * no macro, just simple vistor mut * remvoe redundant trait creation * integration test add, fixed logic, next unit tests * activated impl and cargo fmt + clippy * added unit tests + impl extension * cargo fmt + clippy * cargo clippy * cargo clippy * removed redundant check * fix * merged with Artem PR * cargo fmt + clippy * small fixes * cargo fmt + clippy * update hash commis for datafusion, iceberg-rust (#301) * Update tests.yml (#313) Adding github large runners to tests * Return hardcoded is_dynamic (#318) * Add geospatial UDF st_makeline (#315) * Add geospatial udfs * Add geospatial st_makeline * Fix cargo * Add lic header * Change allocator to snmalloc (#317) * Add ST_MakePolygon, st_polygon (#319) * Add ST_MakePolygon, st_polygon * Add docs * Add ST_DIMENSION (#321) * Add ST_MakePolygon, st_polygon * Add docs * Add ST_dimention * Add license * Add ST_PointN and ST_Endpoint (#322) * Add ST_MakePolygon, st_polygon * Add docs * Add ST_dimention * Add license * Add ST_Endpoint, ST_PointN * Reimplement ST_Point to work with snowflake linestring * Add St_x, st_y, st_srid (#324) * Add ST_MakePolygon, st_polygon * Add docs * Add ST_dimention * Add license * Add ST_Endpoint, ST_PointN * Reimplement ST_Point to work with snowflake linestring * Add St_x, st_y, st_srid * Moving UDF registration * Changing Metastore to use Dashmap * Adding missing geospatial functions after merge * Fixing unit test structure, still need to fix some unit tests * Post-merge cleanup --------- Co-authored-by: Artem Osipov <59066880+osipovartem@users.noreply.github.com> Co-authored-by: Yaroslav Litvinov <yaroslav@embucket.com> Co-authored-by: Denys Tsomenko <denistsem@gmail.com> Co-authored-by: Sergei Turukin <sergei@embucket.com> Co-authored-by: DanCodedThis <94703934+DanCodedThis@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to Extend tests coverage #264