Update add_files operation to support branches

[greenlaw]

Feature Request / Improvement

Write-to-branch functionality was added in #941, but it appears that support for this functionality in add_files operations was notably missing. It would be great if this workflow could be supported in the same way as other write operations.

I also dropped a message about this in the slack channel about this.

[ForeverAngry]

Thats a pretty good question! Do you know, does the JAVA api support this? Ill take a look as well, when i get a moment.

[greenlaw]

Yes, it does. With the Java API I'm able to use AppendFiles, which supports toBranch() along with DataFiles.builder

The code looks something like:

AppendFiles append = table.newAppend().toBranch("my_branch");
InputFile inputFile = HadoopInputFile.fromLocation(uri, hadoopConf);
Metrics metrics = ParquetUtil.fileMetrics(inputFile, metricsConfig);
DataFile file = DataFiles.builder(table.spec())
              .withPath(path)
              .withFormat(FileFormat.PARQUET)
              .withFileSizeInBytes(inputFile.getLength())
              .withMetrics(metrics)
              .build();
append.appendFile(file);
append.commit();

[kevinjqliu]

Thanks for opening this issue. Makes sense to me to allow the add_files operation to support branches.

I think we currently have limited capability to merge a branch back to main, just so you are aware of that

[greenlaw]

@kevinjqliu Thanks for the heads up! For my use case, I don't actually care about merging branches - I just need the ability to create and evolve divergent branches on a single table from a common ancestor.

With add_files branching support, plus the snapshot expiration capabilities in the upcoming 0.10.0 release, I think that's all I would need to move away from the Java API, although I'm still trying to identify any other feature gaps, so I appreciate your comment.

[vinjai]

Hey @greenlaw,
Thanks a lot for pointing this out 🙏. I had initially skipped adding this feature since the function is similar with the Spark implementation of procedure: add_files

That said, I now realize there is not a straightforward way to achieve this in PyIceberg the way it is possible in the Java implementation. Your point makes complete sense and it would be valuable to add this as a feature.

I will raise a PR shortly. It should be a quick fix.

[greenlaw]

@vinjai PR would be greatly appreciated!

It seemed pretty straightforward to wire it up with the snapshot update class, so I started on my own PR but ran into some issues running integration tests on my machine and haven't gotten back to it yet. Hopefully I can straighten out my dev workflow and contribute in the future.

[vinjai]

Hey @greenlaw
Have raised the PR. Hope this addresses your concerns

[greenlaw]

Awesome, thanks so much @vinjai !