Proposal to standardize element-wise arithmetic operations

[kgryte]

Based on the analysis of array library APIs, we know that performing element-wise arithmetic operations is both universally implemented and commonly used. Accordingly, this issue proposes to standardize the following arithmetic operations:

Arithmetic Operations

• add
• subtract
• multiply (mul)
• divide (div)

Criterion

1. Commonly implemented across array libraries.
2. Commonly used by array library consumers.
3. Operates on two arrays.

Questions

1. Naming conventions? Currently, this proposal is biased toward verbose API names following NumPy.
2. Are there any APIs listed above which should not be standardized?
3. Are there basic arithmetic operations not listed above which should be standardized? Preferably, any additions should be supported by usage data.

[kgryte]

I compiled generalized signatures (with respect to each of the above listed interfaces for each library) for element-wise arithmetic operations, where the raw signature data can be found here.

NumPy

numpy.<name>(x1, x2, out=None, *, where=True, casting='same_kind', order='K', dtype=None, subok=True[, signature, extobj]) → ndarray

CuPy

cupy.<name>(x1, x2, out=None, dtype=None) → ndarray

dask.array

JAX

jax.numpy.<name>(x1, x2) → ndarray

MXNet

np.<name>(x1, x2, out=None, **kwargs) → ndarray

PyTorch

torch.<name>(input, other, out=None) → Tensor

Tensorflow

tf.math.<name>(x, y, name=None) → Tensor

---

The minimum common API across most libraries is

<name>(x1, x2, out=None)

For example,

add(x1, x2, out=None)

---

Proposal

Signature of the form:

<name>(x1, x2, *, out=None)

APIs:

add(x1, x2, *, out=None)
subtract(x1, x2, *, out=None)
multiply(x1, x2, *, out=None)
divide(x1, x2, *, out=None)

Notes

Optional arguments as keyword-only arguments for the following reasons:

1. Avoid potential positional variation amongst library implementations.
2. Favor explicit interfaces and minimize readers' need to intuit an optional positional argument's meaning.

[kgryte]

See #12 for a draft specification.

[shoyer]

Meta note: it might be more descriptive to call these "binary arithmetic operations". An operation like abs(x) or -x is also arguable "arithmetic".

My discussion about positional-only arguments and out from #8 is equally relevant here -- we should make a shared decision for both.

[kgryte]

@shoyer Re: naming. The operations in this proposal are not intended to be exclusive. My intent with the issue name was simply to distinguish from the other proposals, but point taken and informs how we might categorize APIs upon formal inclusion in the specification.

[kgryte]

And agreed regarding out.

@rgommers This may be good topic of discussion to add to the agenda for the next meeting.

[shoyer]

Taking a step back: do need these binary arithmetic operations at all when we have access to Python's infix operators?

I'm glad we have codified these semantics, but perhaps it would suffice to recommend using either +/+= or operator.add/operator.iadd rather than defining your own add function? At the very least we should formalize the relationship between these.

[rgommers]

That's a good point. I personally can't remember having ever used np.add over +.

Did a quick search of the SciPy code base, and the only uses of np.add are using np.add.reduce, and that's because at some point in the past np.add.reduce was significantly faster than np.sum.

That said, PyTorch uses torch.add all over the place, also without out or another keyword. It's typed more strictly than __add__, so there's probably a reason.

[shoyer]

That said, PyTorch uses torch.add all over the place, also without out or another keyword. It's typed more strictly than __add__, so there's probably a reason.

tensorflow.add() is also used all over the place. As far as I can tell it's not for any particularly good reason -- mostly just copied from some early examples, possibly examples written by coders who didn't know Python terribly well. It does have an optional name parameter, but that's rarely used.

In theory, tf.add() does have slightly stricter typing semantics (everything gets converted into a Tensor), but TensorFlow is in the process of adding its own dispatch system with __tf_dispatch__ so this will also change.

I know PyTorch also has experimental dispatch, so I suspect the situation there could be pretty similar.

[kgryte]

Downstream libraries do use, e.g., numpy.add (see here and here). An example of using numpy.add in pandas test usage can be found here.

Apart from stricter typing semantics, functional equivalents may be preferred over operator equivalents for purposes of fluent interfaces, lazy evaluation, etc, so I might advise against recommending operator equivalents and not standardizing element-wise arithmetic operation interfaces. These interfaces are widely implemented among analyzed array libraries.

[shoyer]

Downstream libraries do use, e.g., numpy.add (see here and here). An example of using numpy.add in pandas test usage can be found here.

This test is explicitly checking overrides of NumPy's ufuncs (which np.add is), so I don't think this is a great example.

Apart from stricter typing semantics, functional equivalents may be preferred over operator equivalents for purposes of fluent interfaces, lazy evaluation, etc,

operator.add(a, b) from Python's standard library is a builtin function that is exactly equivalent to a + b. We don't need something new for that.

These interfaces are widely implemented among analyzed array libraries.

True, but many of these libraries, including CuPy, Dask and JAX, try to copy the NumPy interface exactly. A function like add() existing may be more of an indication that it was easy to add than an indication that it is actually useful.

(Note that your list is missing dask.array.add, but that does seem to exist.)

[kgryte]

@shoyer I don't see dask.array.add in its docs. Perhaps you can point to where I can find it? If it exists (apart from being a dunder method), I'd like to update the comparison data.

Re: pandas examples. I simply pulled one usage from a GitHub search. Other examples include here and here. You may be able to find others, or they may all fall into the same category. All this to say is that downstream libraries do use these functional equivalents, as evidenced by the record data where all downstream libraries we've analyzed (pandas, dask.array, xarray, scikit-image, matplotlib) invoked add, subtract, divide, and/or multiply when we've run their test suites.

Re: operator.add. Aware.

Re: other array libraries. Would be good to have some record data for API consumption beyond NumPy. But this is still a WIP.

[kgryte]

As a further comment, I think its worth reiterating the main stated goal of the consortium which is to coalesce around what is presently implemented across array libraries. Meaning, we aren't writing an array library spec from first principles.

As such, we'd need to ask, if we left add, subtract, multiply, and divide out of the spec, what would be the desired outcome? Would NumPy simply drop support for element-wise arithmetic interfaces (meaning that numpy.add would no longer exist)? Or would NumPy retain these interfaces? And if retained, would other array libraries continue to provide such interfaces (potentially following NumPy's lead)? If the answer to the last two questions is "yes", then what we after here is to ensure uniformity amongst the array libraries so that users/devs can expect similar signatures as they move from library to library.

I recognize @shoyer that your concern is forward looking. You'd rather not impose undue burdens on future array libraries. However, unless most/all the currently analyzed array libraries remove these interfaces, we're left with de facto standardization, rather than de jure, and without the consistency guarantees afforded by specification compliance.

[saulshanabrook]

Well one advantage of not having them included, even if all array libraries continue to provide them, is that then downstream users won't use them if they are writing to the spec. So it could have some influence there, if we think users should be doing a + b over np.add(a, b), because it's more Pythonic and there should be one and only one way to do things?

[shoyer]

@shoyer I don't see dask.array.add in its docs. Perhaps you can point to where I can find it? If it exists (apart from being a dunder method), I'd like to update the comparison data.

It may not be documented, but it definitely exists:

In [4]: import dask.array

In [5]: dask.array.add(1, 1)
Out[5]: 2

In [6]: type(dask.array.add)
Out[6]: dask.array.ufunc.ufunc

As such, we'd need to ask, if we left add, subtract, multiply, and divide out of the spec, what would be the desired outcome? Would NumPy simply drop support for element-wise arithmetic interfaces (meaning that numpy.add would no longer exist)? Or would NumPy retain these interfaces?

NumPy is certainly not going to drop numpy.add, regardless of what we decide here, but I don't think should be remotely relevant for our decision making here.

There is guaranteed to be a large list of functionality in all of these array libraries that doesn't get standardized. If we insist that libraries remove existing functionality, this spec would never get off the ground.

If our spec is open to extensibility, then I can see a case for allowing optional functions for arithmetic, e.g,. so TensorFlow can expose the optional name parameter. And certainly if a project like TensorFlow makes an add() function, its default behavior should be guaranteed to exactly match +.