Support for newer API version

[mattiapenati]

The current implementation is targeting the ABI v2 and the API v1.15. This choice ensures a wide compatibility of the extensions built using this crate. As already discussed in #534 it can be useful for extension's developers to choose the target API version, in order to make the newer features of NumPy available.

I try to summarize the previous discussion:

• It's fine to target only the ABI v2.
• It can be helpful for extensions' developers to make the choice of the target API version in an explicit way. With runtime checks it's hard to define a good NumPy minimum version.
• Features are not usable because of their additive design and because NumPy does not use semantic versioning.

It seems that an easy solution to provide API selection is not feasible, both at runtime and compile time. The usage of mutually exclusive features was not discussed, I think it is a viable solution and it is hard for me to think of an other solution. I know that it is not without drawbacks, in particular a compilation error arises if multiple incompatible features are enabled in the dependency graph, but how common is this situation?

I'm going to cite all the users that have participated in the previous discussion. I hope you are already interested in this topic, if not, I apologize for the noise. @Icxolu @ngoldbaum @seberg

[Icxolu]

Thank you for continuing to push this.

I recently wondered if we could make use of cargo's metadata table together with extending our build script to make API selection possible via cfgs like PyO3.

Something like this:

# Cargo.toml
# ...

[package.metadata.numpy]
api = "1.18"

which would then enable the flags NPY_1_15, NPY_1_16, NPY_1_17, NPY_1_18.
We could set a sane default for when no metadata is present and let users explicitly overwrite it with it if needed.

[Icxolu]

While mutually exclusive features are more restrictive, I believe this limitation is acceptable: in nearly all cases, the numpy crate is used to write compiled extensions directly, and not to develop an intermediate crate that other crates depend on.

There are quite some crates that use numpy in their dependencies. It's probably also not super uncommon to split extensions in multiple crates. Mutually exclusive features make working with dependencies that themself depend on numpy really annoying, as they have to reexport all the features themself to be usable. So from a usability perspective mutable exclusive features are probably quite bad. I think we have to guess which one is more likely to be triggered by users. My gut feeling says that mutable exclusive features are more likely to cause trouble than not fully additive features due to numpy's versioning. Maybe @ngoldbaum can give us an opinion on this with his numpy hat on.

Another minor downside of mutually exclusive features is, that for each API we have to list every version that is is available in, rather than just the minimum (and potentially maximum).

[ngoldbaum]

The downside is that compilation errors may occur when using changed or removed APIs, and without clear guidance, and the error messages can be misleading.

Are there specific NumPy APIs you're worried about here? While the NumPy 1.x API and ABI are incompatible with the NumPy 2.x API and ABI, for NumPy 2.x I think you're probably safe to not have mutually exclusive verisons and write the FFI wrappers like how we do it in pyo3-ffi. I'd actually consider it a NumPy bug if you're forced to support each version of the NumPy 2.x APIs as disjoint API surfaces. We try to make our API and ABI evolution backward compatible, at least for the public API.

[Icxolu]

I don't think we have something specific in mind here, just that numpy's versioning would theoretically allow it:

It is important to know that NumPy, like Python itself and most other well known scientific Python projects, does not use semantic versioning. Instead, backwards incompatible API changes require deprecation warnings for at least two releases. For more details, see NEP 23 — Backwards compatibility and deprecation policy.

---

I think you're probably safe to not have mutually exclusive verisons and write the FFI wrappers like how we do it in pyo3-ffi. I'd actually consider it a NumPy bug if you're forced to support each version of the NumPy 2.x APIs as disjoint API surfaces. We try to make our API and ABI evolution backward compatible, at least for the public API.

That sounds like it would be pretty rare that numpy would make use of such an backwards incompatible api change. For this proposal I would say that means then that we should go towards additive features. If we would ever encounter something that we can't model with that we can figure out what to do from there.

[ngoldbaum]

I think the NEP is talking about the Python API there, the C API is much more stable.

[Icxolu]

Great! Thanks for lending us your expertise here 💯