Avoiding copies of arrays from python to rust

[grothesque]

Hello,

If I am not mistaken, it is possible to convert a Rust-allocated array into a NumPy array without copying the data: https://docs.rs/numpy/latest/numpy/convert/trait.IntoPyArray.html

However, I do not think that there's a way to go the other way around and obtain a Rust reference to the data of a NumPy array that is valid not only while the GIL is acquired by Rust code, but that is valid permanently (i.e. for the lifetime of the Python interpreter).

I believe that this is the case because there is no way to make sure that Rust's aliasing rules are enforced on the Python side. After all, as far as I know there is no way to make a NumPy array truly immutable.

But what if there was a Python array type that was truly immutable. Wouldn't it then be safe to obtain and hold multiple Rust references (even from different threads) to such an array? After all if these references from Rust are accounted for by Python's GC, they won't be destroyed. And since they are immutable, there's no way to modify them on the Python side as well.

(I am the author of the small numpy-like Python package tinyarray that happens to implement immutable NumPy-like arrays for CPython.)

[mejrs]

But what if there was a Python array type that was truly immutable. Wouldn't it then be safe to obtain and hold multiple Rust references (even from different threads) to such an array? After all if these references from Rust are accounted for by Python's GC, they won't be destroyed. And since they are immutable, there's no way to modify them on the Python side as well.

You can make your own pyclasses that do this. See https://docs.rs/pyo3/latest/pyo3/prelude/struct.Py.html#method.get

PyString and PyBytes also have a similar method.

[grothesque]

Well, OK, frozen pyclasses are for Python types that are implemented in Rust.

My case is indeed more like PyString or PyBytes. But I don't see a method like get. For PyBytes there is as_bytes, but it's part of the depracated gil-refs feature. I don't see a replacement.

[davidhewitt]

The replacement for PyBytes::as_bytes is PyBytesMethods::as_bytes; we could yet do a better job of documenting where to find it.

[mejrs]

Note also that the methods on bytes and string still need the gil held at the time when the reference is initially created. Frozen pyclasses do not.

[grothesque]

Thanks for the pointers. Indeed, I did not notice PyBytesMethods::as_bytes since it is implemented for Bound<PyBytes>.

[davidhewitt]

Hopefully PyO3/pyo3#4325 will help future users avoid the same frustration :)

[grothesque]

After reading through most of the discussions related to #274 (motivated by #433), I wonder:

Right now it doesn't seem to be possible to obtain a (mutable or read-only) view of a NumPy array that is valid not only while the GIL is held by Rust code. But isn't the safety simulation exactly the same as the one described in https://docs.rs/numpy/0.21.0/numpy/borrow/index.html, i.e. that numpy::borrow assures aliasing discipline for safe Rust code only and any unchecked code needs to be checked?

If this is true, shouldn't there be a way to obtain a view that lives beyond GIL acquisition?

[adamreichold]

i.e. that numpy::borrow assures aliasing discipline for safe Rust code only and any unchecked code needs to be checked?

This is correct. Note that one can produce data races using NumPy and plain Python code as NumPy releases the GIL when operating on primitive types and hence one can race on operations on NumPy arrays without writing any C or unsafe Rust code. This is the main argument as to why rust-numpy consider Python code exercising the NumPy API unsafe/trusted.

If this is true, shouldn't there be a way to obtain a view that lives beyond GIL acquisition?

To clarify, you mean a safe API to obtain such a view? We could certainly do that by keeping our internal dynamic borrow alive and tucking away a Py so that the reference count of the underlying Python object does reach zero.

However, I already consider the soundness situation of the current rust-numpy safe API precarious and if nothing else then at least hard to explain. So I would not want to make it even more complicated by additionally decoupling our dynamic borrow checking from the GIL lifetime which writers of trusted Python code would need to keep in mind.

However, this is purely a judgement call w.r.t. explainability/usability so reasonable people might come to different conclusions.