You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current default for these operations on Spark arrays is axis=(0,), which may incur a swap to distribute along that axis (if it isn't already). The default could instead be axis=None which would mean apply over the distributed axes (whatever they are) and would never incur a swap.
This generally seems like a more friendly default, the only issue arises not with map but with reduce, when considering sequences of mixed operations. For example, in the following two cases where the map is a no-op,
if the default for reduce is over the partitioned axes, the answer will be different in the two cases, whereas if the default is over axis=(0,) it will be the same.
I can see an argument that these really should be the same with the default parameters, but curious to get other opinions. Another option is using different defaults for map/filter and reduce.
The current default for these operations on Spark arrays is
axis=(0,)
, which may incur a swap to distribute along that axis (if it isn't already). The default could instead beaxis=None
which would mean apply over the distributed axes (whatever they are) and would never incur aswap
.Suggested by @shoyer, thanks!
This generally seems like a more friendly default, the only issue arises not with
map
but withreduce
, when considering sequences of mixed operations. For example, in the following two cases where themap
is a no-op,if the default for
reduce
is over the partitioned axes, the answer will be different in the two cases, whereas if the default is overaxis=(0,)
it will be the same.I can see an argument that these really should be the same with the default parameters, but curious to get other opinions. Another option is using different defaults for
map
/filter
andreduce
.cc @andrewosh
The text was updated successfully, but these errors were encountered: