Multi-arch container builds are having some trouble

[shaneutt]

We recently updated our multi-arch container build, and since then I've seen quite a few failures that appear may be flakes and on success, I've seen it take as long as 70 minutes.

This isn't tenable to run on every PR, and each update for that PR. We need to evaluate a different strategy for container builds. Perhaps we can do the full multi-arch build on releases, but for PRs only do a basic x86_64 build?

The purpose of this issue is to explore a bit and see what potential solutions there are, pick one and implement it.

[jokestax]

Hey @shaneutt, could I take this up? I’ve been experimenting with matrix builds and exploring this (https://github.com/taiki-e/upload-rust-binary-action)] I’ll raise a PR with my implementation by this week.

[aryan9600]

the point of doing multi-arch builds for PRs is to verify that we aren't breaking compatibility with arm or x86. since our project relies on quite a bit of third party deps and libraries, its relatively easy to make a change that breaks on a platform.
one thing we can do to speed up CI for PRs, is to split the build into two different jobs, one for each arch. we would build the image for each architecture, but do so natively and in parallel.

we can do a proper multi arch build for releases.

[shaneutt]

@jokestax it would be great if you want to take a look at this, thank you!

/assign @jokestax

Keep in mind what @aryan9600 has said above and confer with him going forward. Particularly:

we can do a proper multi arch build for releases.

I do think we should probably only run these tests (only) on releases.

If this ends up getting very difficult or nebulous for some reason though, please check back in here and we'll support you.