xnnpack cannot be cross built from source, because its python3
dependency on a host architecture interpreter cannot be installed. It
really wants to run Python for its scripts and tools during build.
Therefore, it should be requesting a build architecture interpreter by
annotating it :native. I'm attaching a patch for your convenience. Once
applying it, xnnpack cross builds successfully.

Helmut