[revamp] Android NDK API 21 issue

[str4d]

Just when you thought I was done opening issues 😜

$ ANDROIDSDK=/home/str4d/dev/android/sdk ANDROIDAPI=21 ANDROIDNDK=/path/to/android-ndk-r10e p4a apk --private /src/ --package org.example.kivytest --name "Hello world" --version 1.0 --requirements kivy --dist_name kivy21
[INFO]:    This python-for-android revamp is an experimental alpha release!
[INFO]:    It should work (mostly), but you may experience missing features or bugs.
[INFO]:    Found Android API target in $ANDROIDAPI
[INFO]:    Available Android APIs are (4, 10, 19, 20, 21, 22)
[INFO]:    Requested API target 21 is available, continuing.
[INFO]:    Found NDK dir in $ANDROIDNDK
[INFO]:    Got NDK version from $ANDROIDNDKVER
[INFO]:    Got Android NDK version from the NDK dir: it is r10e-rc4
[INFO]:    Found virtualenv at /usr/local/bin/virtualenv
[INFO]:    No existing dists meet the given requirements!
[INFO]:    No dist exists that meets your requirements, so one will be built.
[INFO]:    Loaded recipe kivy (depends on [('sdl2', 'pygame'), 'pyjnius'])
[INFO]:    Loaded recipe pyjnius (depends on ['python2', ('sdl2', 'sdl')])
[INFO]:    Loaded recipe python2 (depends on ['hostpython2'], conflicts ['python3'])
[INFO]:    Loaded recipe hostpython2 (depends on [], conflicts ['hostpython3'])
[INFO]:    Found multiple valid recipe sets:
[INFO]:        ['pyjnius', 'python2', 'sdl2', 'hostpython2', 'kivy']
[INFO]:        ['hostpython2', 'pygame', 'kivy', 'sdl', 'pyjnius', 'python2']
[INFO]:    Using the first of these: ['pyjnius', 'python2', 'sdl2', 'hostpython2', 'kivy']
[INFO]:    Trying to find a bootstrap that matches the given recipes.
[INFO]:    Found 1 acceptable bootstraps: ['sdl2']
[INFO]:    Using the first of these: sdl2
[INFO]:    sdl2 bootstrap appears compatible with the required recipes.
[INFO]:    Checking this...
[INFO]:    Loaded recipe sdl2 (depends on ['python2', 'sdl2_image', 'sdl2_mixer', 'sdl2_ttf'], conflicts ['sdl', 'pygame', 'pygame_bootstrap_components'])
[INFO]:    Loaded recipe sdl2_image (depends on [])
[INFO]:    Loaded recipe sdl2_mixer (depends on [])
[INFO]:    Loaded recipe sdl2_ttf (depends on [])
[INFO]:    The selected bootstrap is sdl2
[INFO]:    # Creating dist with sdl2 bootstrap
[INFO]:    # Downloading recipes 
[INFO]:    # Building all recipes for arch armeabi
[INFO]:    # Unpacking recipes
[INFO]:    # Prebuilding recipes
[INFO]:    # Building recipes
[INFO]:    Building hostpython2 for armeabi
[INFO]:    Building sdl2_image for armeabi
[INFO]:    Building sdl2_mixer for armeabi
[INFO]:    Building sdl2_ttf for armeabi
[INFO]:    Building python2 for armeabi
[INFO]:    Building sdl2 for armeabi
[INFO]:    -> directory context /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni
[DEBUG]:   running ndk-build V=1
[DEBUG]:    Android NDK: WARNING: APP_PLATFORM android-21 is larger than android:minSdkVersion 12 in /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/AndroidManifest.xml
[DEBUG]:    Android NDK: WARNING:/home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni/src/Android.mk:main: non-system libraries in linker flags: -lpython2.7
[DEBUG]:    Android NDK:     This is likely to result in incorrect builds. Try using LOCAL_STATIC_LIBRARIES
[DEBUG]:    Android NDK:     or LOCAL_SHARED_LIBRARIES instead to list the library dependencies of the
[DEBUG]:    Android NDK:     current module
[DEBUG]:    rm -f /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/arm64-v8a/lib*.so /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi/lib*.so /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi-v7a/lib*.so /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi-v7a-hard/lib*.so /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/mips/lib*.so /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/mips64/lib*.so /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/x86/lib*.so /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/x86_64/lib*.so
[DEBUG]:    rm -f /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/arm64-v8a/gdbserver /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi/gdbserver /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi-v7a/gdbserver /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi-v7a-hard/gdbserver /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/mips/gdbserver /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/mips64/gdbserver /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/x86/gdbserver /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/x86_64/gdbserver
[DEBUG]:    rm -f /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/arm64-v8a/gdb.setup /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi/gdb.setup /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi-v7a/gdb.setup /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi-v7a-hard/gdb.setup /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/mips/gdb.setup /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/mips64/gdb.setup /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/x86/gdb.setup /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/x86_64/gdb.setup
[DEBUG]:    [armeabi] Install        : libSDL2.so => libs/armeabi/libSDL2.so
[DEBUG]:    install -p /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/obj/local/armeabi/libSDL2.so /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi/libSDL2.so
[DEBUG]:    /home/str4d/dev/android/android-ndk-r10e/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin/arm-linux-androideabi-strip --strip-unneeded  /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/libs/armeabi/libSDL2.so
[DEBUG]:    [armeabi] Compile thumb  : SDL2_image <= jidctfst.S
[DEBUG]:    /home/str4d/dev/android/android-ndk-r10e/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gcc -MMD -MP -MF /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/obj/local/armeabi/objs/SDL2_image/external/jpeg-9/jidctfst.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector -no-canonical-prefixes -march=armv5te -mtune=xscale -msoft-float -mthumb -Os -g -DNDEBUG -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -I/home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni/SDL2_image -I/home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni/SDL2_image/external/jpeg-9 -I/home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni/SDL2_image/external/libpng-1.6.2 -I/home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni/SDL/include -I/home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni/SDL2_image -DANDROID -DLOAD_BMP -DLOAD_GIF -DLOAD_LBM -DLOAD_PCX -DLOAD_PNM -DLOAD_TGA -DLOAD_XCF -DLOAD_XPM -DLOAD_XV -O3 -fstrict-aliasing -fprefetch-loop-arrays -DLOAD_JPG -DAVOID_TABLES -DLOAD_PNG -Wa,--noexecstack -Wformat -Werror=format-security    -I/home/str4d/dev/android/android-ndk-r10e/platforms/android-21/arch-arm/usr/include -c  /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni/SDL2_image/external/jpeg-9/jidctfst.S -o /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/obj/local/armeabi/objs/SDL2_image/external/jpeg-9/jidctfst.o
[DEBUG]:    /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni/SDL2_image/external/jpeg-9/jidctfst.S:1:0: warning: -fprefetch-loop-arrays not supported for this target (try -march switches) [enabled by default]
[DEBUG]:     /*
[DEBUG]:     ^
[DEBUG]:    In file included from /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/jni/SDL2_image/external/jpeg-9/jidctfst.S:17:0:
[DEBUG]:    /home/str4d/dev/android/android-ndk-r10e/platforms/android-21/arch-arm/usr/include/machine/cpu-features.h:52:6: error: #error Unknown or unsupported ARM architecture
[DEBUG]:     #    error Unknown or unsupported ARM architecture
[DEBUG]:          ^
[DEBUG]:    /home/str4d/dev/android/android-ndk-r10e/build/core/build-binary.mk:465: recipe for target '/home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/obj/local/armeabi/objs/SDL2_image/external/jpeg-9/jidctfst.o' failed
[DEBUG]:    make: *** [/home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2/obj/local/armeabi/objs/SDL2_image/external/jpeg-9/jidctfst.o] Error 1
Traceback (most recent call last):
  File "/home/str4d/.local/bin/p4a", line 11, in <module>
    sys.exit(main())
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2797, in main
    ToolchainCL()
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2442, in __init__
    getattr(self, args.command)(unknown)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 169, in wrapper_func
    args = build_dist_from_args(ctx, dist, args)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2221, in build_dist_from_args
    build_recipes(build_order, python_modules, ctx)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2036, in build_recipes
    recipe.build_arch(arch)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/recipes/sdl2/__init__.py", line 28, in build_arch
    shprint(sh.ndk_build, "V=1", _env=env)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 129, in shprint
    for line in output:
  File "/home/str4d/.local/lib/python2.7/site-packages/sh.py", line 565, in next
    self.wait()
  File "/home/str4d/.local/lib/python2.7/site-packages/sh.py", line 500, in wait
    self.handle_command_exit_code(exit_code)
  File "/home/str4d/.local/lib/python2.7/site-packages/sh.py", line 516, in handle_command_exit_code
    raise exc(self.ran, self.process.stdout, self.process.stderr)
sh.ErrorReturnCode_2: 

platforms/android-21/arch-arm/usr/include/machine/cpu-features.h:

#ifndef __ARM_ARCH__
#  if defined __ARM_ARCH_7__   || defined __ARM_ARCH_7A__ || \
        defined __ARM_ARCH_7R__  || defined __ARM_ARCH_7M__
#    define __ARM_ARCH__ 7
#  elif defined __ARM_ARCH_6__   || defined __ARM_ARCH_6J__ || \
        defined __ARM_ARCH_6K__  || defined __ARM_ARCH_6Z__ || \
        defined __ARM_ARCH_6KZ__ || defined __ARM_ARCH_6T2__
#    define __ARM_ARCH__ 6
#  else
#    error Unknown or unsupported ARM architecture
#  endif
#endif

Compare to platforms/android-19/arch-arm/usr/include/machine/cpu-features.h, which doesn't fail:

#ifndef __ARM_ARCH__

#  if defined __ARM_ARCH_7__   || defined __ARM_ARCH_7A__ || \
      defined __ARM_ARCH_7R__  || defined __ARM_ARCH_7M__

#    define __ARM_ARCH__ 7

#  elif defined __ARM_ARCH_6__   || defined __ARM_ARCH_6J__ || \
      defined __ARM_ARCH_6K__  || defined __ARM_ARCH_6Z__ || \
      defined __ARM_ARCH_6KZ__ || defined __ARM_ARCH_6T2__
#
#    define __ARM_ARCH__ 6
#
#  elif defined __ARM_ARCH_5__ || defined __ARM_ARCH_5T__ || \
        defined __ARM_ARCH_5TE__ || defined __ARM_ARCH_5TEJ__
#
#    define __ARM_ARCH__ 5
#
#  elif defined __ARM_ARCH_4T__
#
#    define __ARM_ARCH__ 4
#
#  elif defined __ARM_ARCH_4__
#    error ARMv4 is not supported, please use ARMv4T at a minimum
#  else
#    error Unknown or unsupported ARM architecture
#  endif
#endif

I can't find any references that justify these changes in the NDK, but they only seem to be in API 21. r10d and r10e both have the same issue.

[inclement]

The same problem was observed in the issue/PR for p4a master, with the solution being to switch the architecture target from armeabi to armeabi-v7 for these APIs. I guess we just have to do the same, but the Architecture stuff in the revamp isn't fully finished and it will be necessary to go through and make all the references correct. This would also be a step toward multi-architecture APKs.

[str4d]

Aha, I'd forgotten that I'd seen this in #410. Thanks for reminding me.

[inclement]

With the new arch support, this should now be easy to do.

[kived]

It would appear that at least as of Kitkat 4.4 (API 19), ARM devices are required to be at least ARMv7a:

MUST report, via android.os.Build.CPU_ABI, only one of the ABIs listed below
* armeabi-v7a
* x86
* mips

It's not altogether surprising that the API to follow 19 (Lollipop API 21) would drop support for ARMv5 completely. From what I've read, official builds of 4.4+ are always ARMv7a, but there are some custom builds for certain devices which run ARMv6.

So the fix is to target a lower API or target the armeabi-v7a architecture. Switching to v7a fixed this problem for me.

[kived]

Reopening, because I realize we can actually do something here. Maybe make p4a automatically select armeabi-v7a if API >= 21 and --arch was not specified, and if --arch is specified and is not compatible, then error out.

[inclement]

I added a check in a recent PR that causes an error and build failure if the api is too high (so that the user knows what's wrong and can pick a fix).