Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Android crash for unsupported CPU instruction set at ggml_vec_dot_q4_K_q8_K #317

Open
WaqasGameDev opened this issue Feb 20, 2025 · 2 comments
Open

Android crash for unsupported CPU instruction set at ggml_vec_dot_q4_K_q8_K #317

WaqasGameDev opened this issue Feb 20, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@WaqasGameDev
Copy link

WaqasGameDev commented Feb 20, 2025 •
edited
Loading

Describe the bug

Hi, I have been testing 2 wrappers for llama.cpp. One is from you with unity bindings and one is from nerve sparks Though both of you are using different commit of llama.cpp but as far as ggml_vec_dot_q4_K_q8_K function is concerned, I do not see any difference between your commit of llama. cpp and their commit

But when i run their kotlin based android app, it is a bit slow in response i.e. it takes 4 to 5 seconds in response or more but it does not crash. But when I try to run your library for unity, it crashes for certain cpu types. I'm just wondering why it is happening and is there a way we can fix it?

Here are the crash logs

0001/01/01 00:00:00.000 -1 -1 Info  --------- beginning of crash
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime FATAL EXCEPTION: UnityMain
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime Process: com.DefaultCompany.llm_testing, PID: 2309
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime java.lang.Error: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime Version '2021.3.14f1 (eee1884e7226)', Build type 'Release', Scripting Backend 'il2cpp', CPU 'arm64-v8a'
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime Build fingerprint: 'google/blueline/blueline:12/SP1A.210812.016.C2/8618562:user/release-keys'
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime Revision: 'MP1.0'
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime ABI: 'arm64'
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime Timestamp: 2025-02-21 01:47:06+0500
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime pid: 2309, tid: 2519, name: Thread-7  >>> com.DefaultCompany.llm_testing <<<
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime uid: 10455
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime signal 4 (SIGILL), code 1 (ILL_ILLOPC), fault addr 0x6e42702c1c (*pc=0x4e909482)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime     x0  0000000000000800  x1  0000006e4ac4f5e0  x2  0000000000000000  x3  0000006e1f8601a0
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime     x4  0000000000000000  x5  00000071cafd1010  x6  0000000000000000  x7  0000000000000001
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime     x8  0000000000000124  x9  0000000000000920  x10 0000000030303030  x11 000000000f0f0f0f
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime     x12 00000000b31b4b90  x13 00000071cafd1010  x14 00000000baf3f2bf  x15 0000006e1f8601d0
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime     x16 0000006e4281b478  x17 0000006e42716994  x18 0000006e48c00000  x19 0000000000000000
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime     x20 0000006e42702bb8  x21 0000000000000001  x22 00000071cafd1010  x23 0000000000000001
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime     x24 0000000000000010  x25 0000006e1f8601a0  x26 0000006e4ac4f5e0  x27 0000000000000000
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime     x28 0000000000000000  x29 0000006e4ac4f740
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime     sp  0000006e4ac4eed0  lr  0000006e426d9274  pc  0000006e42702c1c
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime backtrace:
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #00 pc 000000000045fc1c  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (ggml_vec_dot_q4_K_q8_K+100) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #01 pc 0000000000436270  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (ggml_graph_compute_thread+16276) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #02 pc 00000000004320b4  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (ggml_graph_compute+308) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #03 pc 000000000042db68  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (ggml_backend_cpu_graph_compute(ggml_backend*, ggml_cgraph*)+120) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #04 pc 000000000048a8c8  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (ggml_backend_sched_graph_compute_async+1284) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #05 pc 000000000039e4c8  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (llama_graph_compute(llama_context&, ggml_cgraph*, int, ggml_threadpool*)+124) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #06 pc 00000000003799e0  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (llama_decode+2616) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #07 pc 0000000000323200  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (common_init_from_params(common_params&)+3084) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #08 pc 000000000021f478  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (server_context::load_model(common_params const&)+184) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #09 pc 0000000000222ac0  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (LLM::init(int, char**)+1204) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #10 pc 0000000000222480  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (LLM::LLM(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >)+544) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #11 pc 0000000000235a2c  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libundreamai_android.so (LLM_Construct+160) (BuildId: 065359c016d53d6a4303109622da0026d440c0c8)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #12 pc 0000000000d23598  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #13 pc 0000000000d0fe64  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #14 pc 0000000000d0d41c  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #15 pc 0000000000d0ce68  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #16 pc 0000000000d0c120  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #17 pc 0000000000c03024  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #18 pc 0000000000bf6320  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #19 pc 0000000000c033fc  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #20 pc 0000000000c032c0  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #21 pc 0000000000bfa594  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #22 pc 00000000003c94b8  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #23 pc 0000000000430fcc  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #24 pc 000000000041e864  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #25 pc 000000000049219c  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #26 pc 00000000004071b4  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #27 pc 00000000004a1ecc  /data/app/~~NVLAiOYcEoCMgBpjx-T4pQ==/com.DefaultCompany.llm_testing-P_2W50nud8Z9lGXjS6l1tA==/lib/arm64/libil2cpp.so (BuildId: 9fb84c452ff5af1aa05a189475b23e3a639b41d7)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #28 pc 00000000000b1910  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+264) (BuildId: ba489d4985c0cf173209da67405662f9)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime       #29 pc 00000000000513f0  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: ba489d4985c0cf173209da67405662f9)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.ggml_vec_dot_q4_K_q8_K(ggml_vec_dot_q4_K_q8_K:100)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.ggml_graph_compute_thread(ggml_graph_compute_thread:16276)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.ggml_graph_compute(ggml_graph_compute:308)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.ggml_backend_cpu_graph_compute(ggml_backend*, ggml_cgraph*)(ggml_backend_cpu_graph_compute:120)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.ggml_backend_sched_graph_compute_async(ggml_backend_sched_graph_compute_async:1284)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.llama_graph_compute(llama_context&, ggml_cgraph*, int, ggml_threadpool*)(llama_graph_compute:124)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.llama_decode(llama_decode:2616)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.common_init_from_params(common_params&)(common_init_from_params:3084)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.server_context::load_model(common_params const&)(load_model:184)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.LLM::init(int, char**)(init:1204)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.LLM::LLM(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >)(LLM:544)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libundreamai_android.LLM_Construct(LLM_Construct:160)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xd23598(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xd0fe64(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xd0d41c(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xd0ce68(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xd0c120(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xc03024(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xbf6320(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xc033fc(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xc032c0(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0xbfa594(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0x3c94b8(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0x430fcc(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0x41e864(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0x49219c(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0x4071b4(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libil2cpp.0x4a1ecc(Native Method)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libc.__pthread_start(void*)(__pthread_start:264)
2025/02/21 01:47:07.349 2309 2345 Error AndroidRuntime 	at libc.__start_thread(__start_thread:64)

and here I am testing it on GOOGLE PIXEL 3 device.

Steps to reproduce

Just make a simple unity build with Llama 3.2 1B model (i think it will fail with other models too) and it happens as soon as app is launched

LLMUnity version

Unity 2021.3.14f1

Operating System

Android
@WaqasGameDev WaqasGameDev added the bug Something isn't working label Feb 20, 2025
@amakropoulos
Copy link
Collaborator

Thank you for the details, I'll look into it!

@WaqasGameDev
Copy link
Author

Hi @amakropoulos we spent more time on it and as per our undersranding, your library is optimized for cpu having support for dot product i.e. asimddp support and that is why it just crashes for the mobile cpu's those does not have this support. So for now, we are just not using your library for unsupported mobile cpus.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@amakropoulos @WaqasGameDev