有没有类似llama.cpp的加速，现存的库好像都不支持baichuan2的加速

llama.cpp我提了一个PR：<a class="issue-link js-issue-link" data-error-text="Failed to load t

<a class="user-mention notranslate" data-hovercard-type="user" data-hover

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

最新版本的llama.cpp install Python dependencies <p dir

Comments (15)

jameswu2014 commented on May 21, 2024 4

llama.cpp我提了一个PR：ggerganov/llama.cpp#3009
你先按照Baichuan2的Readme里的Baichuan2->Baichuan1 的lm_head转换修改一下模型，就可以用上面链接里的修改。

from baichuan2.

songkq commented on May 21, 2024 2

@jameswu2014 @dlutsniper quantize gguf model failed on RTX3090 with Driver Version: 525.105.17 CUDA Version: 12.0. Could you please give some advice for this issue?
./quantize /workspace/llama.cpp/models/Baichuan2-13B-Chat-ggml-model-f16.gguf /workspace/llama.cpp/models/Baichuan2-13B-Chat-ggml-model-Q8_0.gguf 7

CUDA error 804 at /llama.cpp/ggml-cuda.cu:5522: forward compatibility was attempted on non supported HW
current device: 0

Solved by building a docker image from nvidia/cuda:12.0.0-devel-ubuntu22.04

from baichuan2.

dereklll commented on May 21, 2024 2

(alpaca_env) chunzhamini@chunzhamini llama.cpp % ./main -m ./zh-models/baichuan/Baichuan2-13B-Chat-ggml-model-q4_0.bin -p '从前有一只小狐狸，他' --temp 0 -ngl 1 Log start main: warning: changing RoPE frequency base to 0 (default 10000.0) main: warning: scaling RoPE frequency by 0 (default 1.0) main: build = 1270 (c091cdf) main: built with Apple clang version 14.0.3 (clang-1403.0.22.14.1) for arm64-apple-darwin22.5.0 main: seed = 1695699630 llama_model_loader: loaded meta data with 20 key-value pairs and 363 tensors from ./zh-models/baichuan/Baichuan2-13B-Chat-ggml-model-q4_0.bin (version GGUF V2 (latest)) llama_model_loader: - tensor 0: token_embd.weight q4_0 [ 5120, 125696, 1, 1 ] llama_model_loader: - tensor 1: blk.0.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 2: blk.0.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 3: blk.0.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 4: blk.0.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 5: blk.0.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 6: blk.0.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 7: blk.1.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 8: blk.1.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 9: blk.1.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 10: blk.1.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 11: blk.1.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 12: blk.1.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 13: blk.2.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 14: blk.2.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 15: blk.2.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 16: blk.2.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 17: blk.2.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 18: blk.2.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 19: blk.3.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 20: blk.3.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 21: blk.3.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 22: blk.3.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 23: blk.3.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 24: blk.3.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 25: blk.4.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 26: blk.4.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 27: blk.4.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 28: blk.4.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 29: blk.4.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 30: blk.4.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 31: blk.5.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 32: blk.5.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 33: blk.5.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 34: blk.5.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 35: blk.5.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 36: blk.5.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 37: blk.6.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 38: blk.6.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 39: blk.6.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 40: blk.6.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 41: blk.6.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 42: blk.6.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 43: blk.7.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 44: blk.7.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 45: blk.7.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 46: blk.7.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 47: blk.7.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 48: blk.7.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 49: blk.8.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 50: blk.8.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 51: blk.8.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 52: blk.8.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 53: blk.8.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 54: blk.8.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 55: blk.9.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 56: blk.9.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 57: blk.9.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 58: blk.9.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 59: blk.9.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 60: blk.9.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 61: blk.10.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 62: blk.10.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 63: blk.10.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 64: blk.10.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 65: blk.10.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 66: blk.10.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 67: blk.11.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 68: blk.11.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 69: blk.11.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 70: blk.11.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 71: blk.11.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 72: blk.11.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 73: blk.12.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 74: blk.12.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 75: blk.12.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 76: blk.12.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 77: blk.12.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 78: blk.12.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 79: blk.13.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 80: blk.13.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 81: blk.13.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 82: blk.0.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 83: blk.0.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 84: blk.0.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 85: blk.1.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 86: blk.1.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 87: blk.1.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 88: blk.2.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 89: blk.2.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 90: blk.2.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 91: blk.3.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 92: blk.3.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 93: blk.3.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 94: blk.4.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 95: blk.4.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 96: blk.4.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 97: blk.5.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 98: blk.5.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 99: blk.5.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 100: blk.6.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 101: blk.6.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 102: blk.6.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 103: blk.7.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 104: blk.7.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 105: blk.7.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 106: blk.8.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 107: blk.8.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 108: blk.8.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 109: blk.9.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 110: blk.9.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 111: blk.9.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 112: blk.10.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 113: blk.10.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 114: blk.10.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 115: blk.11.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 116: blk.11.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 117: blk.11.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 118: blk.12.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 119: blk.12.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 120: blk.12.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 121: blk.13.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 122: blk.13.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 123: blk.13.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 124: blk.13.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 125: blk.13.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 126: blk.13.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 127: blk.14.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 128: blk.14.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 129: blk.14.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 130: blk.14.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 131: blk.14.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 132: blk.14.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 133: blk.15.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 134: blk.15.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 135: blk.15.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 136: blk.15.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 137: blk.15.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 138: blk.15.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 139: blk.16.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 140: blk.16.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 141: blk.16.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 142: blk.16.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 143: blk.16.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 144: blk.16.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 145: blk.17.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 146: blk.17.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 147: blk.17.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 148: blk.17.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 149: blk.17.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 150: blk.17.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 151: blk.18.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 152: blk.18.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 153: blk.18.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 154: blk.18.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 155: blk.18.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 156: blk.18.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 157: blk.19.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 158: blk.19.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 159: blk.19.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 160: blk.19.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 161: blk.19.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 162: blk.19.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 163: blk.20.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 164: blk.20.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 165: blk.20.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 166: blk.20.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 167: blk.20.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 168: blk.20.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 169: blk.21.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 170: blk.21.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 171: blk.21.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 172: blk.21.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 173: blk.21.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 174: blk.21.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 175: blk.22.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 176: blk.22.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 177: blk.22.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 178: blk.22.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 179: blk.22.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 180: blk.22.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 181: blk.23.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 182: blk.23.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 183: blk.23.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 184: blk.23.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 185: blk.23.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 186: blk.23.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 187: blk.24.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 188: blk.24.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 189: blk.24.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 190: blk.24.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 191: blk.24.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 192: blk.24.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 193: blk.25.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 194: blk.25.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 195: blk.25.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 196: blk.25.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 197: blk.25.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 198: blk.25.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 199: blk.26.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 200: blk.26.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 201: blk.26.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 202: blk.26.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 203: blk.26.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 204: blk.26.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 205: blk.27.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 206: blk.27.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 207: blk.27.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 208: blk.27.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 209: blk.27.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 210: blk.27.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 211: blk.28.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 212: blk.28.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 213: blk.28.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 214: blk.28.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 215: blk.28.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 216: blk.28.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 217: blk.29.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 218: blk.29.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 219: blk.14.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 220: blk.14.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 221: blk.14.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 222: blk.15.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 223: blk.15.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 224: blk.15.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 225: blk.16.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 226: blk.16.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 227: blk.16.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 228: blk.17.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 229: blk.17.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 230: blk.17.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 231: blk.18.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 232: blk.18.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 233: blk.18.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 234: blk.19.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 235: blk.19.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 236: blk.19.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 237: blk.20.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 238: blk.20.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 239: blk.20.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 240: blk.21.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 241: blk.21.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 242: blk.21.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 243: blk.22.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 244: blk.22.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 245: blk.22.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 246: blk.23.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 247: blk.23.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 248: blk.23.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 249: blk.24.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 250: blk.24.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 251: blk.24.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 252: blk.25.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 253: blk.25.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 254: blk.25.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 255: blk.26.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 256: blk.26.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 257: blk.26.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 258: blk.27.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 259: blk.27.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 260: blk.27.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 261: blk.28.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 262: blk.28.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 263: blk.28.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 264: blk.29.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 265: blk.29.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 266: blk.29.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 267: blk.29.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 268: blk.29.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 269: blk.29.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 270: blk.29.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 271: blk.30.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 272: blk.30.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 273: blk.30.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 274: blk.30.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 275: blk.30.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 276: blk.30.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 277: blk.31.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 278: blk.31.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 279: blk.31.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 280: blk.31.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 281: blk.31.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 282: blk.31.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 283: blk.32.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 284: blk.32.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 285: blk.32.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 286: blk.32.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 287: blk.32.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 288: blk.32.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 289: blk.33.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 290: blk.33.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 291: blk.33.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 292: blk.33.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 293: blk.33.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 294: blk.33.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 295: blk.34.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 296: blk.34.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 297: blk.34.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 298: blk.34.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 299: blk.34.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 300: blk.34.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 301: blk.35.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 302: blk.35.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 303: blk.35.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 304: blk.35.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 305: blk.35.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 306: blk.35.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 307: blk.36.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 308: blk.36.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 309: blk.36.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 310: blk.36.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 311: blk.36.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 312: blk.36.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 313: blk.37.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 314: blk.37.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 315: blk.37.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 316: blk.37.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 317: blk.37.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 318: blk.37.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 319: blk.38.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 320: blk.38.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 321: blk.38.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 322: blk.38.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 323: blk.38.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 324: blk.38.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 325: blk.39.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 326: blk.39.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 327: blk.39.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ] llama_model_loader: - tensor 328: blk.39.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ] llama_model_loader: - tensor 329: blk.39.attn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 330: blk.39.ffn_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 331: output_norm.weight f32 [ 5120, 1, 1, 1 ] llama_model_loader: - tensor 332: output.weight q6_K [ 5120, 125696, 1, 1 ] llama_model_loader: - tensor 333: blk.30.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 334: blk.30.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 335: blk.30.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 336: blk.31.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 337: blk.31.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 338: blk.31.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 339: blk.32.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 340: blk.32.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 341: blk.32.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 342: blk.33.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 343: blk.33.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 344: blk.33.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 345: blk.34.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 346: blk.34.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 347: blk.34.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 348: blk.35.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 349: blk.35.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 350: blk.35.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 351: blk.36.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 352: blk.36.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 353: blk.36.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 354: blk.37.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 355: blk.37.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 356: blk.37.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 357: blk.38.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 358: blk.38.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 359: blk.38.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 360: blk.39.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 361: blk.39.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - tensor 362: blk.39.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ] llama_model_loader: - kv 0: general.architecture str llama_model_loader: - kv 1: general.name str llama_model_loader: - kv 2: baichuan.tensor_data_layout str llama_model_loader: - kv 3: baichuan.context_length u32 llama_model_loader: - kv 4: baichuan.embedding_length u32 llama_model_loader: - kv 5: baichuan.block_count u32 llama_model_loader: - kv 6: baichuan.feed_forward_length u32 llama_model_loader: - kv 7: baichuan.rope.dimension_count u32 llama_model_loader: - kv 8: baichuan.attention.head_count u32 llama_model_loader: - kv 9: baichuan.attention.head_count_kv u32 llama_model_loader: - kv 10: baichuan.attention.layer_norm_rms_epsilon f32 llama_model_loader: - kv 11: tokenizer.ggml.model str llama_model_loader: - kv 12: tokenizer.ggml.tokens arr llama_model_loader: - kv 13: tokenizer.ggml.scores arr llama_model_loader: - kv 14: tokenizer.ggml.token_type arr llama_model_loader: - kv 15: tokenizer.ggml.bos_token_id u32 llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 llama_model_loader: - kv 17: tokenizer.ggml.padding_token_id u32 llama_model_loader: - kv 18: general.quantization_version u32 llama_model_loader: - kv 19: general.file_type u32 llama_model_loader: - type f32: 81 tensors llama_model_loader: - type q4_0: 281 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_print_meta: format = GGUF V2 (latest) llm_load_print_meta: arch = baichuan llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 125696 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 4096 llm_load_print_meta: n_ctx = 512 llm_load_print_meta: n_embd = 5120 llm_load_print_meta: n_head = 40 llm_load_print_meta: n_head_kv = 40 llm_load_print_meta: n_layer = 40 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: n_ff = 13696 llm_load_print_meta: freq_base = 10000.0 llm_load_print_meta: freq_scale = 1 llm_load_print_meta: model type = 13B llm_load_print_meta: model ftype = mostly Q4_0 llm_load_print_meta: model params = 13.90 B llm_load_print_meta: model size = 7.44 GiB (4.60 BPW) llm_load_print_meta: general.name = Baichuan2-13B-Chat llm_load_print_meta: BOS token = 1 '' llm_load_print_meta: EOS token = 2 '' llm_load_print_meta: UNK token = 0 '' llm_load_print_meta: PAD token = 0 '' llm_load_print_meta: LF token = 1099 '<0x0A>' llm_load_tensors: ggml ctx size = 0.12 MB llm_load_tensors: mem required = 7614.46 MB (+ 400.00 MB per state) ........................................................................................... llama_new_context_with_model: kv self size = 400.00 MB ggml_metal_init: allocating ggml_metal_init: found device: Apple M2 ggml_metal_init: picking default device: Apple M2 ggml_metal_init: loading '/Volumes/WD_sn770/LLAMA2/llamacpp/llama.cpp/ggml-metal.metal' ggml_metal_init: loaded kernel_add 0x119507430 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_add_row 0x119507c60 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul 0x119508180 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_row 0x1195087b0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_scale 0x119508cd0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_silu 0x1195091f0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_relu 0x119509710 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_gelu 0x119509c30 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_soft_max 0x13cf059a0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_soft_max_4 0x13ce07530 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_diag_mask_inf 0x13ce07b70 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_diag_mask_inf_8 0x13ce08340 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_f32 0x13ce089f0 | th_max = 896 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_f16 0x13ce090a0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q4_0 0x13ce09750 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q4_1 0x13ce09e00 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q8_0 0x13ce0a4b0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q2_K 0x13ce0ab60 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q3_K 0x13ce0b210 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q4_K 0x13ce0ba30 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q5_K 0x13ce0c0e0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q6_K 0x13ce0c790 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_rms_norm 0x13ce0ce50 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_norm 0x13ce0d680 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_f32_f32 0x13ce0dee0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_f16_f32 0x13ce0e740 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_f16_f32_1row 0x13ce0efa0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_f16_f32_l4 0x13ce0fa00 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_q4_0_f32 0x13ce10160 | th_max = 896 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_q4_1_f32 0x13ce10b20 | th_max = 896 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_q8_0_f32 0x13ce11280 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_q2_K_f32 0x13ce119e0 | th_max = 640 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_q3_K_f32 0x13ce11f00 | th_max = 576 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_q4_K_f32 0x13ce12660 | th_max = 576 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_q5_K_f32 0x13ce12dc0 | th_max = 640 | th_width = 32 ggml_metal_init: loaded kernel_mul_mat_q6_K_f32 0x13ce13520 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_f32_f32 0x13ce13d30 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_f16_f32 0x13ce14540 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q4_0_f32 0x13ce14d50 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q8_0_f32 0x13ce15560 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q4_1_f32 0x13ce15d70 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q2_K_f32 0x13ce16580 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q3_K_f32 0x13ce16d90 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q4_K_f32 0x13ce175a0 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q5_K_f32 0x11950a320 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q6_K_f32 0x11950ac50 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_rope 0x11950b3d0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_alibi_f32 0x11950bfa0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_cpy_f32_f16 0x11950c830 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_cpy_f32_f32 0x11950d0c0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_cpy_f16_f16 0x11950d950 | th_max = 1024 | th_width = 32 ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 10922.67 MB ggml_metal_init: maxTransferRate = built-in GPU llama_new_context_with_model: compute buffer total size = 256.97 MB llama_new_context_with_model: max tensor size = 503.47 MB ggml_metal_add_buffer: allocated 'data ' buffer, size = 7617.11 MB, ( 7617.61 / 10922.67) ggml_metal_add_buffer: allocated 'eval ' buffer, size = 1.48 MB, ( 7619.09 / 10922.67) ggml_metal_add_buffer: allocated 'kv ' buffer, size = 402.00 MB, ( 8021.09 / 10922.67) ggml_metal_add_buffer: allocated 'alloc ' buffer, size = 255.52 MB, ( 8276.61 / 10922.67) GGML_ASSERT: ggml-metal.m:1146: false && "only power-of-two n_head implemented" GGML_ASSERT: ggml-metal.m:1146: false && "only power-of-two n_head implemented" zsh: abort ./main -m ./zh-models/baichuan/Baichuan2-13B-Chat-ggml-model-q4_0.bin -p 0 按照上面的步骤，GPU推理报错，CPU下正常。大佬可以帮忙看下吗？MAC MINI M2 @jameswu2014

我也是这个问题，CPU正常，但是GPU不行。
CUDA error 9 at ggml-cuda.cu:6829: invalid configuration argument

from baichuan2.

chunzha1 commented on May 21, 2024 1

(alpaca_env) chunzhamini@chunzhamini Log start
main: warning: changing RoPE main: warning: scaling RoPE main: build = 1270 (c091cdf)
main: built with Apple clang main: seed = 1695699630
llama_model_loader: loaded llama_model_loader: - tensor 0: llama_model_loader: - tensor 1: llama_model_loader: - tensor 2: llama_model_loader: - tensor 3: llama_model_loader: - tensor 4: llama_model_loader: - tensor 5: llama_model_loader: - tensor 6: llama_model_loader: - tensor 7: llama_model_loader: - tensor 8: llama_model_loader: - tensor 9: llama_model_loader: - tensor 10: llama_model_loader: - tensor 11: llama_model_loader: - tensor 12: llama_model_loader: - tensor 13: llama_model_loader: - tensor 14: llama_model_loader: - tensor 15: llama_model_loader: - tensor 16: llama_model_loader: - tensor 17: llama_model_loader: - tensor 18: llama_model_loader: - tensor 19: llama_model_loader: - tensor 20: llama_model_loader: - tensor 21: llama_model_loader: - tensor 22: llama_model_loader: - tensor 23: llama_model_loader: - tensor 24: llama_model_loader: - tensor 25: llama_model_loader: - tensor 26: llama_model_loader: - tensor 27: llama_model_loader: - tensor 28: llama_model_loader: - tensor 29: llama_model_loader: - tensor 30: llama_model_loader: - tensor 31: llama_model_loader: - tensor 32: llama_model_loader: - tensor 33: llama_model_loader: - tensor 34: llama_model_loader: - tensor 35: llama_model_loader: - tensor 36: llama_model_loader: - tensor 37: llama_model_loader: - tensor 38: llama_model_loader: - tensor 39: llama_model_loader: - tensor 40: llama_model_loader: - tensor 41: llama_model_loader: - tensor 42: llama_model_loader: - tensor 43: llama_model_loader: - tensor 44: llama_model_loader: - tensor 45: llama_model_loader: - tensor 46: llama_model_loader: - tensor 47: llama_model_loader: - tensor 48: llama_model_loader: - tensor 49: llama_model_loader: - tensor 50: llama_model_loader: - tensor 51: llama_model_loader: - tensor 52: llama_model_loader: - tensor 53: llama_model_loader: - tensor 54: llama_model_loader: - tensor 55: llama_model_loader: - tensor 56: llama_model_loader: - tensor 57: llama_model_loader: - tensor 58: llama_model_loader: - tensor 59: llama_model_loader: - tensor 60: llama_model_loader: - tensor 61: llama_model_loader: - tensor 62: llama_model_loader: - tensor 63: llama_model_loader: - tensor 64: llama_model_loader: - tensor 65: llama_model_loader: - tensor 66: llama_model_loader: - tensor 67: llama_model_loader: - tensor 68: llama_model_loader: - tensor 69: llama_model_loader: - tensor 70: llama_model_loader: - tensor 71: llama_model_loader: - tensor 72: llama_model_loader: - tensor 73: llama_model_loader: - tensor 74: llama_model_loader: - tensor 75: llama_model_loader: - tensor 76: llama_model_loader: - tensor 77: llama_model_loader: - tensor 78: llama_model_loader: - tensor 79: llama_model_loader: - tensor 80: llama_model_loader: - tensor 81: llama_model_loader: - tensor 82: llama_model_loader: - tensor 83: llama_model_loader: - tensor 84: llama_model_loader: - tensor 85: llama_model_loader: - tensor 86: llama_model_loader: - tensor 87: llama_model_loader: - tensor 88: llama_model_loader: - tensor 89: llama_model_loader: - tensor 90: llama_model_loader: - tensor 91: llama_model_loader: - tensor 92: llama_model_loader: - tensor 93: llama_model_loader: - tensor 94: llama_model_loader: - tensor 95: llama_model_loader: - tensor 96: llama_model_loader: - tensor 97: llama_model_loader: - tensor 98: llama_model_loader: - tensor 99: llama_model_loader: - tensor 100: llama_model_loader: - tensor 101: llama_model_loader: - tensor 102: llama_model_loader: - tensor 103: llama_model_loader: - tensor 104: llama_model_loader: - tensor 105: llama_model_loader: - tensor 106: llama_model_loader: - tensor 107: llama_model_loader: - tensor 108: llama_model_loader: - tensor 109: llama_model_loader: - tensor 110: llama_model_loader: - tensor 111: llama_model_loader: - tensor 112: llama_model_loader: - tensor 113: llama_model_loader: - tensor 114: llama_model_loader: - tensor 115: llama_model_loader: - tensor 116: llama_model_loader: - tensor 117: llama_model_loader: - tensor 118: llama_model_loader: - tensor 119: llama_model_loader: - tensor 120: llama_model_loader: - tensor 121: llama_model_loader: - tensor 122: llama_model_loader: - tensor 123: llama_model_loader: - tensor 124: llama_model_loader: - tensor 125: llama_model_loader: - tensor 126: llama_model_loader: - tensor 127: llama_model_loader: - tensor 128: llama_model_loader: - tensor 129: llama_model_loader: - tensor 130: llama_model_loader: - tensor 131: llama_model_loader: - tensor 132: llama_model_loader: - tensor 133: llama_model_loader: - tensor 134: llama_model_loader: - tensor 135: llama_model_loader: - tensor 136: llama_model_loader: - tensor 137: llama_model_loader: - tensor 138: llama_model_loader: - tensor 139: llama_model_loader: - tensor 140: llama_model_loader: - tensor 141: llama_model_loader: - tensor 142: llama_model_loader: - tensor 143: llama_model_loader: - tensor 144: llama_model_loader: - tensor 145: llama_model_loader: - tensor 146: llama_model_loader: - tensor 147: llama_model_loader: - tensor 148: llama_model_loader: - tensor 149: llama_model_loader: - tensor 150: llama_model_loader: - tensor 151: llama_model_loader: - tensor 152: llama_model_loader: - tensor 153: llama_model_loader: - tensor 154: llama_model_loader: - tensor 155: llama_model_loader: - tensor 156: llama_model_loader: - tensor 157: llama_model_loader: - tensor 158: llama_model_loader: - tensor 159: llama_model_loader: - tensor 160: llama_model_loader: - tensor 161: llama_model_loader: - tensor 162: llama_model_loader: - tensor 163: llama_model_loader: - tensor 164: llama_model_loader: - tensor 165: llama_model_loader: - tensor 166: llama_model_loader: - tensor 167: llama_model_loader: - tensor 168: llama_model_loader: - tensor 169: llama_model_loader: - tensor 170: llama_model_loader: - tensor 171: llama_model_loader: - tensor 172: llama_model_loader: - tensor 173: llama_model_loader: - tensor 174: llama_model_loader: - tensor 175: llama_model_loader: - tensor 176: llama_model_loader: - tensor 177: llama_model_loader: - tensor 178: llama_model_loader: - tensor 179: llama_model_loader: - tensor 180: llama_model_loader: - tensor 181: llama_model_loader: - tensor 182: llama_model_loader: - tensor 183: llama_model_loader: - tensor 184: llama_model_loader: - tensor 185: llama_model_loader: - tensor 186: llama_model_loader: - tensor 187: llama_model_loader: - tensor 188: llama_model_loader: - tensor 189: llama_model_loader: - tensor 190: llama_model_loader: - tensor 191: llama_model_loader: - tensor 192: llama_model_loader: - tensor 193: llama_model_loader: - tensor 194: llama_model_loader: - tensor 195: llama_model_loader: - tensor 196: llama_model_loader: - tensor 197: llama_model_loader: - tensor 198: llama_model_loader: - tensor 199: llama_model_loader: - tensor 200: llama_model_loader: - tensor 201: llama_model_loader: - tensor 202: llama_model_loader: - tensor 203: llama_model_loader: - tensor 204: llama_model_loader: - tensor 205: llama_model_loader: - tensor 206: llama_model_loader: - tensor 207: llama_model_loader: - tensor 208: llama_model_loader: - tensor 209: llama_model_loader: - tensor 210: llama_model_loader: - tensor 211: llama_model_loader: - tensor 212: llama_model_loader: - tensor 213: llama_model_loader: - tensor 214: llama_model_loader: - tensor 215: llama_model_loader: - tensor 216: llama_model_loader: - tensor 217: llama_model_loader: - tensor 218: llama_model_loader: - tensor 219: llama_model_loader: - tensor 220: llama_model_loader: - tensor 221: llama_model_loader: - tensor 222: llama_model_loader: - tensor 223: llama_model_loader: - tensor 224: llama_model_loader: - tensor 225: llama_model_loader: - tensor 226: llama_model_loader: - tensor 227: llama_model_loader: - tensor 228: llama_model_loader: - tensor 229: llama_model_loader: - tensor 230: llama_model_loader: - tensor 231: llama_model_loader: - tensor 232: llama_model_loader: - tensor 233: llama_model_loader: - tensor 234: llama_model_loader: - tensor 235: llama_model_loader: - tensor 236: llama_model_loader: - tensor 237: llama_model_loader: - tensor 238: llama_model_loader: - tensor 239: llama_model_loader: - tensor 240: llama_model_loader: - tensor 241: llama_model_loader: - tensor 242: llama_model_loader: - tensor 243: llama_model_loader: - tensor 244: llama_model_loader: - tensor 245: llama_model_loader: - tensor 246: llama_model_loader: - tensor 247: llama_model_loader: - tensor 248: llama_model_loader: - tensor 249: llama_model_loader: - tensor 250: llama_model_loader: - tensor 251: llama_model_loader: - tensor 252: llama_model_loader: - tensor 253: llama_model_loader: - tensor 254: llama_model_loader: - tensor 255: llama_model_loader: - tensor 256: llama_model_loader: - tensor 257: llama_model_loader: - tensor 258: llama_model_loader: - tensor 259: llama_model_loader: - tensor 260: llama_model_loader: - tensor 261: llama_model_loader: - tensor 262: llama_model_loader: - tensor 263: llama_model_loader: - tensor 264: llama_model_loader: - tensor 265: llama_model_loader: - tensor 266: llama_model_loader: - tensor 267: llama_model_loader: - tensor 268: llama_model_loader: - tensor 269: llama_model_loader: - tensor 270: llama_model_loader: - tensor 271: llama_model_loader: - tensor 272: llama_model_loader: - tensor 273: llama_model_loader: - tensor 274: llama_model_loader: - tensor 275: llama_model_loader: - tensor 276: llama_model_loader: - tensor 277: llama_model_loader: - tensor 278: llama_model_loader: - tensor 279: llama_model_loader: - tensor 280: llama_model_loader: - tensor 281: llama_model_loader: - tensor 282: llama_model_loader: - tensor 283: llama_model_loader: - tensor 284: llama_model_loader: - tensor 285: llama_model_loader: - tensor 286: llama_model_loader: - tensor 287: llama_model_loader: - tensor 288: llama_model_loader: - tensor 289: llama_model_loader: - tensor 290: llama_model_loader: - tensor 291: llama_model_loader: - tensor 292: llama_model_loader: - tensor 293: llama_model_loader: - tensor 294: llama_model_loader: - tensor 295: llama_model_loader: - tensor 296: llama_model_loader: - tensor 297: llama_model_loader: - tensor 298: llama_model_loader: - tensor 299: llama_model_loader: - tensor 300: llama_model_loader: - tensor 301: llama_model_loader: - tensor 302: llama_model_loader: - tensor 303: llama_model_loader: - tensor 304: llama_model_loader: - tensor 305: llama_model_loader: - tensor 306: llama_model_loader: - tensor 307: llama_model_loader: - tensor 308: llama_model_loader: - tensor 309: llama_model_loader: - tensor 310: llama_model_loader: - tensor 311: llama_model_loader: - tensor 312: llama_model_loader: - tensor 313: llama_model_loader: - tensor 314: llama_model_loader: - tensor 315: llama_model_loader: - tensor 316: llama_model_loader: - tensor 317: llama_model_loader: - tensor 318: llama_model_loader: - tensor 319: llama_model_loader: - tensor 320: llama_model_loader: - tensor 321: llama_model_loader: - tensor 322: llama_model_loader: - tensor 323: llama_model_loader: - tensor 324: llama_model_loader: - tensor 325: llama_model_loader: - tensor 326: llama_model_loader: - tensor 327: llama_model_loader: - tensor 328: llama_model_loader: - tensor 329: llama_model_loader: - tensor 330: llama_model_loader: - tensor 331: llama_model_loader: - tensor 332: llama_model_loader: - tensor 333: llama_model_loader: - tensor 334: llama_model_loader: - tensor 335: llama_model_loader: - tensor 336: llama_model_loader: - tensor 337: llama_model_loader: - tensor 338: llama_model_loader: - tensor 339: llama_model_loader: - tensor 340: llama_model_loader: - tensor 341: llama_model_loader: - tensor 342: llama_model_loader: - tensor 343: llama_model_loader: - tensor 344: llama_model_loader: - tensor 345: llama_model_loader: - tensor 346: llama_model_loader: - tensor 347: llama_model_loader: - tensor 348: llama_model_loader: - tensor 349: llama_model_loader: - tensor 350: llama_model_loader: - tensor 351: llama_model_loader: - tensor 352: llama_model_loader: - tensor 353: llama_model_loader: - tensor 354: llama_model_loader: - tensor 355: llama_model_loader: - tensor 356: llama_model_loader: - tensor 357: llama_model_loader: - tensor 358: llama_model_loader: - tensor 359: llama_model_loader: - tensor 360: llama_model_loader: - tensor 361: llama_model_loader: - tensor 362: llama_model_loader: - kv 0: llama_model_loader: - kv 1: llama_model_loader: - kv 2: llama_model_loader: - kv 3: llama_model_loader: - kv 4: llama_model_loader: - kv 5: llama_model_loader: - kv 6: llama_model_loader: - kv 7: llama_model_loader: - kv 8: llama_model_loader: - kv 9: llama_model_loader: - kv llama_model_loader: - kv 11: llama_model_loader: - kv 12: llama_model_loader: - kv 13: llama_model_loader: - kv 14: llama_model_loader: - kv 15: llama_model_loader: - kv 16: llama_model_loader: - kv 17: llama_model_loader: - kv 18: llama_model_loader: - kv 19: llama_model_loader: - type f32: llama_model_loader: - type q4_0: llama_model_loader: - type q6_K: llm_load_print_meta: format llm_load_print_meta: arch llm_load_print_meta: vocab type llm_load_print_meta: n_vocab llm_load_print_meta: n_merges llm_load_print_meta: n_ctx_train llm_load_print_meta: n_ctx llm_load_print_meta: n_embd llm_load_print_meta: n_head llm_load_print_meta: n_head_kv llm_load_print_meta: n_layer llm_load_print_meta: n_rot llm_load_print_meta: n_gqa llm_load_print_meta: f_norm_eps llm_load_print_meta: f_norm_rms_eps llm_load_print_meta: n_ff llm_load_print_meta: freq_base llm_load_print_meta: freq_scale llm_load_print_meta: model type llm_load_print_meta: model ftype llm_load_print_meta: model params llm_load_print_meta: model size llm_load_print_meta: general.name llm_load_print_meta: BOS llm_load_print_meta: EOS llm_load_print_meta: UNK llm_load_print_meta: PAD llm_load_print_meta: LF token llm_load_tensors: ggml ctx size = llm_load_tensors: mem required .................................. llama_new_context_with_model: kv self size ggml_metal_init: allocating
ggml_metal_init: found device: ggml_metal_init: picking ggml_metal_init: loading ggml_metal_init: loaded kernel_add ggml_metal_init: loaded kernel_add_row ggml_metal_init: loaded kernel_mul ggml_metal_init: loaded kernel_mul_row ggml_metal_init: loaded kernel_scale ggml_metal_init: loaded kernel_silu ggml_metal_init: loaded kernel_relu ggml_metal_init: loaded kernel_gelu ggml_metal_init: loaded kernel_soft_max ggml_metal_init: loaded kernel_soft_max_4 ggml_metal_init: loaded kernel_diag_mask_inf ggml_metal_init: loaded kernel_diag_mask_inf_8 ggml_metal_init: loaded kernel_get_rows_f32 ggml_metal_init: loaded kernel_get_rows_f16 ggml_metal_init: loaded kernel_get_rows_q4_0 ggml_metal_init: loaded kernel_get_rows_q4_1 ggml_metal_init: loaded kernel_get_rows_q8_0 ggml_metal_init: loaded kernel_get_rows_q2_K ggml_metal_init: loaded kernel_get_rows_q3_K ggml_metal_init: loaded kernel_get_rows_q4_K ggml_metal_init: loaded kernel_get_rows_q5_K ggml_metal_init: loaded kernel_get_rows_q6_K ggml_metal_init: loaded kernel_rms_norm ggml_metal_init: loaded kernel_norm ggml_metal_init: loaded kernel_mul_mat_f32_f32 ggml_metal_init: loaded kernel_mul_mat_f16_f32 ggml_metal_init: loaded kernel_mul ggml_metal_init: loaded kernel_mul ggml_metal_init: loaded kernel_mul_mat_q4_0_f32 ggml_metal_init: loaded kernel_mul_mat_q4_1_f32 ggml_metal_init: loaded kernel_mul_mat_q8_0_f32 ggml_metal_init: loaded kernel_mul_mat_q2_K_f32 ggml_metal_init: loaded kernel_mul_mat_q3_K_f32 ggml_metal_init: loaded kernel_mul_mat_q4_K_f32 ggml_metal_init: loaded kernel_mul_mat_q5_K_f32 ggml_metal_init: loaded kernel_mul_mat_q6_K_f32 ggml_metal_init: loaded kernel_mul_mm_f32_f32 ggml_metal_init: loaded kernel_mul_mm_f16_f32 ggml_metal_init: loaded kernel_mul_mm_q4_0_f32 ggml_metal_init: loaded kernel_mul_mm_q8_0_f32 ggml_metal_init: loaded kernel_mul_mm_q4_1_f32 ggml_metal_init: loaded kernel_mul_mm_q2_K_f32 ggml_metal_init: loaded kernel_mul_mm_q3_K_f32 ggml_metal_init: loaded kernel_mul_mm_q4_K_f32 ggml_metal_init: loaded kernel_mul_mm_q5_K_f32 ggml_metal_init: loaded kernel_mul_mm_q6_K_f32 ggml_metal_init: loaded kernel_rope ggml_metal_init: loaded kernel_alibi_f32 ggml_metal_init: loaded kernel_cpy_f32_f16 ggml_metal_init: loaded kernel_cpy_f32_f32 ggml_metal_init: loaded kernel_cpy_f16_f16 ggml_metal_init: hasUnifiedMemory ggml_metal_init: recommendedMaxWorkingSetSize ggml_metal_init: maxTransferRate llama_new_context_with_model: llama_new_context_with_model: ggml_metal_add_buffer: allocated 'data ggml_metal_add_buffer: allocated 'eval ggml_metal_add_buffer: allocated 'kv ggml_metal_add_buffer: allocated 'alloc GGML_ASSERT: ggml-metal.m:1146: GGML_ASSERT: ggml-metal.m:1146: zsh: abort ./main -m 按照上面的步骤，GPU推理报 llama.cpp % ./main -m ./zh-models/baichuan/Baichuan2-13B-Chat-ggml-model-q4_0.bin -p '从前有一只小狐狸，他' --temp 0 -ngl 1
frequency base to 0 (default 10000.0)
frequency by 0 (default 1.0)
version 14.0.3 (clang-1403.0.22.14.1) for arm64-apple-darwin22.5.0
meta data with 20 key-value pairs and 363 tensors from ./zh-models/baichuan/Baichuan2-13B-Chat-ggml-model-q4_0.bin (version GGUF V2 (latest))
token_embd.weight q4_0 [ 5120, 125696, 1, 1 ]
blk.0.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.0.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.0.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.0.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.0.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.0.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.1.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.1.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.1.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.1.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.1.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.1.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.2.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.2.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.2.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.2.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.2.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.2.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.3.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.3.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.3.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.3.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.3.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.3.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.4.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.4.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.4.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.4.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.4.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.4.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.5.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.5.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.5.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.5.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.5.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.5.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.6.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.6.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.6.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.6.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.6.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.6.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.7.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.7.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.7.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.7.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.7.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.7.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.8.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.8.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.8.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.8.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.8.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.8.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.9.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.9.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.9.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.9.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.9.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.9.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.10.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.10.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.10.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.10.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.10.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.10.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.11.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.11.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.11.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.11.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.11.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.11.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.12.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.12.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.12.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.12.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.12.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.12.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.13.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.13.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.13.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.0.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.0.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.0.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.1.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.1.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.1.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.2.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.2.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.2.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.3.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.3.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.3.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.4.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.4.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.4.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.5.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.5.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.5.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.6.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.6.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.6.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.7.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.7.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.7.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.8.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.8.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.8.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.9.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.9.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.9.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.10.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.10.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.10.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.11.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.11.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.11.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.12.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.12.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.12.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.13.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.13.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.13.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.13.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.13.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.13.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.14.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.14.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.14.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.14.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.14.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.14.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.15.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.15.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.15.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.15.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.15.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.15.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.16.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.16.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.16.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.16.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.16.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.16.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.17.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.17.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.17.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.17.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.17.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.17.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.18.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.18.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.18.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.18.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.18.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.18.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.19.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.19.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.19.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.19.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.19.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.19.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.20.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.20.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.20.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.20.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.20.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.20.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.21.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.21.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.21.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.21.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.21.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.21.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.22.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.22.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.22.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.22.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.22.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.22.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.23.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.23.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.23.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.23.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.23.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.23.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.24.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.24.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.24.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.24.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.24.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.24.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.25.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.25.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.25.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.25.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.25.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.25.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.26.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.26.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.26.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.26.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.26.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.26.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.27.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.27.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.27.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.27.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.27.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.27.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.28.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.28.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.28.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.28.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.28.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.28.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.29.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.29.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.14.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.14.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.14.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.15.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.15.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.15.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.16.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.16.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.16.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.17.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.17.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.17.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.18.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.18.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.18.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.19.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.19.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.19.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.20.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.20.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.20.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.21.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.21.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.21.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.22.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.22.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.22.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.23.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.23.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.23.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.24.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.24.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.24.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.25.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.25.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.25.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.26.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.26.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.26.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.27.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.27.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.27.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.28.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.28.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.28.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.29.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.29.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.29.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.29.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.29.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.29.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.29.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.30.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.30.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.30.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.30.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.30.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.30.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.31.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.31.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.31.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.31.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.31.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.31.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.32.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.32.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.32.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.32.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.32.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.32.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.33.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.33.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.33.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.33.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.33.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.33.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.34.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.34.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.34.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.34.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.34.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.34.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.35.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.35.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.35.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.35.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.35.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.35.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.36.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.36.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.36.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.36.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.36.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.36.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.37.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.37.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.37.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.37.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.37.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.37.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.38.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.38.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.38.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.38.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.38.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.38.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.39.attn_output.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.39.ffn_gate.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.39.ffn_down.weight q4_0 [ 13696, 5120, 1, 1 ]
blk.39.ffn_up.weight q4_0 [ 5120, 13696, 1, 1 ]
blk.39.attn_norm.weight f32 [ 5120, 1, 1, 1 ]
blk.39.ffn_norm.weight f32 [ 5120, 1, 1, 1 ]
output_norm.weight f32 [ 5120, 1, 1, 1 ]
output.weight q6_K [ 5120, 125696, 1, 1 ]
blk.30.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.30.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.30.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.31.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.31.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.31.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.32.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.32.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.32.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.33.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.33.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.33.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.34.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.34.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.34.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.35.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.35.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.35.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.36.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.36.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.36.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.37.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.37.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.37.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.38.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.38.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.38.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.39.attn_q.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.39.attn_k.weight q4_0 [ 5120, 5120, 1, 1 ]
blk.39.attn_v.weight q4_0 [ 5120, 5120, 1, 1 ]
general.architecture str
general.name str
baichuan.tensor_data_layout str
baichuan.context_length u32
baichuan.embedding_length u32
baichuan.block_count u32
baichuan.feed_forward_length u32
baichuan.rope.dimension_count u32
baichuan.attention.head_count u32
baichuan.attention.head_count_kv u32
10: baichuan.attention.layer_norm_rms_epsilon f32
tokenizer.ggml.model str
tokenizer.ggml.tokens arr
tokenizer.ggml.scores arr
tokenizer.ggml.token_type arr
tokenizer.ggml.bos_token_id u32
tokenizer.ggml.eos_token_id u32
tokenizer.ggml.padding_token_id u32
general.quantization_version u32
general.file_type u32
81 tensors
281 tensors
1 tensors
= GGUF V2 (latest)
= baichuan
= SPM
= 125696
= 0
= 4096
= 512
= 5120
= 40
= 40
= 40
= 128
= 1
= 0.0e+00
= 1.0e-06
= 13696
= 10000.0
= 1
= 13B
= mostly Q4_0
= 13.90 B
= 7.44 GiB (4.60 BPW)
= Baichuan2-13B-Chat
token = 1 ''
token = 2 ''
token = 0 ''
token = 0 ''
= 1099 '<0x0A>'
0.12 MB
= 7614.46 MB (+ 400.00 MB per state)
.........................................................
= 400.00 MB
Apple M2
default device: Apple M2
'/Volumes/WD_sn770/LLAMA2/llamacpp/llama.cpp/ggml-metal.metal'
0x119507430 | th_max = 1024 | th_width = 32
0x119507c60 | th_max = 1024 | th_width = 32
0x119508180 | th_max = 1024 | th_width = 32
0x1195087b0 | th_max = 1024 | th_width = 32
0x119508cd0 | th_max = 1024 | th_width = 32
0x1195091f0 | th_max = 1024 | th_width = 32
0x119509710 | th_max = 1024 | th_width = 32
0x119509c30 | th_max = 1024 | th_width = 32
0x13cf059a0 | th_max = 1024 | th_width = 32
0x13ce07530 | th_max = 1024 | th_width = 32
0x13ce07b70 | th_max = 1024 | th_width = 32
0x13ce08340 | th_max = 1024 | th_width = 32
0x13ce089f0 | th_max = 896 | th_width = 32
0x13ce090a0 | th_max = 1024 | th_width = 32
0x13ce09750 | th_max = 1024 | th_width = 32
0x13ce09e00 | th_max = 1024 | th_width = 32
0x13ce0a4b0 | th_max = 1024 | th_width = 32
0x13ce0ab60 | th_max = 1024 | th_width = 32
0x13ce0b210 | th_max = 1024 | th_width = 32
0x13ce0ba30 | th_max = 1024 | th_width = 32
0x13ce0c0e0 | th_max = 1024 | th_width = 32
0x13ce0c790 | th_max = 1024 | th_width = 32
0x13ce0ce50 | th_max = 1024 | th_width = 32
0x13ce0d680 | th_max = 1024 | th_width = 32
0x13ce0dee0 | th_max = 1024 | th_width = 32
0x13ce0e740 | th_max = 1024 | th_width = 32
_mat_f16_f32_1row 0x13ce0efa0 | th_max = 1024 | th_width = 32
_mat_f16_f32_l4 0x13ce0fa00 | th_max = 1024 | th_width = 32
0x13ce10160 | th_max = 896 | th_width = 32
0x13ce10b20 | th_max = 896 | th_width = 32
0x13ce11280 | th_max = 1024 | th_width = 32
0x13ce119e0 | th_max = 640 | th_width = 32
0x13ce11f00 | th_max = 576 | th_width = 32
0x13ce12660 | th_max = 576 | th_width = 32
0x13ce12dc0 | th_max = 640 | th_width = 32
0x13ce13520 | th_max = 1024 | th_width = 32
0x13ce13d30 | th_max = 768 | th_width = 32
0x13ce14540 | th_max = 768 | th_width = 32
0x13ce14d50 | th_max = 768 | th_width = 32
0x13ce15560 | th_max = 768 | th_width = 32
0x13ce15d70 | th_max = 768 | th_width = 32
0x13ce16580 | th_max = 768 | th_width = 32
0x13ce16d90 | th_max = 768 | th_width = 32
0x13ce175a0 | th_max = 768 | th_width = 32
0x11950a320 | th_max = 768 | th_width = 32
0x11950ac50 | th_max = 768 | th_width = 32
0x11950b3d0 | th_max = 1024 | th_width = 32
0x11950bfa0 | th_max = 1024 | th_width = 32
0x11950c830 | th_max = 1024 | th_width = 32
0x11950d0c0 | th_max = 1024 | th_width = 32
0x11950d950 | th_max = 1024 | th_width = 32
= true
= 10922.67 MB
= built-in GPU
compute buffer total size = 256.97 MB
max tensor size = 503.47 MB
' buffer, size = 7617.11 MB, ( 7617.61 / 10922.67)
' buffer, size = 1.48 MB, ( 7619.09 / 10922.67)
' buffer, size = 402.00 MB, ( 8021.09 / 10922.67)
' buffer, size = 255.52 MB, ( 8276.61 / 10922.67)
false && "only power-of-two n_head implemented"
false && "only power-of-two n_head implemented"
./zh-models/baichuan/Baichuan2-13B-Chat-ggml-model-q4_0.bin -p 0
错，CPU下正常。大佬可以帮忙看下吗？MAC MINI M2 @jameswu2014

from baichuan2.

wzp123123 commented on May 21, 2024 1

同样的问题，有没有解决的思路

from baichuan2.

dlutsniper commented on May 21, 2024

最新版本的llama.cpp

install Python dependencies

python3 -m pip install -r requirements.txt

转换

python convert-baichuan-hf-to-gguf.py /Users/wy/Downloads/Baichuan2-13B-Chat --outfile Baichuan2-13B-Chat-ggml-model-f16.gguf
27.8GB

量化

./build/bin/quantize ./Baichuan2-13B-Chat-ggml-model-f16.gguf ./Baichuan2-13B-Chat-ggml-model-q4_0.gguf q4_0
7.99GB

运行

./build/bin/server -ngl 0 -m ./Baichuan2-13B-Chat-ggml-model-q4_0.gguf -c 4096 --embedding

macbook pro 15年款完美可以推理
有个小问题，不知道提示词模版是不是需要调整 @jameswu2014

from baichuan2.

dlutsniper commented on May 21, 2024

不知道为什么，llama.cpp server方式，推理结果没有命令行好，命令行还可以的哈，使用server的话，就比较奇怪了。
请问这个问题怎么回事儿啊 @jameswu2014

选择模型 Baichuan2-13B-Chat
最新版本llama.cpp

main推理命令行

main推理测试结果

server推理服务命令行

server推理服务测试结果

from baichuan2.

songkq commented on May 21, 2024

@jameswu2014 @dlutsniper quantize gguf model failed on RTX3090 with Driver Version: 525.105.17 CUDA Version: 12.0. Could you please give some advice for this issue?

./quantize /workspace/llama.cpp/models/Baichuan2-13B-Chat-ggml-model-f16.gguf /workspace/llama.cpp/models/Baichuan2-13B-Chat-ggml-model-Q8_0.gguf 7

CUDA error 804 at /llama.cpp/ggml-cuda.cu:5522: forward compatibility was attempted on non supported HW
current device: 0

from baichuan2.

zhangqiangauto commented on May 21, 2024

最新版本的llama.cpp

install Python dependencies

python3 -m pip install -r requirements.txt

最新开发版本gguf

cd llama.cpp/gguf-py pip install --editable .

转换

python convert-baichuan-hf-to-gguf.py /Users/wy/Downloads/Baichuan2-13B-Chat --outfile Baichuan2-13B-Chat-ggml-model-f16.gguf 27.8GB

量化

./build/bin/quantize ./Baichuan2-13B-Chat-ggml-model-f16.gguf ./Baichuan2-13B-Chat-ggml-model-q4_0.gguf q4_0 7.99GB

运行

./build/bin/server -ngl 0 -m ./Baichuan2-13B-Chat-ggml-model-q4_0.gguf -c 4096 --embedding

macbook pro 15年款完美可以推理有个小问题，不知道提示词模版是不是需要调整 @jameswu2014

我按照这个步骤量化了Baichuan2-7B 的 chat 版本，但问答输出大部分是英文，且质量很差，不知道什么原因。
附件是日志记录：debug.txt

from baichuan2.

zhangqiangauto commented on May 21, 2024

最新版本的llama.cpp

install Python dependencies

python3 -m pip install -r requirements.txt

最新开发版本gguf

cd llama.cpp/gguf-py pip install --editable .

转换

python convert-baichuan-hf-to-gguf.py /Users/wy/Downloads/Baichuan2-13B-Chat --outfile Baichuan2-13B-Chat-ggml-model-f16.gguf 27.8GB

量化

./build/bin/quantize ./Baichuan2-13B-Chat-ggml-model-f16.gguf ./Baichuan2-13B-Chat-ggml-model-q4_0.gguf q4_0 7.99GB

运行

./build/bin/server -ngl 0 -m ./Baichuan2-13B-Chat-ggml-model-q4_0.gguf -c 4096 --embedding
macbook pro 15年款完美可以推理有个小问题，不知道提示词模版是不是需要调整 @jameswu2014

我按照这个步骤量化了Baichuan2-7B 的 chat 版本，但问答输出大部分是英文，且质量很差，不知道什么原因。附件是日志记录：debug.txt

问题解决了。需要按照@jameswu2014的步骤，将 Baichuan2转 Baichuan1. 所以当前版本的 llama.cpp 还不能直接支持 Baichuan2 模型的 convert

from baichuan2.

aisensiy commented on May 21, 2024

llama.cpp我提了一个PR：ggerganov/llama.cpp#3009 你先按照Baichuan2的Readme里的Baichuan2->Baichuan1 的lm_head转换修改一下模型，就可以用上面链接里的修改。

This works for Q8_0 Q5_0 Q4_0 but failed for others with such error message:

$ ./quantize /models/baichuan2-13b-chat.gguf /models/baichuan2-13b-chat-Q4_K_M.gguf Q4_K

...
llama_model_loader: - type  f32:   81 tensors
llama_model_loader: - type  f16:  282 tensors
llama_model_quantize_internal: meta size = 2883232 bytes
[   1/ 363]                    token_embd.weight - [ 5120, 125696,     1,     1], type =    f16, quantizing to q4_K .. size =  1227.50 MB ->   345.23 MB | hist: 
[   2/ 363]             blk.0.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_K .. size =    50.00 MB ->    14.06 MB | hist: 
[   3/ 363]                blk.0.ffn_gate.weight - [ 5120, 13696,     1,     1], type =    f16, quantizing to q4_K .. size =   133.75 MB ->    37.62 MB | hist: 
[   4/ 363]                blk.0.ffn_down.weight - [13696,  5120,     1,     1], type =    f16, 

get_k_quant_type : tensor cols 13696 x 5120 are not divisible by 256, required for k-quants
llama_model_quantize: failed to quantize: Unsupported tensor size encountered

main: failed to quantize model from '/output/baichuan2-13b-chat.gguf'

from baichuan2.

wzp123123 commented on May 21, 2024

ggerganov/llama.cpp#3740

from baichuan2.

chunzha1 commented on May 21, 2024

好像和我的问题不太一样，请问你是什么设备？

from baichuan2.

guoqiangqi commented on May 21, 2024

使用最新版本llama.cpp 量化后server推理结果不准确，量化前未将baichuan2-13b-chat转为baichuan1模式，是这个导致的吗？

from baichuan2.

VJJJJJJ1 commented on May 21, 2024

llama.cpp我提了一个PR：ggerganov/llama.cpp#3009 你先按照Baichuan2的Readme里的Baichuan2->Baichuan1 的lm_head转换修改一下模型，就可以用上面链接里的修改。

请问微调后的baichuan2也可以用这个方法来加速吗？

from baichuan2.

类似 llama.cpp的加速？ about baichuan2 HOT 15 OPEN

Comments (15)

install Python dependencies

最新开发版本gguf

转换

量化

运行

install Python dependencies

最新开发版本gguf

转换

量化

运行

install Python dependencies

最新开发版本gguf

转换

量化

运行

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent