I couldn't get the mps python version to run, it's insane how much setup it requires... I don't think the Gerganov C++ version uses CoreML or Neural Engine.
I previously tried to play with ANE based off prior reverse engineering work [0] but couldn't get it to work nicely. It's actually beyond me how Gerganov's version performs so well -- the output quality of the non-quantised version running on A100 (AWS) isn't noticably better than the one I'm getting.
I previously tried to play with ANE based off prior reverse engineering work [0] but couldn't get it to work nicely. It's actually beyond me how Gerganov's version performs so well -- the output quality of the non-quantised version running on A100 (AWS) isn't noticably better than the one I'm getting.
[0] https://i.blackhat.com/asia-21/Friday-Handouts/as21-Wu-Apple...