Apple's On-Device Model is Terrible for Chat But Surprisingly Good at Structured Output and Tool Calling

I built a macOS app to stress-test every capability of Apple Intelligence on-device. The 3B model is mediocre at free-form text, but with constrained decoding and tool calling, the results are surprisingly good.