AI Card Bao AR02.AICARDBAO.COM

Industry News

Inkling 2 Mini ran to 210 tokens per second on the 96GB M5 Ultra, and the 48GB MacBook Pro also had 125

EXO Labs and Thinking Machines worked together to optimize the local inference of Inkling 2 Mini on Apple chips and released two sets of measured speeds.

M5 Ultra Mac Studio with 96GB RAM: decoding speed 210 tokens per second.

M5 Max MacBook Pro with 48GB RAM: 125 tokens per second.

Compare the physical sensations. The output speed of mainstream cloud models usually ranges from tens to more than a hundred tokens per second. The number 210 means that a machine placed on the table can pronounce words faster than most cloud interfaces. The 125 on the laptop has also reached the point where there is no need to wait at all.

In addition to speed, they compared it to Claude Sonnet 5.5 on a specific task: building a native weather application from scratch. The result is that the Inkling 2 Mini does it 1.8x faster and costs 11x less at API prices.

This set of comparisons should be viewed with a discount. There is only one task, and the topic is chosen by the publisher. It is not difficult to make a weather application. Small models do not suffer in such clearly defined tasks. What it can illustrate is: for this level of difficulty, a model that can run locally is enough, and it is faster and cheaper. It doesn't tell who is better at complex engineering tasks.

The more realistic threshold lies in the hardware, and these two sets of figures are currently only reported by the publisher. In public independent tests, the M5 Ultra with the same 96GB was 83.4 tokens per second in the 4k context, and dropped to 56.2 when the context was pulled to 32k; the 48GB M5 Pro was only around 24. Looking at the actual measurements of several machines together, the difference mainly comes from chip grade and quantification accuracy.

For developers who already have such machines, the scenarios that can be tried are clear: code completion, small application construction, and internal projects where it is inconvenient to transfer data. It can be used even if the network is disconnected, and the token fee is not included. These are two things that the cloud model cannot provide.

Related