As a friendly heads-up to the people interested - you can buy the board inside this off Ebay for like 300-400 dollars cheaper. "Jetson AGX Orin 64GB" specifically, then you wouldn't have to deal with whatever SaaS middleware that the Truffle ships with.
The 22 tokens per second claim for Mixtral conveniently fails to mention what type of quantization is going on with that benchmark.
Looks cute!
> NVIDIA Orin Module
So it has some NVIDIA chip inside it looks like?
https://www.nvidia.com/en-us/autonomous-machines/embedded-sy...
The 64GB Orin module is sold at about $2K on Amazon. https://www.amazon.com/NVIDIA-Jetson-Orin-64GB-Developer/dp/...
im not 100% sure yet what this is for. is this basically stand in for an always on laptop that is running mixtral, that has an api endpoint? and effectively no different than self hosting mixtral on some cloud somewhere?
More details here: https://twitter.com/iamgingertrash/status/176759390225142176...
I want a DIY guide that basically spells out from hardware purchases -> usably running models. I haven't seen one yet.
can it be used for whisper and asr as well? I would buy it if its cheaper.
The documentation says you start with:
But the Truffle formula currently on Homebrew - https://formulae.brew.sh/formula/truffle - is for something else, its for an Ethereum testing environment of some sort: https://archive.trufflesuite.com/