Recently fine-tuned, quantized and deployed Phi-2 model with DPO technique and did 4 bit quantization using bitsandbytes and then deployed on Inferless.com, got around ~7 seconds of cold-start and ~21.34 tokens/second.
Recently fine-tuned, quantized and deployed Phi-2 model with DPO technique and did 4 bit quantization using bitsandbytes and then deployed on Inferless.com, got around ~7 seconds of cold-start and ~21.34 tokens/second.