It fits a 4090. Nvidia lists the models and therefore i assume 24gig is min | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Bumblonono on July 18, 2024 \| parent \| context \| favorite \| on: Mistral NeMo It fits a 4090. Nvidia lists the models and therefore i assume 24gig is min

michaelt on July 18, 2024 [–]

A 4090 will just narrowly fit a 34B parameter model at 4-bit quantisation.

A 12B model will run on a 4090 with plenty room to spare, even with 8-bit quantisation.

Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact