Show HN: How to guide on training Llama-405B using PyTorch distributed APIs

by lambda-researchon 10/15/2024, 1:51:36 PM with 3 comments
  • by lostmsuon 10/16/2024, 9:40:26 PM

    The guide does not say how efficient this run was in terms of GPU utilization (tops/theoretical max tops).

  • by lambda-researchon 10/15/2024, 1:57:03 PM

    Let me know if there are any questions or suggestions!

    Feel free to open issue on github, and contributions are welcome also