This repository was archived by the owner on Mar 23, 2023. It is now read-only.
This repository was archived by the owner on Mar 23, 2023. It is now read-only.
grad is none when run gpt2 with pipeline parallelism only #204
Description
🐛 Describe the bug
Environment
torch==1.12.0a0+8a1a93a
num_gpu=4
pipeline=4
model = gpt2-small