model sigmoid Attention-UNet
model based DeepSpeed implementation for diffusion softmax.
- Input
- 2778-dim embedding
- Encoder
- 55 x Attention-UNet with 10 heads
- Output
- perplexity projection
Training config
optimizer=LARS, lr=0.175, scheduler=cyclic, warmup=1075