ARK v0.5.0

• Integrate with MSCCL++
• Removed dependency on gpudma
• Add AMD CDNA3 architecture support
• Support communication for AMD GPUs
• Optimize OpGraph scheduling
• Add a multi-GPU Llama2 example

See details from #168.

ARK v0.4.1

What's Changed

• Fix graph optimization by @chhwang in #171
• v0.4.1 by @chhwang in #172

Full Changelog: v0.4.0...v0.4.1

ARK v0.4.0

• Support AMD GPUs (CDNA2, single-GPU only)
• Add high-performance AllReduce & AllGather algorithms with MSLL
• Fix major bugs in the scheduler

See details from #137.

ARK v0.3.0

• Enable heuristic model graph optimization
• Revise Python interfaces
• Add more operators & support mixed-precision models & support bfloat16
• Add a Llama2-7B example
• Fix connection setup bugs for large & distributed models
• Fix correctness bugs from a few operators
• Minor scheduler improvements

See details from #113.

ARK v0.2.1

v0.2.1 (#116)

* Fix a header installation bug
* Check the Python version during build
* Resolve compilation warnings

ARK v0.2.0

Timeline

Released Date: Sep. 5th, 2023

Work Items

Model

1. • Interface: expose the underlying buffer info to Tensor (#79)

Communication Stack

1. • Interface: hide GpuCommSw implementation from the interface (#81)
2. • Interface: extend the current interface (#104)

Operators Support

1. • Operator: add more operators (#62)
2. • Operator: upgrade CUTLASS (#105)

Python

1. • Interface: #96

Examples

1. • Example: parallel matmul example (#64)

Bug Fix

• #65
• #66
• #75
• #77
• #90
• #94
• #98
• #97
• #99
• #100
• #103
• #101
• #35

Documents

1. • Docs: update documents (#76, #78, #87)

CI

1. • Code Coverage: add code coverage (#110)
2. • Unit Tests: add a unit test pipeline (#88)
3. • Unit Tests: #91

ARK v0.1.0

Features

• Scheduler
  • The default scheduler
• Communication
  • A simple software communication stack
• Operators
  • Tensor
  • Reshape
  • Identity
  • Sharding
  • ReduceSum
  • ReduceMean
  • ReduceMax
  • Layernorm
  • Softmax
  • Transpose
  • Matmul
  • Im2col
  • Scale
  • Relu
  • Gelu
  • Add
  • Mul
  • Send
  • SendDone
  • Recv
  • SendMM
  • RecvMM
  • AllReduce
  • AllGather
• Examples
  • Tutorials
  • A simple FFN training
  • Transformer inference
  • Megatron inference

New Contributors

• @wusar made their first contribution in #1

Full Changelog: https://github.com/microsoft/ark/commits/v0.1.0