Skip to content

Releases: foundation-model-stack/fms-hf-tuning

v2.0.1

01 Oct 16:07
9b8245e
Compare
Choose a tag to compare

New major features:

  1. Support for LoRA for the following model architectures - llama3, llama3.1, granite (GPTBigCode and LlamaForCausalLM), mistral, mixtral, and allam
  2. Support for QLora for the following model architectures - llama3, granite (GPTBigCode and LlamaForCausalLM), mistral, mixtral
  3. Addition of post-processing function to format tuned adapters as required by vLLM for inference. Refer to README on how to run as a script. When tuning on image, post-processing can be enabled using the flag lora_post_process_for_vllm. See build README for details on how to set this flag.
  4. Enablement of new flags for throughput improvements: padding_free to process multiple examples without adding padding tokens, multipack for multi-GPU training to balance the number of tokens processed on each device, and fast_kernels for optimized tuning with fused operations and triton kernels. See README for details on how to set these flags and use cases.

Dependency upgrades:

  1. Upgraded transformers to version 4.44.2 needed for tuning of all models
  2. Upgraded accelerate to version 0.33 needed for tuning of all models. Version 0.34.0 has a bug for FSDP.

API /interface changes:

  1. train() API now returns a tuple of trainer instance and additional metadata as a dict

Additional features and fixes

  1. Support of resume tuning from the existing checkpoint. Refer to README on how to use it as a flag. Flag resume_training defaults to True.
  2. Addition of default pad token in tokenizer when EOS and PAD tokens are equal to improve training quality.
  3. JSON compatability for input datasets. See docs for details on data formats.
  4. Fix to not resize embedding layer by default, embedding layer can continue to be resized as needed using flag embedding_size_multiple_of.

Full List of what's Changed

New Contributors

Full Changelog: v1.2.2...v2.0.0

v2.0.0

30 Sep 21:03
3b150ab
Compare
Choose a tag to compare

This version has old dependency and users should move to v2.0.1 instead

v2.0.0-rc.2

27 Sep 23:08
a37f074
Compare
Choose a tag to compare
v2.0.0-rc.2 Pre-release
Pre-release

What's Changed

  • fix: check for wte.weight along with embed_tokens.weight by @willmj in #356

Full Changelog: v2.0.0-rc.1...v2.0.0-rc.2

v2.0.0-rc.1

27 Sep 17:17
0c6a062
Compare
Choose a tag to compare
v2.0.0-rc.1 Pre-release
Pre-release

What's Changed

New Contributors

Full Changelog: v1.2.1...v2.0.0-rc.1

v1.2.2

03 Sep 21:48
16543ee
Compare
Choose a tag to compare

What's Changed

  • deps: Add protobuf to support ALLaM models by @willmj in #328
  • deps: set previous versions for accelerate and trl for patch release by @willmj in #329

Full Changelog: v1.2.1...v1.2.2

v1.2.1

19 Aug 17:20
a6d093e
Compare
Choose a tag to compare

What's Changed

Full Changelog: v1.2.0...v1.2.1

v1.2.1-rc.1

16 Aug 16:00
a6d093e
Compare
Choose a tag to compare
v1.2.1-rc.1 Pre-release
Pre-release

What's Changed

Full Changelog: v1.2.0...v1.2.1-rc.1

v1.2.0

14 Aug 22:38
2d1c17c
Compare
Choose a tag to compare

Dependency Updates

  1. Update packaging requirement from less than 23.2 to a max of 24 to less than 23.2 to max of 25

API/Interface Changes

  1. Add optional save_model_dir where final checkpoint is saved. See https://github.com/foundation-model-stack/fms-hf-tuning/blob/main/README.md#saving-checkpoints-while-training

Full List of What's Changed

New Contributors

Full Changelog: v1.1.0...v1.2.0

v1.2.0-rc.1

14 Aug 13:34
78909af
Compare
Choose a tag to compare
v1.2.0-rc.1 Pre-release
Pre-release

What's Changed

New Contributors

Full Changelog: v1.1.0-rc.1...v1.2.0-rc.1

v1.1.0

01 Aug 00:45
ab3b331
Compare
Choose a tag to compare

What's Changed

Full Changelog: v1.0.0...v1.1.0