fix: unable to find output_dir in multi-GPU during resume_from_checkp…

…oint check (#352) * fix: output_dir doesn't exist during resume_from_checkpoint Signed-off-by: Abhishek <[email protected]> * fix: fmt Signed-off-by: Abhishek <[email protected]> --------- Signed-off-by: Abhishek <[email protected]> Signed-off-by: Anh Uong <[email protected]>
foundation-model-stack · Sep 26, 2024 · 0c6a062 · 0c6a062
1 parent 1350f8a
commit 0c6a062
Showing 1 changed file with 2 additions and 0 deletions.
diff --git a/build/accelerate_launch.py b/build/accelerate_launch.py
@@ -98,6 +98,8 @@ def main():
     #
     ##########
     output_dir = job_config.get("output_dir")
+    if not os.path.exists(output_dir):
+        os.makedirs(output_dir)
     try:
         # checkpoints outputted to tempdir, only final checkpoint copied to output dir
         launch_command(args)