Skip to content

Commit

Permalink
fix(smstudio): Swap wget->curl in copy_s3_content
Browse files Browse the repository at this point in the history
Recently I've been observing wget fails in SMStudio as per:
aws/sagemaker-distribution#435 - so to get
fmbench debugged successfully in this environment I had to swap out
wget for curl calls in the `copy_s3_content` script.
  • Loading branch information
athewsey committed Jun 11, 2024
1 parent d8d6f19 commit dfaf619
Showing 1 changed file with 2 additions and 4 deletions.
6 changes: 2 additions & 4 deletions copy_s3_content.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,12 @@ mkdir -p $FMBENCH_READ_DIR/tokenizer
mkdir -p $FMBENCH_READ_DIR/llama2_tokenizer
mkdir -p $FMBENCH_READ_DIR/llama3_tokenizer
mkdir -p $FMBENCH_READ_DIR/mistral_tokenizer
wget https://${BUCKET}.s3.amazonaws.com/artifacts/ML-FMBT/manifest.txt -P ${FMBENCH_READ_DIR}/
curl --output-dir ${FMBENCH_READ_DIR}/ -O https://${BUCKET}.s3.amazonaws.com/artifacts/ML-FMBT/manifest.txt

# copy each file of the public content for FMBench
for i in `cat ${FMBENCH_READ_DIR}/manifest.txt`
do
dir_path=`dirname $i`
mkdir -p ${FMBENCH_READ_DIR}/$dir_path
wget https://${BUCKET}.s3.amazonaws.com/artifacts/ML-FMBT/$i -P ${FMBENCH_READ_DIR}/$dir_path
curl --output-dir ${FMBENCH_READ_DIR}/$dir_path -O https://${BUCKET}.s3.amazonaws.com/artifacts/ML-FMBT/$i
done


0 comments on commit dfaf619

Please sign in to comment.