-
Notifications
You must be signed in to change notification settings - Fork 589
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sl dr update docker base image #8614
Closed
Closed
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
d8fe231
Update the GATK base image to a newer LTS ubuntu release
droazen 50cfd0b
Incorporate base image changes from Sam Lee's PR, fix md5 check, othe…
droazen 4a4b5c6
pip
droazen 59762d5
Back out update to newer conda, try 20.04 instead of 22.04
droazen 87d6706
Back to 22.04
droazen ccaea7c
Add an h5diff epsilon + reporting of differences to ScoreVariantAnnot…
droazen 2187cf2
Capture h5diff output in exception messages
droazen 5576927
Remove epsilon temporarily
droazen 636d139
restore epsilon, add debugging statement
droazen 5934691
Remove debugging, remove obsolete runSystemCommand() method, final re…
droazen da06b4a
conda 23.10.0 + conda-forge pip
samuelklee File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,12 +1,16 @@ | ||
# Using OpenJDK 17 | ||
# This Dockerfile does not require any files that are in the GATK4 repo. | ||
FROM ubuntu:18.04 | ||
FROM ubuntu:22.04 | ||
|
||
# Avoid interactive prompts during apt installs/upgrades | ||
ENV DEBIAN_FRONTEND noninteractive | ||
|
||
#### Basic image utilities | ||
RUN apt-get update && \ | ||
apt-get upgrade -y && \ | ||
apt-get install -y --no-install-recommends \ | ||
RUN apt update && \ | ||
apt full-upgrade -y && \ | ||
apt install -y --no-install-recommends \ | ||
python3 \ | ||
python3-pip \ | ||
wget \ | ||
curl \ | ||
bc \ | ||
|
@@ -15,14 +19,17 @@ RUN apt-get update && \ | |
less \ | ||
bedtools \ | ||
samtools \ | ||
bcftools \ | ||
tabix \ | ||
git \ | ||
gpg-agent \ | ||
build-essential \ | ||
openjdk-17-jdk \ | ||
vim \ | ||
software-properties-common && \ | ||
apt-get -y clean && \ | ||
apt-get -y autoclean && \ | ||
apt-get -y autoremove && \ | ||
apt -y clean && \ | ||
apt -y autoclean && \ | ||
apt -y autoremove && \ | ||
rm -rf /var/lib/apt/lists/* | ||
|
||
RUN java -version | ||
|
@@ -31,11 +38,11 @@ RUN java -version | |
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" \ | ||
| tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg \ | ||
| apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - && \ | ||
apt-get update -y && \ | ||
apt-get install -y --no-install-recommends google-cloud-cli && \ | ||
apt-get -y clean && \ | ||
apt-get -y autoclean && \ | ||
apt-get -y autoremove && \ | ||
apt update -y && \ | ||
apt install -y --no-install-recommends google-cloud-cli && \ | ||
apt -y clean && \ | ||
apt -y autoclean && \ | ||
apt -y autoremove && \ | ||
rm -rf /var/lib/apt/lists/* && \ | ||
# Remove the anthos cli tool and related files since they are very large and we / anyone using the docker are unlikely to use them | ||
# Remove the bundled python because we have python installed separately | ||
|
@@ -55,16 +62,16 @@ ENV JAVA_LIBRARY_PATH /usr/lib/jni | |
|
||
# Install miniconda | ||
ENV DOWNLOAD_DIR /downloads | ||
ENV CONDA_URL https://repo.anaconda.com/miniconda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh | ||
ENV CONDA_MD5 = "32d73e1bc33fda089d7cd9ef4c1be542616bd8e437d1f77afeeaf7afdb019787" | ||
ENV CONDA_URL https://repo.anaconda.com/miniconda/Miniconda3-py310_23.10.0-1-Linux-x86_64.sh | ||
ENV CONDA_SHA256 "c7a34df472feb69805b64df6e8db58363c5ccab41cd3b40b07e3e6dfb924359a" | ||
ENV CONDA_PATH /opt/miniconda | ||
ENV PATH $CONDA_PATH/bin:$PATH | ||
RUN mkdir $DOWNLOAD_DIR && \ | ||
wget -nv -O $DOWNLOAD_DIR/miniconda.sh $CONDA_URL && \ | ||
test "`md5sum $DOWNLOAD_DIR/miniconda.sh | awk -v FS=' ' '{print $1}'` = $CONDA_MD5" && \ | ||
test "$(sha256sum $DOWNLOAD_DIR/miniconda.sh | awk -v FS=' ' -v ORS='' '{print $1}')" = "$CONDA_SHA256" && \ | ||
bash $DOWNLOAD_DIR/miniconda.sh -p $CONDA_PATH -b && \ | ||
rm $DOWNLOAD_DIR/miniconda.sh && \ | ||
conda clean -afy && \ | ||
find /opt/miniconda/ -follow -type f -name '*.a' -delete && \ | ||
find /opt/miniconda/ -follow -type f -name '*.pyc' -delete && \ | ||
rm -rf /root/.cache/pip | ||
conda config --set auto_update_conda false && \ | ||
conda config --set solver libmamba && \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is not strictly needed, since libmamba is the default solver in newer conda versions, but it might be worth hinting at the new use of libmamba here. |
||
rm -rf /root/.cache/pip |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
#!/bin/bash | ||
|
||
if [ $# -ne 1 ]; then | ||
echo "Usage: $0 docker_image_version" | ||
exit 1 | ||
fi | ||
|
||
IMAGE_VERSION=$1 | ||
IMAGE_NAME="us.gcr.io/broad-dsde-methods/gatk-base-image-staging-area" | ||
DOCKER_IMAGE_TAG="${IMAGE_NAME}:${IMAGE_VERSION}" | ||
|
||
gcloud builds submit --tag ${DOCKER_IMAGE_TAG} --timeout=24h --machine-type n1_highcpu_32 | ||
|
||
if [ $? -ne 0 ]; then | ||
echo "gcloud builds submit failed" | ||
exit 1 | ||
fi | ||
|
||
echo "Successfully published image to staging area at ${DOCKER_IMAGE_TAG}" | ||
exit 0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will hopefully ensure that devs experimenting with updates to the environment within the Docker don't inadvertently update to a different conda version, which might otherwise introduce discrepancies.