From 99fb03ff6d211831961aa817e771f353893c91de Mon Sep 17 00:00:00 2001 From: Jorge Date: Mon, 22 Aug 2022 12:18:28 +0200 Subject: [PATCH] Verify cuda buildchain in smoke pipeline (#502) * Fix keyword "Wait Until All Builds Are Complete" Previously the keyword "Wait Until All Builds Are Complete" was iterating the list of builds, but if the build wasn't already created it didn't wait for it to be in state Complete To fix this, we use the buildconfigs instead Signed-off-by: Jorge Garcia Oncins * Add "Verify That CUDA Build Chain Succeeds" to smoke Previously this test was called "Verify Pytorch And Tensorflow Can Be Spawned" but that test doesn't exist in Polarion anymore. Same for polarion id's ODS-480 and ODS-481 Signed-off-by: Jorge Garcia Oncins * Fix "Wait Until All Builds Are Complete" when build doesn't exist Signed-off-by: Jorge Garcia Oncins * Add tag ODS-481 to "Verify That CUDA Build Chain Succeeds" Signed-off-by: Jorge Garcia Oncins Signed-off-by: Jorge Garcia Oncins --- .../Page/OCPDashboard/Builds/Builds.robot | 56 +++++++++++++------ .../100__installation/102__post_install.robot | 14 +++-- 2 files changed, 48 insertions(+), 22 deletions(-) diff --git a/tests/Resources/Page/OCPDashboard/Builds/Builds.robot b/tests/Resources/Page/OCPDashboard/Builds/Builds.robot index 3c2d0722f..9e8e64065 100644 --- a/tests/Resources/Page/OCPDashboard/Builds/Builds.robot +++ b/tests/Resources/Page/OCPDashboard/Builds/Builds.robot @@ -4,6 +4,7 @@ Library OpenShiftLibrary Resource ../../OCPDashboard/Page.robot Resource ../../ODH/ODHDashboard/ODHDashboard.robot + *** Keywords *** Get Build Status [Arguments] ${namespace} ${build_search_term} @@ -43,7 +44,10 @@ Check Image Build Status Search Last Build [Documentation] Returns latest(sorted by creation time) build which match the ${build_name_includes} [Arguments] ${namespace} ${build_name_includes} - ${build} = Run oc get builds --sort-by=.metadata.creationTimestamp -n ${namespace} | grep ${build_name_includes} | awk '{print $1;}' + ${build} = Run oc get builds --sort-by=.metadata.creationTimestamp -n ${namespace} | grep ${build_name_includes} | tail -n 1 | awk '{print $1;}' + IF "${build}" == "${EMPTY}" + Fail msg=Could not find any build including ${build_name_includes} in ${namespace} namespace + END @{builds} = Split String ${build} \n [Return] ${builds}[-1] @@ -68,10 +72,28 @@ Get Build Status From Oc [Return] ${status} Build Status Should Be - [Documentation] Get status and check with ${expected_status} + [Documentation] Gets build status and fails if not equal to ${expected_status} [Arguments] ${namespace} ${build_name} ${expected_status}=Complete ${status} = Get Build Status From Oc namespace=${namespace} build_name=${build_name} - Should Be Equal ${status} ${expected_status} + IF "${status}" != "${expected_status}" + Fail msg=Unexpected Build status for ${build_name} (expected_status: ${expected_status}, status: ${status}) + END + +Build Status Should Not Be + [Documentation] Gets build status and fails if equal to ${unexpected_status} + [Arguments] ${namespace} ${build_name} ${unexpected_status}=Error + ${status} = Get Build Status From Oc namespace=${namespace} build_name=${build_name} + IF "${status}" == "${unexpected_status}" + Fail msg=Build ${build_name} shoud not have build status ${unexpected_status} + END + +Wait Until Build Exists + [Documentation] Waits until a build exist with name including ${build_name_includes} or timeout exceeded + ... Returns build name + [Arguments] ${namespace} ${build_name_includes} ${timeout}=20 min + ${build_name} = Wait Until Keyword Succeeds ${timeout} 1 min + ... Search Last Build namespace=${namespace} build_name_includes=${build_name_includes} + [Return] ${build_name} Wait Until Build Status Is [Documentation] Check status build with ${expected_status} for every min until is succeed or timeout @@ -80,20 +102,22 @@ Wait Until Build Status Is ... Build Status Should Be ${namespace} ${build_name} ${expected_status} Wait Until All Builds Are Complete - [Documentation] Obtains the list of builds in ${namespace} and, for each of - ... them, fails if state is Failed or Error. If not, waits until state - ... is Complete or ${build_timeout} is reached + [Documentation] Obtains the list of buildsconfigs in ${namespace} and, for each of + ... them, search the last build with similar name and fails if state is Failed or Error. + ... If not, waits until state is Complete or ${build_timeout} is reached [Arguments] ${namespace} ${build_timeout}=20 min - ${builds_data} = Oc Get kind=Build namespace=${namespace} - FOR ${build_data} IN @{builds_data} - ${build_name} = Set Variable ${build_data['metadata']['name']} - ${build_status} = Set Variable ${build_data['status']['phase']} - IF "${build_status}" == "Failed" or "${build_status}" == "Error" - Fail msg=Build ${build_name} is in ${build_status} state - ELSE - Wait Until Build Status Is namespace=${namespace} build_name=${build_name} - ... expected_status=Complete timeout=${build_timeout} - END + + ${buildconfigs_data} = Oc Get kind=BuildConfig namespace=${namespace} + + FOR ${buildconfig_data} IN @{buildconfigs_data} + ${buildconfig_name} = Set Variable ${buildconfig_data['metadata']['name']} + ${build_name} = Wait Until Build Exists namespace=${namespace} build_name_includes=${buildconfig_name} + Build Status Should Not Be namespace=${namespace} build_name=${build_name} + ... unexpected_status=Error + Build Status Should Not Be namespace=${namespace} build_name=${build_name} + ... unexpected_status=Failed + Wait Until Build Status Is namespace=${namespace} build_name=${build_name} + ... expected_status=Complete timeout=${build_timeout} END Verify All Builds Are Complete diff --git a/tests/Tests/100__deploy/100__installation/102__post_install.robot b/tests/Tests/100__deploy/100__installation/102__post_install.robot index 5c6dfe41f..7982f3fb6 100644 --- a/tests/Tests/100__deploy/100__installation/102__post_install.robot +++ b/tests/Tests/100__deploy/100__installation/102__post_install.robot @@ -136,14 +136,16 @@ Verify Oath-Proxy Image Is fetched From CPaaS Verify Container Image redhat-ods-applications ${pod} oauth-proxy ... "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8" -Verify Pytorch And Tensorflow Can Be Spawned - [Documentation] Check Cuda builds are complete and Verify Pytorch and Tensorflow can be spawned - [Tags] Sanity +Verify That CUDA Build Chain Succeeds + [Documentation] Check Cuda builds are complete. Verify CUDA (minimal-gpu), + ... Pytorch and Tensorflow can be spawned successfully + [Tags] Smoke ... Tier1 - ... ODS-480 ODS-481 + ... ODS-316 ODS-481 Wait Until All Builds Are Complete namespace=redhat-ods-applications - Verify Image Can Be Spawned image=pytorch size=Default - Verify Image Can Be Spawned image=tensorflow size=Default + Verify Image Can Be Spawned image=minimal-gpu size=Default + Verify Image Can Be Spawned image=pytorch size=Default + Verify Image Can Be Spawned image=tensorflow size=Default Verify That Blackbox-exporter Is Protected With Auth-proxy [Documentation] Vrifies the blackbok-exporter inludes 2 containers one for application and second for oauth proxy