Bugfixes for releasing ods-ci 1.19. Add AutomationBug and ProductBug (#…

…629) * Fix tests checking monitoring image urls Signed-off-by: Jorge Garcia Oncins <[email protected]> * Fix blackbox-exporter test updating expected number of endpoints Signed-off-by: Jorge Garcia Oncins <[email protected]> * Add retries when checking Grafana,Prometheus and BlackboxExpoter readyReplicas These tests where failing in some sanity tests. Adding retries to try to fix them Signed-off-by: Jorge Garcia Oncins <[email protected]> * Fix Notebook CPU Usage test Signed-off-by: Jorge Garcia Oncins <[email protected]> * Add AutomationBug to "Verify Clean Up ODS Deployer" This test was designed for the migration from jupyterhub to notebook controller in RHODS 1.17. It should be adapted for RHODS > 1.17 Signed-off-by: Jorge Garcia Oncins <[email protected]> * Disable some Grafana tests for RHODS 1.20. Fix bug Signed-off-by: Jorge Garcia Oncins <[email protected]> * Fix robocop issues in 200__metrics.robot Signed-off-by: Jorge Garcia Oncins <[email protected]> * Fix test Verify Unauthorized User Is Not Able To Spawn Jupyter Notebook Signed-off-by: Jorge Garcia Oncins <[email protected]> * Fix flakyness of "Verify Dashboard Is Shipped And Enabled Within ODS" Signed-off-by: Jorge Garcia Oncins <[email protected]> * Skip "Verify Unauthorized User Is Not..." on Self-Managed This test failed on Self-Managed and then, 28 tests failed after it. Signed-off-by: Jorge Garcia Oncins <[email protected]> * Fix test checking env vars Signed-off-by: Jorge Garcia Oncins <[email protected]> * Try to fix "Verify User Can Access Only Its Owned Projects" Signed-off-by: Jorge Garcia Oncins <[email protected]> * Add tag AutomationBug to "Verify Unauthorized User Is Not Able To..." Signed-off-by: Jorge Garcia Oncins <[email protected]> Signed-off-by: Jorge Garcia Oncins <[email protected]>
red-hat-data-services · Nov 27, 2022 · 9e830b7 · 9e830b7
1 parent e4381e1
commit 9e830b7
Show file tree

Hide file tree

Showing 12 changed files with 114 additions and 106 deletions.
diff --git a/tests/Resources/Page/OCPDashboard/Pods/Pods.robot b/tests/Resources/Page/OCPDashboard/Pods/Pods.robot
diff --git a/tests/Tests/100__deploy/100__installation/102__post_install.robot b/tests/Tests/100__deploy/100__installation/102__post_install.robot
@@ -16,8 +16,8 @@ Resource            ../../../Resources/Common.robot
 Suite Setup         RHOSi Setup
 Suite Teardown      RHOSi Teardown
 
-*** Test Cases ***
 
+*** Test Cases ***
 Verify Dashbord has no message with NO Component Found
     [Tags]  Sanity
     ...     Tier1
@@ -43,14 +43,14 @@ Verify Traefik Deployment
     Skip      msg=Traefik proxy is removed after KFNBC migration
 
 Verify Notebook Controller Deployment
-    [Documentation]  Verifies RHODS Notebook Controller deployment
+    [Documentation]    Verifies RHODS Notebook Controller deployment
     [Tags]    Sanity
     ...       ODS-546  ODS-294  ODS-1250  ODS-237
-     @{NBC} =  Oc Get    kind=Pod  namespace=redhat-ods-applications  label_selector=app=notebook-controller
-     @{ONBC}=  Oc Get    kind=Pod  namespace=redhat-ods-applications  label_selector=app=odh-notebook-controller
-     ${containerNames} =  Create List  manager
-     Verify Deployment  ${NBC}  1  1  ${containerNames}
-     Verify Deployment  ${ONBC}  1  1  ${containerNames}
+    @{NBC} =  Oc Get    kind=Pod  namespace=redhat-ods-applications  label_selector=app=notebook-controller
+    @{ONBC} =  Oc Get    kind=Pod  namespace=redhat-ods-applications  label_selector=app=odh-notebook-controller
+    ${containerNames} =  Create List  manager
+    Verify Deployment  ${NBC}  1  1  ${containerNames}
+    Verify Deployment  ${ONBC}  1  1  ${containerNames}
 
 Verify GPU Operator Deployment  # robocop: disable
     [Documentation]  Verifies Nvidia GPU Operator is correctly installed
@@ -81,8 +81,8 @@ Verify GPU Operator Deployment  # robocop: disable
     Verify DaemonSet Status  label=app=nvidia-dcgm-exporter  dsname=nvidia-dcgm-exporter
     Verify DaemonSet Status  label=app=nvidia-dcgm  dsname=nvidia-dcgm
     Verify DaemonSet Status  label=app=nvidia-device-plugin-daemonset  dsname=nvidia-device-plugin-daemonset
-    #app=nvidia-driver-daemonset-410.84.202205191234-0
-    #Verify DaemonSet Status  label=app=nvidia-driver-daemonset-*  dsname=nvidia-driver-daemonset-*
+    # app=nvidia-driver-daemonset-410.84.202205191234-0
+    # Verify DaemonSet Status  label=app=nvidia-driver-daemonset-*  dsname=nvidia-driver-daemonset-*
     Verify DaemonSet Status  label=app=nvidia-node-status-exporter  dsname=nvidia-node-status-exporter
     Verify DaemonSet Status  label=app=nvidia-operator-validator  dsname=nvidia-operator-validator
     Verify CR Status  crd=NodeFeatureDiscovery  cr_name=ocp-gpu-addon
@@ -93,47 +93,48 @@ Verify That Prometheus Image Is A CPaaS Built Image
     ...       Tier1
     ...       ODS-734
     ${pod} =    Find First Pod By Name    namespace=redhat-ods-monitoring    pod_start_with=prometheus-
-    Verify Container Image    redhat-ods-monitoring    ${pod}    prometheus
-    ...    "registry.redhat.io/openshift4/ose-prometheus"
-    Verify Container Image    redhat-ods-monitoring    ${pod}    oauth-proxy
-    ...    "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8"
+    Container Image Url Should Contain    redhat-ods-monitoring    ${pod}    prometheus
+    ...    registry.redhat.io/openshift4/ose-prometheus
+    Container Image Url Should Contain    redhat-ods-monitoring    ${pod}    oauth-proxy
+    ...    registry.redhat.io/openshift4/ose-oauth-proxy
 
-Verify That Grafana Image Is A Red Hat Built Image
+Verify That Grafana Image Is A CPaaS Built Image
     [Documentation]    Verifies the images used for grafana
     [Tags]    Sanity
     ...       Tier1
     ...       ODS-736
     ${pod} =    Find First Pod By Name    namespace=redhat-ods-monitoring    pod_start_with=grafana-
-    Verify Container Image    redhat-ods-monitoring    ${pod}    grafana
-    ...    "registry.redhat.io/rhel8/grafana:7"
-    Verify Container Image    redhat-ods-monitoring    ${pod}    auth-proxy
-    ...    "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8"
+    Container Image Url Should Contain    redhat-ods-monitoring    ${pod}    grafana
+    ...    registry.redhat.io/rhel8/grafana
+    Container Image Url Should Contain    redhat-ods-monitoring    ${pod}    auth-proxy
+    ...    registry.redhat.io/openshift4/ose-oauth-proxy
 
 Verify That Blackbox-exporter Image Is A CPaaS Built Image
     [Documentation]    Verifies the image used for blackbox-exporter
     [Tags]    Sanity
     ...       Tier1
     ...       ODS-735
     ${pod} =    Find First Pod By Name    namespace=redhat-ods-monitoring    pod_start_with=blackbox-exporter-
-    Verify Container Image    redhat-ods-monitoring    ${pod}    blackbox-exporter
-    ...    "quay.io/integreatly/prometheus-blackbox-exporter:v0.19.0"
+    Container Image Url Should Contain    redhat-ods-monitoring    ${pod}    blackbox-exporter
+    ...    quay.io/integreatly/prometheus-blackbox-exporter
 
 Verify That Alert Manager Image Is A CPaaS Built Image
     [Documentation]    Verifies the image used for alertmanager
     [Tags]    Sanity
     ...       Tier1
     ...       ODS-733
     ${pod} =    Find First Pod By Name    namespace=redhat-ods-monitoring    pod_start_with=prometheus-
-    Verify Container Image    redhat-ods-monitoring    ${pod}    alertmanager
-    ...    "registry.redhat.io/openshift4/ose-prometheus-alertmanager"
+    Container Image Url Should Contain    redhat-ods-monitoring    ${pod}    alertmanager
+    ...    registry.redhat.io/openshift4/ose-prometheus-alertmanager
 
-Verify Oath-Proxy Image Is fetched From CPaaS
+Verify Oath-Proxy Image Is A CPaaS Built Image
+    [Documentation]    Verifies the image used for oauth-proxy
     [Tags]      Sanity
     ...         Tier1
     ...         ODS-666
     ${pod} =    Find First Pod By Name  namespace=redhat-ods-applications   pod_start_with=rhods-dashboard-
-    Verify Container Image      redhat-ods-applications     ${pod}      oauth-proxy
-    ...     "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8"
+    Container Image Url Should Contain      redhat-ods-applications     ${pod}      oauth-proxy
+    ...     registry.redhat.io/openshift4/ose-oauth-proxy
 
 Verify That CUDA Build Chain Succeeds
     [Documentation]    Check Cuda builds are complete. Verify CUDA (minimal-gpu),
@@ -207,8 +208,10 @@ Verify Grafana Is Connected To Prometheus Using TLS
     [Documentation]    Verifies Grafana is connected to Prometheus using TLS
     [Tags]    Tier2
     ...       ODS-963
-    ...       AutomationBug
+    ...       ProductBug
     [Setup]  Set Library Search Order  Selenium Library
+    Skip If RHODS Version Greater Or Equal Than    version=1.20.0
+    ...    msg=Grafana was removed in RHODS 1.20
     Verify Grafana Datasources Have TLS Enabled
     Verify Grafana Can Obtain Data From Prometheus Datasource
     [Teardown]  Close Browser
@@ -297,7 +300,7 @@ Verify Authentication Is Required To Access BlackboxExporter
     ...    pm_token=${RHODS_PROMETHEUS_TOKEN}
     ...    username=${OCP_ADMIN_USER.USERNAME}
     ...    password=${OCP_ADMIN_USER.PASSWORD}
-    Length Should Be    ${links}    2
+    Length Should Be    ${links}    3    msg=Unexpected number of target endpoints in blackbox-exporter
     ${pod_name} =    Find First Pod By Name    namespace=redhat-ods-monitoring    pod_start_with=prometheus-
     FOR    ${link}    IN    @{links}
         ${command} =    Set Variable    curl --silent --insecure ${link}
@@ -341,7 +344,7 @@ Verify Grafana Can Obtain Data From Prometheus Datasource
     Run Promql Query  query=traefik_backend_server_up
     Page Should Contain  text=Graph
 
-Verify CPU And Memory Requests And Limits Are Defined For All Containers In All Pods in Project
+Verify CPU And Memory Requests And Limits Are Defined For All Containers In All Pods In Project
     [Documentation]    Verifies that CPU and Memory requests and limits are defined
     ...                for all containers in all pods for the specified project
     ...    Args:

diff --git a/tests/Tests/100__deploy/100__installation/103__grafana.robot b/tests/Tests/100__deploy/100__installation/103__grafana.robot
@@ -17,7 +17,6 @@ Verify Grafana Is Shipped And Enabled Within ODS
     @{grafana_deployment_info} =    Fetch Grafana Deployments Info
     @{grafana_services_info} =    Fetch Grafana Services Info
     @{grafana_routes_info} =    Fetch Grafana Routes Info
-    @{grafana_replicasets_info} =    Fetch Grafana ReplicaSets Info
     OpenShift Resource Field Value Should Be Equal As Strings    status.phase    Running    @{grafana_pods_info}
     OpenShift Resource Field Value Should Be Equal As Strings    status.containerStatuses[0].name    auth-proxy    @{grafana_pods_info}
     OpenShift Resource Field Value Should Be Equal As Strings    status.containerStatuses[1].name    grafana    @{grafana_pods_info}
@@ -30,12 +29,12 @@ Verify Grafana Is Shipped And Enabled Within ODS
     OpenShift Resource Field Value Should Be Equal As Strings    spec.ports[0].targetPort    8443    @{grafana_services_info}
     OpenShift Resource Field Value Should Match Regexp    spec.clusterIP    ^((25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])(\.(?!$)|$)){4}$
     ...    @{grafana_services_info}
-    OpenShift Resource Field Value Should Be Equal As Strings    status.readyReplicas    2    @{grafana_replicasets_info}
-    OpenShift Resource Field Value Should Be Equal As Strings    status.replicas    2    @{grafana_replicasets_info}
+    Wait Until Keyword Succeeds    10 times  5s    Verify Grafana ReplicaSets Info
     OpenShift Resource Field Value Should Be Equal As Strings    spec.port.targetPort    https    @{grafana_routes_info}
     OpenShift Resource Field Value Should Be Equal As Strings    spec.to.name    grafana    @{grafana_routes_info}
     OpenShift Resource Field Value Should Match Regexp    spec.host    ^(grafana-redhat-ods-monitoring.*)    @{grafana_routes_info}
 
+
 *** Keywords ***
 Fetch Grafana Pods Info
     [Documentation]    Fetch information from Grafana pods
@@ -75,12 +74,11 @@ Fetch Grafana Routes Info
     ...    namespace=redhat-ods-monitoring
     [Return]    @{grafana_routes_info}
 
-Fetch Grafana ReplicaSets Info
-    [Documentation]    Fetch information from Grafana replicasets
-    ...    Args:
-    ...        None
-    ...    Returns:
-    ...        grafana_replicasets_info(list(dict)): Grafana replicasets selected by label and namespace
+Verify Grafana ReplicaSets Info
+    [Documentation]    Fetchs and verifies information for Grafana replicasets
     @{grafana_replicasets_info} =    Oc Get    kind=ReplicaSet    api_version=v1    namespace=redhat-ods-monitoring
     ...    label_selector=app=grafana
-    [Return]    @{grafana_replicasets_info}
+    OpenShift Resource Field Value Should Be Equal As Strings
+    ...     status.readyReplicas    2    @{grafana_replicasets_info}
+    OpenShift Resource Field Value Should Be Equal As Strings    status.replicas    2    @{grafana_replicasets_info}
+
diff --git a/tests/Tests/100__deploy/100__installation/103__post_install_monitoring_port_forwarding.robot b/tests/Tests/100__deploy/100__installation/103__post_install_monitoring_port_forwarding.robot
@@ -19,12 +19,12 @@ Verify That MT-SRE Can Connect To Grafana Using Port Forwarding
     [Documentation]    Verifies that Grafana is accessible by MT-SRE when using oc port-forwarding
     [Tags]   Tier2
     ...      ODS-754
-    ...      AutomationBug
+    Skip If RHODS Version Greater Or Equal Than    version=1.20.0
+    ...    msg=Grafana was removed in RHODS 1.20
     ${grafana_port_forwarding_process} =  Enable Access To Grafana Using OpenShift Port Forwarding
     Wait Until HTTP Status Code Is  url=http://localhost:3001/api/health  retry=7x  expected_status_code=200
     Open Browser  http://localhost:3001  browser=${BROWSER.NAME}  options=${BROWSER.OPTIONS}
     Go To Grafana Dashboard Search
-    Verify If Jupyterhub Sli Is Present
     [Teardown]  Terminate Process  ${grafana_port_forwarding_process}
 
 Verify MT-SRE Can Connect To Prometheus Using Port-Forwarding
@@ -49,10 +49,6 @@ Verify MT-SRE Can Connect To Alert Manager Using Port-forwarding
 
 
 *** Keywords ***
-Verify If Jupyterhub Sli Is Present
-    [Documentation]    Verifies Jupyterhub SLI element is present
-    Wait Until Element Is Visible    ${JUPYTERHB_SLI}   error="Jupyterhub SLIs Not Found"
-
 Verify Access To Prometheus Using Browser
     [Documentation]  Verifies if we are able to access Prometheus without asking to login
     Wait Until Page Contains    text=Prometheus

diff --git a/tests/Tests/100__deploy/100__installation/104__dashboard.robot b/tests/Tests/100__deploy/100__installation/104__dashboard.robot
@@ -15,13 +15,11 @@ Verify Dashboard Is Shipped And Enabled Within ODS
     [Tags]    Sanity
     ...       Tier1
     ...       ODS-233
-    ...       AutomationBug
     [Setup]     Set Expected Replicas Based On Version
     @{dashboard_pods_info} =    Fetch Dashboard Pods
     @{dashboard_deployments_info} =    Fetch Dashboard Deployments
     @{dashboard_services_info} =    Fetch Dashboard Services
     @{dashboard_routes_info} =    Fetch Dashboard Routes
-    @{dashboard_replicasets_info} =    Fetch Dashboard ReplicaSets
     Verify Dashboard Deployment
     OpenShift Resource Field Value Should Be Equal As Strings    status.phase    Running    @{dashboard_pods_info}
     OpenShift Resource Field Value Should Be Equal As Strings    status.conditions[2].status    True    @{dashboard_pods_info}
@@ -30,8 +28,7 @@ Verify Dashboard Is Shipped And Enabled Within ODS
     OpenShift Resource Field Value Should Be Equal As Strings    spec.ports[0].targetPort    8443    @{dashboard_services_info}
     OpenShift Resource Field Value Should Match Regexp    spec.clusterIP    ^((25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])(\.(?!$)|$)){4}$
     ...    @{dashboard_services_info}
-    OpenShift Resource Field Value Should Be Equal As Strings    status.readyReplicas    ${EXP_DASHBOARD_REPLICAS}    @{dashboard_replicasets_info}
-    OpenShift Resource Field Value Should Be Equal As Strings    status.replicas    ${EXP_DASHBOARD_REPLICAS}    @{dashboard_replicasets_info}
+    Wait Until Keyword Succeeds    10 times  5s    Verify Dashboard ReplicaSets Info
     OpenShift Resource Field Value Should Be Equal As Strings    spec.port.targetPort    8443    @{dashboard_routes_info}
     OpenShift Resource Field Value Should Be Equal As Strings    spec.to.name    rhods-dashboard    @{dashboard_routes_info}
     OpenShift Resource Field Value Should Match Regexp    spec.host    dashboard-redhat-ods-applications.*    @{dashboard_routes_info}
@@ -167,15 +164,14 @@ Fetch Dashboard Routes
     ...    namespace=redhat-ods-applications
     [Return]    @{dashboard_routes_info}
 
-Fetch Dashboard ReplicaSets
-    [Documentation]    Fetch information from Dashboard replicasets
-    ...    Args:
-    ...        None
-    ...    Returns:
-    ...        dashboard_replicasets_info(list(dict)): Dashboard replicasets selected by label and namespace
+Verify Dashboard ReplicaSets Info
+    [Documentation]    Fetchs and verifies information from Dashboard replicasets
     @{dashboard_replicasets_info} =    Oc Get    kind=ReplicaSet    api_version=v1    namespace=redhat-ods-applications
     ...    label_selector=app=rhods-dashboard
-    [Return]    @{dashboard_replicasets_info}
+    OpenShift Resource Field Value Should Be Equal As Strings    status.readyReplicas
+    ...    ${EXP_DASHBOARD_REPLICAS}    @{dashboard_replicasets_info}
+    OpenShift Resource Field Value Should Be Equal As Strings    status.replicas
+    ...    ${EXP_DASHBOARD_REPLICAS}    @{dashboard_replicasets_info}
 
 Verify Dashboard Deployment
     [Documentation]  Verifies RHODS Dashboard deployment

diff --git a/tests/Tests/100__deploy/100__installation/105__prometheus.robot b/tests/Tests/100__deploy/100__installation/105__prometheus.robot
@@ -17,7 +17,6 @@ Verify Prometheus Is Shipped And Enabled Within ODS
     @{prometheus_deployment_info} =    Fetch Prometheus Deployments Info
     @{prometheus_services_info} =    Fetch Prometheus Services Info
     @{prometheus_routes_info} =    Fetch Prometheus Routes Info
-    @{prometheus_replicasets_info} =    Fetch Prometheus ReplicaSets Info
     OpenShift Resource Field Value Should Be Equal As Strings    status.phase    Running    @{prometheus_pods_info}
     OpenShift Resource Field Value Should Be Equal As Strings    status.containerStatuses[0].name    alertmanager    @{prometheus_pods_info}
     OpenShift Resource Field Value Should Be Equal As Strings    status.containerStatuses[1].name    alertmanager-proxy    @{prometheus_pods_info}
@@ -32,12 +31,12 @@ Verify Prometheus Is Shipped And Enabled Within ODS
     OpenShift Resource Field Value Should Be Equal As Strings    spec.ports[0].targetPort    https    @{prometheus_services_info}
     OpenShift Resource Field Value Should Match Regexp    spec.clusterIP    ^((25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])(\.(?!$)|$)){4}$
     ...    @{prometheus_services_info}
-    OpenShift Resource Field Value Should Be Equal As Strings    status.readyReplicas    1    @{prometheus_replicasets_info}
-    OpenShift Resource Field Value Should Be Equal As Strings    status.replicas    1    @{prometheus_replicasets_info}
+    Wait Until Keyword Succeeds    10 times  5s    Verify Prometheus ReplicaSets Info
     OpenShift Resource Field Value Should Be Equal As Strings    spec.port.targetPort    https    @{prometheus_routes_info}
     OpenShift Resource Field Value Should Be Equal As Strings    spec.to.name    prometheus    @{prometheus_routes_info}
     OpenShift Resource Field Value Should Match Regexp    spec.host    ^(prometheus-redhat-ods-monitoring.*)    @{prometheus_routes_info}
 
+
 *** Keywords ***
 Fetch Prometheus Pods Info
     [Documentation]    Fetch information from Prometheus pods
@@ -77,12 +76,10 @@ Fetch Prometheus Routes Info
     ...    namespace=redhat-ods-monitoring
     [Return]    @{prometheus_routes_info}
 
-Fetch Prometheus ReplicaSets Info
-    [Documentation]    Fetch information from Prometheus replicasets
-    ...    Args:
-    ...        None
-    ...    Returns:
-    ...        prometheus_replicasets_info(list(dict)): Prometheus replicasets selected by label and namespace
+Verify Prometheus ReplicaSets Info
+    [Documentation]    Fetches and verifies information from Prometheus replicasets
     @{prometheus_replicasets_info} =    Oc Get    kind=ReplicaSet    api_version=v1    namespace=redhat-ods-monitoring
     ...    label_selector=deployment=prometheus
-    [Return]    @{prometheus_replicasets_info}
+    OpenShift Resource Field Value Should Be Equal As Strings
+    ...    status.readyReplicas    1    @{prometheus_replicasets_info}
+    OpenShift Resource Field Value Should Be Equal As Strings    status.replicas    1    @{prometheus_replicasets_info}