update Magpie==4.1.1 and Twitcher==0.10.0

[fmigneault]

Overview

Security updates.

Changes

Non-breaking changes

• Magpie/Twitcher: update Python packages and base Docker image to address security vulnerabilities

  • Magpie 4.1.1 (relates to Ouranosinc/Magpie#622).
  • Twitcher 0.10.0 (relates to bird-house/twitcher#136).

Breaking changes

• n/a

Related Issue / Discussion

PRs that are an agglomeration of vulnerability fixes flagged by Snyk.

• security updates for setuptools, requests, urllib, zipp, twitcher Ouranosinc/Magpie#622
• python updates and security fixes twitcher#136

Additional Information

• The change from Magpie 3.x to 4.x is caused by the drop of Python 3.5 to 3.7, and the addition of 3.12 some time ago. Major version was used only to highlight this change in case older versions were still employed by some instances. From a technical aspect in birdhouse-deploy, Magpie with Python 3.11 was already in use for a while (see https://github.com/Ouranosinc/Magpie/blob/3.38.0/Dockerfile#L1). Only minor package dependency differences are actually expected.

CI Operations

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

[tlvu]

Looks good to me but I see the IaC pipeline did not trigger again !!! Could this PR at least pass under the IaC pipeline because I don't be performing manual verification this time.

I have the impression the IaC pipeline has not triggered automatically since a few weeks already. I already noticed this with the GeoServer upgrade.

[fmigneault]

Manual trigger: http://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2742/

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2742/
Result ❌ FAILURE

BIRDHOUSE_DEPLOY_BRANCH : update-magpie-twitcher
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1656/

NOTEBOOK TEST RESULTS

    
[2024-07-24T18:18:05.137Z] ============================= test session starts ==============================
[2024-07-24T18:18:05.137Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-07-24T18:18:05.137Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-07-24T18:18:05.138Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-07-24T18:18:05.138Z] collected 308 items
[2024-07-24T18:18:05.138Z] 
[2024-07-24T18:18:15.097Z] notebooks-auth/geoserver.ipynb ..................                        [  5%]
[2024-07-24T18:18:30.554Z] notebooks-auth/test_cowbird_jupyter.ipynb ......FFF.                     [  9%]
[2024-07-24T18:18:38.018Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-07-24T18:19:25.344Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-07-24T18:19:43.672Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 16%]
[2024-07-24T18:19:52.103Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 18%]
[2024-07-24T18:19:57.686Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 21%]
[2024-07-24T18:27:39.077Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 25%]
[2024-07-24T18:27:39.078Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 25%]
[2024-07-24T18:27:39.858Z] ...............                                                          [ 30%]
[2024-07-24T18:27:47.094Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2024-07-24T18:27:57.824Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2024-07-24T18:28:12.965Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 36%]
[2024-07-24T18:28:18.614Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 38%]
[2024-07-24T18:28:23.142Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 39%]
[2024-07-24T18:31:46.792Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2024-07-24T18:32:43.838Z] .............                                                            [ 49%]
[2024-07-24T18:32:48.293Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 50%]
[2024-07-24T18:32:49.811Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2024-07-24T18:33:05.845Z] .................                                                        [ 58%]
[2024-07-24T18:33:13.548Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 60%]
[2024-07-24T18:33:14.489Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2024-07-24T18:33:32.015Z] .........                                                                [ 63%]
[2024-07-24T18:33:41.485Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2024-07-24T18:33:50.467Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2024-07-24T18:33:51.857Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 69%]
[2024-07-24T18:33:54.925Z] .....                                                                    [ 71%]
[2024-07-24T18:34:13.282Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 71%]
[2024-07-24T18:34:30.026Z] ............                                                             [ 75%]
[2024-07-24T18:34:48.141Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-07-24T18:35:11.823Z] .....s.                                                                  [ 77%]
[2024-07-24T18:35:18.426Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 78%]
[2024-07-24T18:35:35.883Z] ..                                                                       [ 78%]
[2024-07-24T18:35:42.499Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 79%]
[2024-07-24T18:36:41.689Z] .........                                                                [ 82%]
[2024-07-24T18:36:51.735Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-07-24T18:41:42.433Z] ....                                                                     [ 83%]
[2024-07-24T18:41:44.351Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 84%]
[2024-07-24T18:41:49.659Z] ....                                                                     [ 85%]
[2024-07-24T18:41:53.868Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
[2024-07-24T18:41:59.199Z] .....                                                                    [ 87%]
[2024-07-24T18:42:03.407Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
[2024-07-24T18:42:28.222Z] .............                                                            [ 91%]
[2024-07-24T18:42:32.436Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
[2024-07-24T18:42:33.508Z] ......                                                                   [ 94%]
[2024-07-24T18:42:35.806Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-07-24T18:45:09.962Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-07-24T18:45:09.962Z] 
[2024-07-24T18:45:09.963Z] =================================== FAILURES ===================================
    
  

[tlvu]

FYI le notebook cowbird n'a pas passé.

[fmigneault]

@tlvu
Thanks. Will need to investigate some more...
The error seems to be caused by a JSON parsing issue on the Jupyter API endpoint.
Not sure what could have caused it since all other notebooks doing Magpie logins worked...
Maybe I will have time Friday, but might be delayed until the end of August otherwise.

[fmigneault]

run tests

[fmigneault]

run tests

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2795/
Result ❌ FAILURE

BIRDHOUSE_DEPLOY_BRANCH : update-magpie-twitcher
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1682/

NOTEBOOK TEST RESULTS

    
[2024-09-10T17:26:17.814Z] ============================= test session starts ==============================
[2024-09-10T17:26:17.814Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-09-10T17:26:17.814Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-09-10T17:26:17.814Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-09-10T17:26:17.814Z] collected 308 items
[2024-09-10T17:26:17.814Z] 
[2024-09-10T17:26:28.528Z] notebooks-auth/geoserver.ipynb ..................                        [  5%]
[2024-09-10T17:27:34.815Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  9%]
[2024-09-10T17:27:40.134Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-09-10T17:28:27.255Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-09-10T17:28:35.946Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 16%]
[2024-09-10T17:28:45.344Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 18%]
[2024-09-10T17:28:54.285Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .FFFFFFF       [ 21%]
[2024-09-10T17:36:21.054Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 25%]
[2024-09-10T17:36:22.447Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 25%]
[2024-09-10T17:36:30.473Z] ...............                                                          [ 30%]
[2024-09-10T17:36:38.314Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2024-09-10T17:36:51.142Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2024-09-10T17:37:06.340Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 36%]
[2024-09-10T17:37:12.261Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 38%]
[2024-09-10T17:37:16.781Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 39%]
[2024-09-10T17:40:27.401Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2024-09-10T17:41:37.085Z] .............                                                            [ 49%]
[2024-09-10T17:41:38.474Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 50%]
[2024-09-10T17:41:40.311Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2024-09-10T17:41:56.400Z] .................                                                        [ 58%]
[2024-09-10T17:42:04.452Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 60%]
[2024-09-10T17:42:05.398Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2024-09-10T17:42:23.736Z] .........                                                                [ 63%]
[2024-09-10T17:42:32.824Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2024-09-10T17:42:41.391Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2024-09-10T17:42:42.772Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 69%]
[2024-09-10T17:42:45.846Z] .....                                                                    [ 71%]
[2024-09-10T17:43:07.824Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 71%]
[2024-09-10T17:43:22.764Z] ............                                                             [ 75%]
[2024-09-10T17:43:37.684Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-09-10T17:44:00.321Z] .....s.                                                                  [ 77%]
[2024-09-10T17:44:08.471Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 78%]
[2024-09-10T17:44:24.775Z] ..                                                                       [ 78%]
[2024-09-10T17:44:31.381Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 79%]
[2024-09-10T17:45:32.119Z] .........                                                                [ 82%]
[2024-09-10T17:45:40.259Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-09-10T17:50:41.745Z] ....                                                                     [ 83%]
[2024-09-10T17:50:41.745Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 84%]
[2024-09-10T17:50:41.745Z] ....                                                                     [ 85%]
[2024-09-10T17:50:41.745Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
[2024-09-10T17:50:45.327Z] .....                                                                    [ 87%]
[2024-09-10T17:50:49.533Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
[2024-09-10T17:51:16.797Z] .............                                                            [ 91%]
[2024-09-10T17:51:21.001Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
[2024-09-10T17:51:23.002Z] ......                                                                   [ 94%]
[2024-09-10T17:51:25.739Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-09-10T17:54:11.262Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-09-10T17:54:11.262Z] 
[2024-09-10T17:54:11.262Z] =================================== FAILURES ===================================
    
  

[fmigneault]

@tlvu
Requesting review again since it's been a while. I want to make sure it doesn't cause a surprise on your side.
The triggers are working again though! 🎉
Seems like the Cowbird tests are working now. Might have been just a sporadic issue?

Looks like the WMS is still failing. This one has very sporadic failures. Each of the failing tests below are caused by this WMS notebook (whether using these new versions of Magpie/Twitcher or the actual ones), but we can see it also passes ~½ times.

[tlvu]

Requesting review again since it's been a while. I want to make sure it doesn't cause a surprise on your side.

Is there any DB upgrade? Otherwise it should be fine.

[fmigneault]

@tlvu

Is there any DB upgrade

There shouldn't be any changes.

The 3.x⇾4.x change was only to highlight the deprecation of older Python version (<3.8), which were not used anyway by the docker already using Python 3.11.

[tlvu]

LGTM