[Spike] Update the bundle used by a given released crc binary

[praveenkumar]

As of now each crc release go with specific version of bundle and by default that specific bundle downloaded as part of setup command. We publish the bundles to mirror which is tested by internal CI pipeline on all the platform by QE. Since the team responsibility is widen now and minor releases of openshift doesn't really require a different crc release just sake of updating the bundle, we need to put a plan around how we point a existing crc release to a particular version of bundle instead hardcoded it in the binary.

Assumption we have here:

• Mirror is source of truth for bundle release and only tested bundles available there.
• Bundle is tested with that specific version of crc binary on all the platform before it mirrored to release location.
• User should still get warning if use a bundle which is not part of supported matrix

Suggestions during the internal chat to achieve this

• Have a marker json file which contain the list of bundle supported for a specific released version of crc and maintain this file
  • This file can be at top level of https://mirror.openshift.com/pub/openshift-v4/clients/crc/ location where we need to make request to release team to update it everytime there is update on supported bundle filed.
  • Or we can have this file to our github release https://github.com/crc-org/crc/releases and have it as part of assets for each single release this way it is easier to update it without involving the release team.

If we go with the quay then #3206 (comment) and subsequent discussion have good overview.

[adrianriobo]

list of bundle supported for a specific released version of crc

Would this affect only to latest released version? I mean, lest say latest is 2.38.1 and as of now 4.15.17 is supported on that . Now we decide to publish 4.15.19 as so we check and it is supported so we publish and added a record on the marker json.

If this happens I guess the crc will check the latest supported version and that one would be "default" on setup

Maybe also having a command to show supported versions ( which basically will show the values on the marker) and then the user can directly set one of the supported versions?

Also going back to the question Are we planning on maintain X number of latest releases (i.e. the last 2 crc releases should be checked?) Or as I asked only the latest?

[adrianriobo]

Also are we gonna ensure supported versions should match for ocp and microshift? Or would it be possible at some point a version on some preset may not be supported?

[gbraad]

Maybe also having a command to show supported versions

this would be necessary, and as with the rules of user interfaces: "the user is in control". So we will inform the user about a new version, but will not default to this unless chosen by the user. Otherwise there is no easy way to install the older (or original) version that the CRC version was released with.

[anjannath]

Have a marker json file which contain the list of bundle supported for a specific released version of crc and maintain this file

this means we have to keep updating the marker file after its published, so this might not be easily possible in mirror, if we publish it in the github release, then once there's a new bundle released we have to replace the existing marker file in the same github release. I don't think this is a big problem, but usually we don't want to in-place change something in a published release

a possible marker file could look like:

{
    "crc_version": "2.40.0",
    "latest_bundle_version": "4.16.7",
    "supported_bundle_versions": [
        "4.16.0",
        "4.16.4",
        "4.16.7"
    ]
}

and then on the crc cli side we adjust the validation checks and possibly add some caching for the marker file so that its not downloaded for every crc start

[cfergeau]

Imo there are 2 distinct problems discussed here.
Once we release a newer crc version, I don't think we'll keep "supporting" newer bundles with an older crc version? If that's correct, to know if bundle version 5.37.7 is supported with crc version 8.1, it's enough to know the bundle version at release time for each crc release:

• crc version 8.1 released with bundle version 5.37.2
• crc version 8.2 released with bundle version 5.37.7
• crc version 8.3 released with bundle version 5.37.15

Just from this, we know that crc version 8.1 supported bundle 5.37.5, that crc version 8.2 supported bundle 5.37.10 and so on.

However, if we want to be able to inform the user of which bundles are available, then we need a list of released bundles, but this does not necessarily need to be tied to the 'support' data if what I suggested before is good enough. In a way, we could even get it from curl https://mirror.openshift.com/pub/openshift-v4/clients/crc/bundles/openshift/ but that's messy ;)

[praveenkumar]

I really think this is also a good topic to discuss during f2f. cc @gbraad

[anjannath]

Just from this, we know that crc version 8.1 supported bundle 5.37.5, that crc version 8.2 supported bundle 5.37.10 and so on.

so, any crc version that was released already knows the minimum supported bundle version, but to know what other bundle versions it can support, we have to check:

• if there's a newer crc release and if yes, what bundle that newer release contains (from release-info.json)
• then from the available bundles any bundle version that is lower then the bundle version contained in the latest release and is higher then the original hard coded bundle version is supported

In a way, we could even get it from curl https://mirror.openshift.com/pub/openshift-v4/clients/crc/bundles/openshift/ but that's messy ;)

tried to find out how messy it could be, with the following code we are able to get the list of bundle versions:

package main

import (
	"fmt"
	"net/http"
	"os"
	"time"

	"golang.org/x/net/html"
)

func main() {
	fmt.Println("Downloading bundle html page")
	bundlePageURL := "https://mirror.openshift.com/pub/openshift-v4/clients/crc/bundles/openshift/"

	client := &http.Client{
		Timeout: 5 * time.Second,
	}

	res, err := client.Get(bundlePageURL)
	if err != nil {
		fmt.Printf("Error: %v", err)
		os.Exit(1)
	}
	parsed, err := html.Parse(res.Body)
	if err != nil {
		fmt.Printf("Error: %v", err)
		os.Exit(1)
	}
	var f func(*html.Node)
	f = func(n *html.Node) {
		if n.Type == html.ElementNode && n.Data == "span" {
			fmt.Println(n.FirstChild.Data)
		}
		for c := n.FirstChild; c != nil; c = c.NextSibling {
			f(c)
		}
	}
	f(parsed)
}

[praveenkumar]

tried to find out how messy it could be, with the following code we are able to get the list of bundle versions:

@anjannath Going through complete webpage and then parsing it already a messy until there is some short of api. Assume tomorrow if mirror location change the way it deliver or show case bundle or put it behind some kind of auth. Since we have to do it from scratch then better to go with some kind of structured data either maintained by us (if we put that in our GH release) or to mirror side.

[anjannath]

tried to find out how messy it could be, with the following code we are able to get the list of bundle versions:

@anjannath Going through complete webpage and then parsing it already a messy until there is some short of api. Assume tomorrow if mirror location change the way it deliver or show case bundle or put it behind some kind of auth. Since we have to do it from scratch then better to go with some kind of structured data either maintained by us (if we put that in our GH release) or to mirror side.

yeah true that parsing the webpage is not very future proof and even if they update a little thing and diverge from what the parsing code expects then this will break

but its also worth considering the effort to have an api vs just updating the parsing code when things change, or if that webpage is ever going to change, if i am not mistaken mirror web page has not changed at all in the past

for me i just wanted to see how much code it needed to parse the website and get the bundle versions

[praveenkumar]

for me i just wanted to see how much code it needed to parse the website and get the bundle versions

@anjannath for experiment purpose sure but if we have structured data then code would be much lesser and easy maintainable.