Default External Builder Approach for Kubernetes

[jkneubuh]

Bundle a default k8s Chaincode Builder into the peer #3405

In this discussion: Converge on an approach for running an external chaincode builder / chaincode pods on Kubernetes.

Requirements:

• Easy to use / works "out of the box" with minimal (ideally zero) configuration or Kubernetes expertise

• Extensible / customizable for deployments with additional integration needs

• Chaincode image build is run independently from Fabric: assume a CC image exists and is available at a container registry.

• No Docker in anything

• Retain the existing chaincode / peer builder lifecycle

The current / best stake is @jt-nti approach articulated in fabric-builder-k8s

[SamYuan1990]

ref to myself at #3405 (comment)

At peer side

Peer does NOT compile CC or prepare Docker Image
Peer does NOT start the CC process / Kubernetes pod to avoid extend k8s permission or any risk with k8s secret access.
Enhance Peer with or "ccaas" Builder, "DIND" Builder, "k8s" Builder and make it able to config by environment variable.
To provide user friendly, an k8s operator

User installs CC package containing Docker Image URL/Service URL/Source code link(some Customer resource definition).
Operator able to create k8s resources with necessary permission.
Operator able to access github with necessary permission to checkout code.
Operator response for chaincode lifecycle on peers with in Organization.
As a sidecar mode interaction with peer, when peer pod been deployed, overwrite peer chaincode builder setting, as an optional approach?

[jt-nti]

The current k8s builder prototype doesn't use an operator because an operator is not required- the standard kubernetes objects are sufficient to run chaincode so I didn't want to introduce another moving part. I think that with the current Fabric builder/launcher limitations, the right approach is to use a pod directly, i.e. instead of a deployment which is what the prototype currently uses. That's next on my list of things to experiment with anyway. (Note: that issue also includes a possible future enhancement to the Fabric builder/launcher implementation which would completely hand over managing the chaincode process to kubernetes, but that's unlikely to happen in the near future.)

Working nicely with CI/CD pipelines is a really key part of the motivation for the builder, and I've updated the sample k8s chaincode project so that the build pipeline publishes everything required to deploy the chaincode. You could then easily start hooking in to the Fabric chaincode deploy process but at some point you get into multiple organisations and manual steps to approve/commit things in production environments, or at least I think you should do!

Using an operator to help configure custom builders seems like a good option, especially if the decision is not to include them in the peer image out of the box.

[SamYuan1990]

to be honest, I didn't see too much problems in build phase with CCAAS. In my sight, chaincode as a source code repo and well in CI/CD area there lots of way to make it from source code to a k8s pod, deployment, service and so on. Well, we can template the docker build process with a "Default External Builder"
But what are we talking about?

• the "builder" in peer, which response for ensure peer knows where the chaincode service endpoint is.
• some template/cicd process

sample k8s chaincode project

already show the case to work nicely with CI/CD pipeline which is good.

I think that with the current Fabric builder/launcher limitations, the right approach is to use a pod directly, i.e. instead of a deployment which is what the prototype currently uses.

@jt-nti , may I know the reason for above? In my point of view, make chaincode as a service in k8s is good enough. so the network is not direct from peer to pod, but peer to service endpoint.

multiple organisations

@jt-nti , I am agree that multiple organisations makes things complex.
But what are we talking about?

• the "builder" in peer, which response for ensure peer knows where the chaincode service endpoint is.
• some way to management chaincode lifecycle

[SamYuan1990]

in my point of view the "builder" in peer, which response for ensure peer knows where the chaincode service endpoint is.
it is not necessary for "builder" to build from source to image or deployment the image in k8s.

if Default External Builder means

1. release some new binary files with peer image
2. extends k8s permission to start peer, as build from source and deployment image in k8s

please let me know according to principle of least privilege, which one is better?

• "CCAAS with CI/CD"
• "making peer able to access k8s secret for github and docker reg, meanwhile able to deploy k8s pod."(what a Default External Builder meaning for me now, please correct me for any misunderstanding)

[SamYuan1990]

Thanks @SamYuan1990

YES. The k8s-builder is and will be compatible with a fabric operator.

One challenge with this approach is that we do not, in general, have consensus on a standard Kubernetes operator for Fabric. The fabric-builder-k8s technique as a "default" builder, allows for successful chaincode execution in environments using Kubernetes, but may not be based on a formal operator.

• Peer does NOT compile CC or prepare a Docker image
• Peer does NOT start the CC process / k8s pod. The peer does NOT have visibility to the k8s secrets. The k8s-builder relies on the standard Kubernetes RBAC and service account configuration.
• Selection of "ccaas" vs "DIND" vs "k8s" vs ..."xyzzy" builder is specified in the peer chaincode package.tgz. Availability of the builders is determined by registration in the core yaml builder stanza.

A Kubernetes operator can certainly launch the chaincode pods. However - we still must tie in to the chaincode lifecycle using the existing builder hooks, OR provide an alternate pipeline within the peer golang routines. Both Matthew and I did extensive reviews of the peer logic in this area and came to the same conclusion: "let's re-use the existing practices, and write external builders."

If we followed the sequence above, it would need to:

1. User installs CC "k8s" package
2. Peer invokes an external builder script.
3. External builder script must instruct the operator to launch pods and await an async message that "CC is ready."
4. Operator receives request to start a CC pod, reconciles request, and goes back to sleep
5. External builder script receives "it's up" async message and resumes control to the peer.

I suppose if we away from builder inside peer to keep peer pod following principle of least privilege.
is it really need to have assumptions above?

1. User installs CC "k8s" package

the install can be handle over to ci/cd pipeline as Tekton, so this one is optional.

2. Peer invokes an external builder script.

Yes.

3. External builder script must instruct the operator to launch pods and await an async message that "CC is ready."

No. I don't want create any k8s resource from peer which will request peer to against principle of least privilege. Instead, the operator should trigger chaincode lifecycle, as External builder script can be CCAAS.

4. Operator receives request to start a CC pod, reconciles request, and goes back to sleep

Start a CC pod etc can be hand over to Tekton, so this one is optional.

5. External builder script receives "it's up" async message and resumes control to the peer.

Not necessary, the status can be show and checked in CRD.

In my point of view, the operator is just help user do chaincode lifecycle among peers within same org, and return status in CRD.
The chaincode "install" process will not something as
peer chaincode install but kubectl apply CRD,yml
the operator just do things as peer chaincode install to peer pods with in same org, as minimal scope. (start CC pod and other things is optional if hand over to CICD tools.)

[jt-nti]

If you're happy with the current CCaaS approach, that's great- it's already there and much improved since there's now a CCaaS builder provided with Fabric. The intention is not to replace the CCaaS but to provide an alternative.

There is no "default" builder in Fabric as such- the core.yaml file contains a section to define the builders you want to use and if none of these match the chaincode package being deployed, the peer falls back to the old DIND approach, and fails if that doesn't work (which happened on the community call demo!)

[yacovm]

#3405 (comment)

[yacovm]

The design in this thread refers to a default builder that maintains the "traditional" CC --> Peer directionality of the gRPC handshake.

This approach supplements the CCaaS builder, which inverts the handshake parity and makes no assumption about the run state of the CC, other than the gRPC(s) URL for the connection.

Oh, that part was not clear to me. I thought this entire design is related only to the chaincode as a service.

So now I am even more puzzled - why would you want to run the chaincode on Kubernetes in the legacy model where the peer governs the chaincode runtime? The entire point of the chaincode as a service was that someone else (e.g K8S) takes care of running the chaincode, not the peer. Having the peer interacting with Kubernetes to build the chaincode container does not make any sense to me at all.

[jkneubuh]

That's great that we can use name:version in lieu of label:hash. I understand that the CCaaS may be launched in advance of any interaction with the peer (client CLI) to install the package, etc. It was / is a hassle to determine the CORE_CHAINCODE_ID_NAME by running sha hashes on the client side or scraping the output from the peer CLI installing the package.

You may be confused as you are mixing the two models together. This is understandable as the chaincode lifecycle mechanisms are poorly articulated, documented, and implemented in Fabric. We are all getting better at finding crisp terminology to refer to an ambiguous circumstance -- this is why this discussion page is extremely helpful.

We can certainly write a k8s builder that launches the CC pods "as a service." The challenge that we are facing with the current (state in Fabric 2.5.x) is that "someone else (e.g. K8S)" needs to take an action on the Kube cluster to launch the CC service. Unfortunately the "someone else" - literally means Someone else, i.e.. a person who either sits down and does the action manually, or invests time to script the approach in an environment that is relevant for their cluster / fabric network. This is the user / manual step that is being removed - by bundling a default implementation into the peer that understands enough to bring up CC pods in the cluster without manual intervention. In cases requiring customization (e.g. @SamYuan1990 's target of linking in a Tekton flow to integrate with a CI/CD flow), the user has full flexibility to implement a custom approach using the Sykes build lifecycle / Builder hooks.

The proposal is to introduce a new "external chaincode builder" that skips the "chaincode compilation" phase, launches a chaincode pod in a target k8s namespace, using the traditional "CC calls Peer" gRPC(s) handshake direction at launch time.

Really what we are converging on is an approach that will cut out the CC "compilation" phase, retaining the feature whereby "the system" (kube) launches a CC routine on behalf of the over-burndened sys admin. (Ultimately the parity of handshake initialization between peer/CC is irrelevant.)

I agree strongly with this statement:

Having the peer interacting with Kubernetes to build the chaincode container does not make any sense to me at all.

It' s not clear if you are referring to "build the chaincode container" (e.g. run a CC compilation and image generation), or "launch the chaincode process" in the statement above. I absolutely agree that we can, should, and must offload the "build the chaincode container" practices, moving the responsibilities of this step to the CC author.

If I may - one thing that may help alleviate and illuminate the various aspects of this discussion - is to go through the actual exercise of running the approach called out in the fabric-builder-k8s project. It really is quite a spectacular improvement for users of Fabric, and worth watching first-hand.

[yacovm]

You may be confused as you are mixing the two models together. This is understandable as the chaincode lifecycle mechanisms are poorly articulated, documented, and implemented in Fabric.

Fortunately, Fabric is open source and instead of reading documentation one can just read the source code ;-)

The challenge that we are facing with the current (state in Fabric 2.5.x) is that "someone else (e.g. K8S)" needs to take an action on the Kube cluster to launch the CC service. Unfortunately the "someone else" - literally means Someone else, i.e.. a person who either sits down and does the action manually, or invests time to script the approach in an environment that is relevant for their cluster / fabric network. This is the user / manual step that is being removed - by bundling a default implementation into the peer that understands enough to bring up CC pods in the cluster without manual intervention.

This is exactly why I am puzzled. Why do you consider the above cited text a problem? It's literally how things work everywhere. If you run stuff on Kubernetes, someone is responsible for defining what is to be run, and that someone is obviously a real person.
But why is that a problem? In the real world, you deploy chaincodes very rarely and once you do that - you can forget about them whether you run them as a service or in the old way.

Here, you are trying to run Fabric as a Kubernetes application, but you say that you want to control K8S (chaincode shim launching) from within Fabric peers.
Why would you do that? That means you need to supply to the Fabric peer filesystem sufficient credentials to interact with Kubernetes and let it spin up pods and what-not... then, if you ever need to change these credentials, you'll need to update them across all peers, how do you intend on doing that? And isn't that bad practice to put credentials of your infrastructure inside the application that runs inside your infrastructure?

[jkneubuh]

@yacovm I think you are projecting some misunderstanding of Kubernetes mechanics into this discussion. These are valid concerns related to credential management, RBAC, and permissions. Fortunately these mechanics are largely solved with sufficient rigor by the cloud-native community. What we are proposing is to leverage, rather than replicate, the Kube-approach towards process state management on a cluster. May I refer you to the docs on service accounts and Kube RBAC:

• Kubernetes RBAC Authorization
• Kubernetes Service Accounts

Again what is proposed here is to tie in to the natural lifecycle for managing chaincode, allowing the person in the loop to focus on the documented process for managing chaincode via the peer CLI, rather than build up or maintain additional expertise with Kubernetes.

[yacovm]

Fortunately these mechanics are largely solved with sufficient rigor by the cloud-native community.

May I refer you to the docs on service accounts and Kube RBAC:

rather than build up or maintain additional expertise with Kubernetes.

Don't you see a contradiction in these two statements? You expect people to know how to properly manage RBAC in K8S but at the same time you don't expect them to have expertise in it?

Again what is proposed here is to tie in to the natural lifecycle for managing chaincode

I don't see that as something natural at all. I actually think the chaincode as a service approach is the right approach for Fabric, even if it came very late. I know why and how the original chaincode lifecycle was designed, and I believe that it makes little sense that a Fabric peer needs to take care of running services when they're not even running inside of it.
It's true that you need more expertise to deploy and run the chaincode when it's running as a service, but in real environments you anyway run the peer as well, so it's not like you don't have this expertise.

[jt-nti]

Fyi, I just updated the k8s builder docs to take advantage of a prebuilt chaincode package, which eliminates the nasty .json nested tarball step I mentioned on the community call. This simplifies the chaincode deployment process even further if anyone wants to try it out on the k8s test network.

[jkneubuh]

Nice one - @jt-nti .... how does this impact / overlap with the discussions over in #3368 ? I agree it is a nice improvement to stop asking Fabric devs / admins to ever type "tar" ... again when working with chaincode.

[jt-nti]

It's related in that if the peer lifecycle chaincode package command was updated as @mbwhite suggests, so that it could create a k8s chaincode package, I would update the GitHub Action to use the peer command and delete the current hacky shell script :)

[jkneubuh]

@SamYuan1990

On the other hand the new k8s builder allows the peer to manage the lifecycle of the chaincode seamlessly using kubernetes..

Peer starts the CC process / Kubernetes pod
according to https://github.com/hyperledgendary/fabric-builder-k8s/blob/main/docs/TEST_NETWORK_K8S.md#kubernetes-permissions

what if user can't accept with this permission?

If the service account running the peers does not have permissions to create pods in Kubernetes, the options are:

• Don't run Fabric on Kubernetes.
• DIND / traditional lifecycle (CC process somehow runs in peer pod and communicates via RPC to peer)
• Use CCaaS (e.g. the Administrator - not the kube service account - launches the pod/deployment/service for the CC -- OR runs the CC service ... "somewhere in the cloud.")

For restricted Kube clusters that do NOT want to grant access for the service account to create pods :

Use CCaaS and have a Kube Administrator manage the CC / pod lifecycle.

[jt-nti]

Hopefully a restricted set of required permissions an the option of using a chaincode namespace should avoid too many complaints

[SamYuan1990]

@jkneubuh , @jt-nti in my point of view if we want to make a "Default External Builder". it is better to avoid change k8s permission if possible.

after "Default External Builder", we changed k8s permission if user can't accept with this permission then

• don't run Fabric on Kubernetes, against with our precondition.
• DIND / traditional lifecycle, why I need this "Default External Builder"? the default one is DIND.
• Use CCAAS, why I need this "Default External Builder"? I am already in use of CCAAS.

But, either DIND or CCAAS way makes the k8s builder not default option.
which means, if it just a builder, it works, but not DEFAULT one.

What are we talking about?

• a new external builder for k8s specific
• a DEFAULT external builder

p.s. when I talking with you guys on DEFAULT external builder, I am assuming that the binary for the new builder will release with peer image.

[jt-nti]

There is no concept of a default external builder in fabric and you will definitely not need to change any permissions if you don't want to use the k8s builder.

[SamYuan1990]

my most concern is from principle of least privilege for peer pod.
I suppose

• https://github.com/hyperledgendary/fabric-ccs-builder with CICD tools is following the principle.
• https://github.com/hyperledgendary/fabric-builder-k8s is not following the principle for now as it request additional permission.

if fabric-builder-k8s able to provide more features above fabric-ccs-builder without additional permission, that's cool.

[jkneubuh]

Hi all,

In the interest of moving this discussion and design forward, may I propose the following compromise:

1. An "external chaincode builder" will be prepared: fabric-builder-k8s.

2. fabric-builder-k8s will NOT be included in the default core.yaml, and will NOT be distributed into the hyperledger/peer Docker image.

3. fabric-builder-k8s will respond to the "Sykes lifecycle" events:
   - detect : activates for cc packages declared with type ccaas-k8s
   - build : No-Op
   - release : Extracts relevant CC package metadata and prepares peer for CCaaS connection
   - run : Creates K8s Pod and Services in Kubernetes, per permissions granted via the peer's service account

4. CC endpoints will be activated in "As a Service" (peer --> CC handshake) mode

5. User chaincode activation:

   • User compiles, builds, tags, and uploads the CC image offline / out of band.
   • peer CLI (or manual tar/gz) generates a CC package archive with metadata:
     • type : ccaas-k8s
     • image : container-registry/image-name:image-label
     • imagePullSecret: (optional) reference to k8s image pull secret
     • imagePullPolicy : (optional) Always | IfNotPresent | Never
     • labels : (optional) K/V labels
   • User runs standard CC lifecycle events via peer CLI

6. fabric-builder-k8s installation (Cluster agnostic):

   • K8s admin applies Role / Rolebinding to allow pod to create/edit/delete Pod, Service resources.
   • K8s admin runs a Job (or out of band process) to copy fabric-builder-k8s binaries to a persistent volume mount.
   • K8s admin will update peer core.yaml to register fabric-builder-k8s binaries located on the persistent volume mount.

7. fabric-builder-k8s installation (fabric-operator):

   • peer CRD types include a feature gate to enable fabric-builder-k8s
   • fabric-operator launches pods with sufficient Role / Rolebinding / Service accounts to manipulate Pods and Services
   • fabric-operator co-deploys peer pods with a sidecar container. The sidecar will perform the copy task to install the builder.
   • fabric-operator configures the peer core.yaml to activate the "ccaas-k8s" builder.

8. Users seeking "full control" over the CCaaS lifecycle (e.g. interactive debug, CI/CD customization, ...) may:

   • Use the ccaas "external chaincode builder"
   • Author a custom "external chaincode builder"

[yacovm]

CC endpoints will be activated in "As a Service" (peer --> CC handshake) mode

This is not possible with the current Fabric builder/launcher implementation. Until that changes, the chaincode will need to start in the traditional CC --> peer handshake mode.

Why not? What is the problem?

The peer does not call a builder's run binary when using the peer --> CC handshake (I think it could/should, and there are other opportunities to improve the builder/launcher implementation, but making those changes is not a priority)

If you change this file and add there: >&2 echo "The quick brown fox jumps over the lazy dog" and then run this test you will see in the output of the peer: [e][Org1.peer0] 2022-05-13 17:09:36.983 IDT 003d INFO [chaincode.externalbuilder.binary] waitForExit -> The quick brown fox jumps over the lazy dog command=detect

So it does call at least detect and I think also build.
Why do you need it to call run ? Can't you run the pod in the build stage if not exists or something like that?

[jt-nti]

The peer calls, detect, build, and release in the peer --> CC handshake case, but only run gets the chaincode.json

[yacovm]

ok but why do you need it to execute run ? You can just do all the things you need in the previous invocations can't you?

[jt-nti]

I think it would work if one of the previous phases had access to the chaincode_id but they don't. (Would also need the rest of the information from chaincode.json some other way, except the peer address.) On the other hand running with the traditional CC --> peer connection just works.

[yacovm]

Should be pretty easy to add:

diff --git a/core/container/externalbuilder/externalbuilder.go b/core/container/externalbuilder/externalbuilder.go
index deda3a12b..1ff049a1d 100644
--- a/core/container/externalbuilder/externalbuilder.go
+++ b/core/container/externalbuilder/externalbuilder.go
@@ -304,6 +304,7 @@ func (b *Builder) Build(buildContext *BuildContext) error {
        build := filepath.Join(b.Location, "bin", "build")
        cmd := b.NewCommand(build, buildContext.SourceDir, buildContext.MetadataDir, buildContext.BldDir)
 
+       cmd.Env = append(cmd.Env, fmt.Sprintf("CCID=%s", buildContext.CCID))
        err := b.runCommand(cmd)
        if err != nil {
                return errors.Wrapf(err, "external builder '%s' failed", b.Name)

But I guess it's also possible to add in the artifacts for the build no?