Tools for setup-free CircuitPython builds

[timchinowsky]

Setting up tooling for CircuitPython builds, particularly for ESP32, can be confusing. This PR contains tools which enable setup-free containerized building of any CP port with the one-liner sudo tools/docker-build/build <port> <board-name> <option>.

There is doubtless a lot of room for improvement in these scripts, in particular in the way file ownership is fixed up at the end of build, but they seem to work, and I thought I'd share in case you're interested.

[dhalbert]

Thanks for doing this. It's a one-stop shop for someone who doesn't want to set up a build enviornment.

I don't know how docker works. Does the Dockerfile run on every invocation, or are you just including it to show how you built timchinowsky/build_circuitpython?

Rather than put this into the circuitpython repo, I am thinking that it could be left as a repo that you control, since you control the docker image anyway. Otherwise we become ultimately responsible for maintaining it. We could add documentation for it in the circuitpython repo and/or the Building CircuitPython guide.

[dhalbert]

This gets the requirements files from the tip of main, but I think you want to get them from the repo?

[timchinowsky]

The Dockerfile is only used to build the image. I'm including it so that folks can see what it is doing (basically the same steps you would need to do on the command line to install the tools) and be empowered to create their own images.

[timchinowsky]

This gets the requirements files from the tip of main, but I think you want to get them from the repo?

You mean the repo that CP is being built from? Can't do that because the image is given access to that repo at runtime, but the requirements need to be available at image build time so that the python built into the image can be configured. If you wanted to do it the other way, you'd need to have python pull in the requirements every time it is asked to build CP.

[dhalbert]

Is it typical that docker usage runs things as root? It seems like the whole build will be run as root in the docker container.

[timchinowsky]

Yeah this is one of the things that could probably be improved. The container doesn't have to be run with sudo but the processes inside the container run as root so the owner of anything created by the container ends up as root which makes git unhappy. The reason I'm having it run as sudo is so that I can patch this up after the fact and change the owner back to who it should be.

[timchinowsky]

This changes ownership of the files created by the container back to the owner of the repo (as cribbed from the metadata of Makefile

[bill88t]

I also do not think this belongs in the repo.

Also the current approach, other than completely breaking permissions (cough cough, add --user $UID:$(id -g)), cannot do specific languages, which sometimes we need to test flash size.

I feel like having it separate would be better since then it could work as is with a local clone of the repo, but it could also be told to just clone it (or a fork, specific commit, etc.) on the fly and discard everything other than the output folder.

[timchinowsky]

I also do not think this belongs in the repo.

Also the current approach, other than completely breaking permissions (cough cough, add --user $UID:$(id -g)), cannot do specific languages, which sometimes we need to test flash size.

I feel like having it separate would be better since then it could work as is with a local clone of the repo, but it could also be told to just clone it (or a fork, specific commit, etc.) on the fly and discard everything other than the output folder.

Yeah, thanks, I figured --user was somewhere on the path to righteousness, I'll give it a whirl...

If all that languages requires is e.g. TRANSLATION=es then that should work as is, like sudo tools/docker-build/build atmel-samd grandcentral_m4_express TRANSLATION=es but I should tack on some more arguments so you can add DEBUG=1 etc. as well.

[timchinowsky]

Thanks for doing this. It's a one-stop shop for someone who doesn't want to set up a build enviornment.

I don't know how docker works. Does the Dockerfile run on every invocation, or are you just including it to show how you built timchinowsky/build_circuitpython?

Rather than put this into the circuitpython repo, I am thinking that it could be left as a repo that you control, since you control the docker image anyway. Otherwise we become ultimately responsible for maintaining it. We could add documentation for it in the circuitpython repo and/or the Building CircuitPython guide.

Makes sense, thanks!

[bablokb]

How does this compare to the scripts in .devcontainer? In the end, the Github codespaces are also docker containers, but the casual developer does need to have a working docker runtime environment installed on their system.

[jepler]

github codespaces are a proprietary development environment hosted in the cloud with free access limited based on hours per month. This feature seems to be something that a user can run locally on their own hardware. These can both be important considerations for developers.

If both are ultimately "made of" docker images, it would be nice if the code for creating the development environment could be shared between them, but I don't know enough to say how feasible this is.

[jepler]

(maybe it's possible for devcontainer to be run locally? I don't actually know anything about them or use them; I inferred from your comment, possibly incorrectly, that devcontainer ONLY works with github codespaces)

[bablokb]

My point is a bit different: the scripts in .devcontainer are just simple shell scripts. When you create a docker-container, you can run anything. That is what is going on behind the scenes of Github codespaces.

So the question is if the existing scripts can be modified in a way that allows them to run outside of the Github codespace environment in your own docker containers. I think only a few changes should be necessary (e.g. the scripts currently hardcode REPO_ROOT but they could be changed to pick up this variable from the environment if already set).

The ultimate goal would be not to have multiple sets of scripts for the setup of a development environment, but just a single set, making future changes easier.

[bablokb]

Oh, and I just had a short read here: https://code.visualstudio.com/docs/devcontainers/containers

So it seems the existing stuff should also run locally. I will put this on my TODO list and try if this works.

It is still not that simple for casual developers, because they need a local docker environment, but setting this up is a one time task so that should be possible.

[timchinowsky]

Thanks for the discussion! When I wrote this I wanted to make a tool that (1) exemplified and expanded on the published build procedure while making it easier and (2) provided an example of how Docker can be used to make all sorts of jobs like this less painful (at least if you run Linux). I looked at the CI scripts and decided that something simpler might be more helpful to others.