Skip to content

v0.0.37

Latest
Latest
Compare
Choose a tag to compare
@jlewitt1 jlewitt1 released this 12 Dec 00:41
· 39 commits to main since this release
v0.0.37
aa92087
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

Highlights

⚠️⚠️ Envs are no longer supported ⚠️⚠️

The Runhouse Env class (rh.Env or rh.env) is no longer supported. Instead, we introduce the concept of a process to handle running on a specific worker, and the Image replaces the default_env of a cluster, specifying any setup steps to take on the cluster.

Process

To specify env vars, compute, conda envs, or anything necessary to run on a specific process, you can create an process through the cluster.ensure_cluster_process() function

Instead of:

env = rh.env(name="my_env", reqs=["numpy, "pandas"], env_vars=MY_ENV_VARS, compute={"CPU": 0.5})
rh.function(local_fn).to(cluster, env=env)

You can do:

cluster.install_packages(["numpy, "pandas"])
process = cluster.ensure_process_created("my_process", env_vars=MY_ENV_VARS, compute={"CPU": 0.5})
rh.function(local_fn).to(cluster, process=process)

Image

Image is a new primitive that makes it easier to specify environment and setup steps to start the cluster with. It replaces the previous default_env cluster argument. It supports installing packages, running commands, creating a conda env, loading from a machine/docker image, and more.

my_image = (
    rh.Image(name="base_image")
    .setup_conda_env(
        conda_env_name="base_env",
        conda_yaml={"dependencies": ["python=3.11"], "name": "base_env"},
    )
    .install_packages(["numpy", "pandas"])
    .set_env_vars({"OMP_NUM_THREADS": 1})
)
cluster = rh.cluster("rh-cpu", instance_type="CPU:2+", image=my_image)
cluster.up_if_not()

Den Launcher

Now you can manage clusters (up, teardown, and status checks) via the Runhouse launcher / control plane. This is especially useful for cases where a local sky database would not be available, like upping a cluster in distributed workflows. Set launcher="den" in the cluster factory or include launcher: den in your local ~/.rh/config.yaml to use this feature.

You can also track cluster status, memory consumption, and view logs for Den launched clusters in the resources dashboard.

New Features

  • introduce DockerRegistrySecret for pulling from private or cloud provider registries

Updates

  • rename cluster launched_properties as compute_properties, and save generated internal and external ips in there
  • turn autosave to False by default, configurable in your rh config
  • allow cluster name to be passed into runhouse server cli commands
  • add rh cli alias

Bugfixes

  • cluster factory to properly handle differences when loading ints and autostop_mins

Examples

  • updated examples to remove env dependency, and follow the new process/Image flow
Assets 2
Loading