Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker build for salad and salad_app.py to setup environment #204

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ad-astra-video
Copy link
Collaborator

This PR adds a special docker build and salad_app.py to automatically download the model when the container starts up and then start the API.

The Dockerfile.salad is a pretty simple approach to customize the livepeer runner container to enable auto model downloading and startup after download by using the livepeer/ai-runner:base tag by default (or will use --build-arg BASE_IMAGE=[image tag] set on launch of docker build). SAM2 would start with the livepeer/ai-runner:segment-anything-2 tag in BASE_IMAGE.

Salad Container Group Setup

  1. Set the Image Source to the docker image built using the Dockerfile.salad included in this PR.
  2. Select CPU and memory for the runners. Suggest 4cpu and 8gb ram minimum. 1 cpu may work but have not tested this yet. Ram may need to be increased for some models.
  3. Choose replica count and GPUs needed. Size the GPUs based on the model served, if do not know, start with 4090, 3090ti and 3090. Multiple GPU types can be selected to increase up time of containers.
  4. Select 50GB disk
  5. Enable Container Gateway. The url provided after enabling this will be the url used with the external container in aiModels.json.
  6. Setup external logging service. I could not get Axiom to work. I had better results with NewRelic.
  7. Setup environment variables applicable to pipeline and model (see screenshot below). Host must be :: to enable the salad gateway to allocate requests to the port.

Example Environment Variables
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant