Deploying to a Cloud VM or Cloud Run · Deploying a GenAI App in Generative AI | Generative AI & LLMs

Two Common Options

	Cloud Run (serverless containers)	Cloud VM (Compute Engine, EC2)
You manage	only the container	OS, updates, process manager, Nginx
Scaling	automatic, down to zero	manual or with instance groups
Streaming	supported (raise the request timeout for long answers)	supported (turn off proxy buffering)
Good for	most API backends	GPUs for self-hosted models, long-running workers

Cloud Run

Deploy

Store the key in Secret Manager and deploy straight from source:

echo -n "sk-ant-..." | gcloud secrets create anthropic-key --data-file=-
gcloud run deploy shopease-assistant \
  --source . \
  --region asia-south1 \
  --set-secrets ANTHROPIC_API_KEY=anthropic-key:latest \
  --timeout 300 --concurrency 40 --min-instances 1

Example output

Building using Dockerfile and deploying container to Cloud Run service [shopease-assistant]
  OK Building and deploying... Done.
  OK Creating Revision...
  OK Routing traffic...
Service [shopease-assistant] revision [shopease-assistant-00001-kq7] has been deployed and is serving 100 percent of traffic.
Service URL: https://shopease-assistant-xxxxxxxx-el.a.run.app

asia-south1 is Mumbai, close to Indian users. --min-instances 1 avoids cold starts.

A VM with Nginx

server {
    listen 80;
    server_name assistant.shopease.in;
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_buffering off;          # needed for streaming
        proxy_read_timeout 300s;
    }
}

On the VM

sudo docker run -d --restart unless-stopped -p 127.0.0.1:8000:8000 --env-file /etc/shopease/assistant.env shopease-assistant
sudo nginx -t && sudo systemctl reload nginx
sudo certbot --nginx -d assistant.shopease.in

Example output

nginx: configuration file /etc/nginx/nginx.conf test is successful
Successfully deployed certificate for assistant.shopease.in