Romain Rigaux dcca94e573 HUE-8724 [k8s] Provide how to run Hue server in Kubernetes 6 жил өмнө
..
templates 2dcf14ef7c HUE-8699 [core] Adding skeleton of Hue on Kubernetes 6 жил өмнө
.helmignore 2dcf14ef7c HUE-8699 [core] Adding skeleton of Hue on Kubernetes 6 жил өмнө
Chart.yaml 2dcf14ef7c HUE-8699 [core] Adding skeleton of Hue on Kubernetes 6 жил өмнө
README.md dcca94e573 HUE-8724 [k8s] Provide how to run Hue server in Kubernetes 6 жил өмнө
values.yaml 2dcf14ef7c HUE-8699 [core] Adding skeleton of Hue on Kubernetes 6 жил өмнө

README.md

Impala Chart

This is a MVP to get an Helm chart to get an Impala compute cluster ported to Kubernetes. It provides a basic Impala warehouses creation and autoscaling.

All the interaction can be performed via native kubectl commands, or by interacting via the Frontend UI or REST API or command line.

Install

To boot a 'finance' compute warehouse:

helm install impala-engine --set-string name=finance -n finance

Or copy impala-engine/values.yaml and edit and run:

helm install impala-engine -f values.yaml

E.g. by default with only 1 worker:

kubectl get pods
NAME                                          READY   STATUS    RESTARTS   AGE
impala-catalog-finance-0                      1/1     Running   0          5s
impala-coordinator-finance-0                  1/1     Running   0          5s
impala-statestore-finance-94c8c5f85-9hs9s     1/1     Running   0          5s
impala-worker-finance-55df4576c8-67s8g        1/1     Running   0          5s

Note:

Not tested yet with an external storage, so the mock storage (local HDFS/HMS) that can be installed via the mock-storage chart is recommended.

S3, HDFS, HMS minimal properties are configurable via values.yaml.

Query

Currently the querying can be done by going directly:

  • Opening-up the coordinator and pointing the shell to it

    kubectl port-forward svc/impala-coordinator-finance 21000:21000 &
    impala-shell -i localhost
    

(for connecting from a remote machine, need to upgrade kubectl to 13.1 to have binding option to --address 0.0.0.0. Note: NodePort might prevent more than one warehouse)

  • Log-in into a worker node and use the impala-shell which is there

    kubectl exec -it impala-worker-finance-55df4576c8-67s8g bash
    impala-shell -i impala-coordinator-finance-0
    
  • Install the Frontend chart and use the Web UI app

Autoscaling

Can be done via native kubectl:

kubectl autoscale deployment impala-worker-finance --min=1 --max=3 --cpu-percent=80
kubectl delete hpa impala-worker-finance

Follow-ups

There are a lot of possibilities to iterate on, e.g.:

All services

  • Do we want Services for all pods & ports? (probably only coordinators)
  • Hook-in to Tim's images (integrate basic graceful shutdown scripts...).
  • Iterate on naming convention. HDFS dependency removable
  • Could parameterize more some options (e.g. resource requests, pull image policy...)
  • Running as non root
  • Readiness Probes

Impala Server

  • Could split resource Request & Limit
  • Add HPA YAML config/parameters for autoscaling by default

Impala Catalog

  • Convert to Deployment
  • Need refactoring to get hive-site.xml information from properties

Impala Statestore

  • Convert to Deployment
  • Should be the same as Impala Server