|
|
@@ -1,98 +0,0 @@
|
|
|
-# Impala Chart
|
|
|
-
|
|
|
-This is a MVP to get an Helm chart to get an Impala compute cluster ported to Kubernetes.
|
|
|
-It provides a basic Impala warehouses creation and autoscaling.
|
|
|
-
|
|
|
-All the interaction can be performed via native `kubectl` commands, or by interacting via the Frontend UI or REST API or command line.
|
|
|
-
|
|
|
-
|
|
|
-## Install
|
|
|
-
|
|
|
-To boot a 'finance' compute warehouse:
|
|
|
-
|
|
|
-```
|
|
|
-helm install impala-engine --set-string name=finance -n finance
|
|
|
-```
|
|
|
-
|
|
|
-Or copy impala-engine/values.yaml and edit and run:
|
|
|
-
|
|
|
-```
|
|
|
-helm install impala-engine -f values.yaml
|
|
|
-```
|
|
|
-
|
|
|
-E.g. by default with only 1 worker:
|
|
|
-
|
|
|
-```
|
|
|
-kubectl get pods
|
|
|
-NAME READY STATUS RESTARTS AGE
|
|
|
-impala-catalog-finance-0 1/1 Running 0 5s
|
|
|
-impala-coordinator-finance-0 1/1 Running 0 5s
|
|
|
-impala-statestore-finance-94c8c5f85-9hs9s 1/1 Running 0 5s
|
|
|
-impala-worker-finance-55df4576c8-67s8g 1/1 Running 0 5s
|
|
|
-```
|
|
|
-
|
|
|
-Note:
|
|
|
-
|
|
|
-Not tested yet with an external storage, so the mock storage (local HDFS/HMS) that can be installed via the
|
|
|
-[mock-storage](../mock-storage) chart is recommended.
|
|
|
-
|
|
|
-S3, HDFS, HMS minimal properties are configurable via [values.yaml](values.yaml).
|
|
|
-
|
|
|
-
|
|
|
-## Query
|
|
|
-
|
|
|
-Currently the querying can be done by going directly:
|
|
|
-
|
|
|
-* Opening-up the coordinator and pointing the shell to it
|
|
|
-
|
|
|
-```
|
|
|
-kubectl port-forward svc/impala-coordinator-finance 21000:21000 &
|
|
|
-impala-shell -i localhost
|
|
|
-```
|
|
|
-
|
|
|
- (for connecting from a remote machine, need to upgrade kubectl to 13.1 to have binding option to --address 0.0.0.0. Note: NodePort might prevent more than one warehouse)
|
|
|
-* Log-in into a worker node and use the impala-shell which is there
|
|
|
-
|
|
|
-```
|
|
|
-kubectl exec -it impala-worker-finance-55df4576c8-67s8g bash
|
|
|
-impala-shell -i impala-coordinator-finance-0
|
|
|
-```
|
|
|
-
|
|
|
-* Install the Frontend chart and use the Web UI app
|
|
|
-
|
|
|
-## Autoscaling
|
|
|
-
|
|
|
-Can be done via native kubectl:
|
|
|
-
|
|
|
-```
|
|
|
-kubectl autoscale deployment impala-worker-finance --min=1 --max=3 --cpu-percent=80
|
|
|
-kubectl delete hpa impala-worker-finance
|
|
|
-```
|
|
|
-
|
|
|
-## Follow-ups
|
|
|
-
|
|
|
-There are a lot of possibilities to iterate on, e.g.:
|
|
|
-
|
|
|
-### All services
|
|
|
-
|
|
|
-* Do we want Services for all pods & ports? (probably only coordinators)
|
|
|
-* Hook-in to Tim's images (integrate basic graceful shutdown scripts...).
|
|
|
-* Iterate on naming convention. HDFS dependency removable
|
|
|
-* Could parameterize more some options (e.g. resource requests, pull image policy...)
|
|
|
-* Running as non root
|
|
|
-* Readiness Probes
|
|
|
-
|
|
|
-### Impala Server
|
|
|
-
|
|
|
-* Could split resource Request & Limit
|
|
|
-* Add HPA YAML config/parameters for autoscaling by default
|
|
|
-
|
|
|
-### Impala Catalog
|
|
|
-
|
|
|
-* Convert to Deployment
|
|
|
-* Need refactoring to get hive-site.xml information from properties
|
|
|
-
|
|
|
-### Impala Statestore
|
|
|
-
|
|
|
-* Convert to Deployment
|
|
|
-* Should be the same as Impala Server
|