Browse Source

HUE-9192 [tools] Remove old impala helm charts

Romain 5 years ago
parent
commit
87a5cd37bb

+ 0 - 1
tools/kubernetes/helm/impala-engine/.helmignore

@@ -1 +0,0 @@
-.DS_Store

+ 0 - 5
tools/kubernetes/helm/impala-engine/Chart.yaml

@@ -1,5 +0,0 @@
-name: impala-engine
-description: Cloudera Data Warehouse Impala Engine
-version: 1.0.0
-appVersion: "3.1.0-cdh6.x"
-apiVersion: v1

+ 0 - 98
tools/kubernetes/helm/impala-engine/README.md

@@ -1,98 +0,0 @@
-# Impala Chart
-
-This is a MVP to get an Helm chart to get an Impala compute cluster ported to Kubernetes.
-It provides a basic Impala warehouses creation and autoscaling.
-
-All the interaction can be performed via native `kubectl` commands, or by interacting via the Frontend UI or REST API or command line.
-
-
-## Install
-
-To boot a 'finance' compute warehouse:
-
-```
-helm install impala-engine --set-string name=finance -n finance
-```
-
-Or copy impala-engine/values.yaml and edit and run:
-
-```
-helm install impala-engine -f values.yaml
-```
-
-E.g. by default with only 1 worker:
-
-```
-kubectl get pods
-NAME                                          READY   STATUS    RESTARTS   AGE
-impala-catalog-finance-0                      1/1     Running   0          5s
-impala-coordinator-finance-0                  1/1     Running   0          5s
-impala-statestore-finance-94c8c5f85-9hs9s     1/1     Running   0          5s
-impala-worker-finance-55df4576c8-67s8g        1/1     Running   0          5s
-```
-
-Note:
-
-Not tested yet with an external storage, so the mock storage (local HDFS/HMS) that can be installed via the
-[mock-storage](../mock-storage) chart is recommended.
-
-S3, HDFS, HMS minimal properties are configurable via [values.yaml](values.yaml).
-
-
-## Query
-
-Currently the querying can be done by going directly:
-
-* Opening-up the coordinator and pointing the shell to it
-
-```
-kubectl port-forward svc/impala-coordinator-finance 21000:21000 &
-impala-shell -i localhost
-```
-
-  (for connecting from a remote machine, need to upgrade kubectl to 13.1 to have binding option to --address 0.0.0.0. Note: NodePort might prevent more than one warehouse)
-* Log-in into a worker node and use the impala-shell which is there
-
-```
-kubectl exec -it impala-worker-finance-55df4576c8-67s8g bash
-impala-shell -i impala-coordinator-finance-0
-```
-
-* Install the Frontend chart and use the Web UI app
-
-## Autoscaling
-
-Can be done via native kubectl:
-
-```
-kubectl autoscale deployment impala-worker-finance --min=1 --max=3 --cpu-percent=80
-kubectl delete hpa impala-worker-finance
-```
-
-## Follow-ups
-
-There are a lot of possibilities to iterate on, e.g.:
-
-### All services
-
-* Do we want Services for all pods & ports? (probably only coordinators)
-* Hook-in to Tim's images (integrate basic graceful shutdown scripts...).
-* Iterate on naming convention. HDFS dependency removable
-* Could parameterize more some options (e.g. resource requests, pull image policy...)
-* Running as non root
-* Readiness Probes
-
-### Impala Server
-
-* Could split resource Request & Limit
-* Add HPA YAML config/parameters for autoscaling by default
-
-### Impala Catalog
-
-* Convert to Deployment
-* Need refactoring to get hive-site.xml information from properties
-
-### Impala Statestore
-
-* Convert to Deployment
-* Should be the same as Impala Server

+ 0 - 27
tools/kubernetes/helm/impala-engine/templates/NOTES.txt

@@ -1,27 +0,0 @@
-Congratulations, you've launched one Cloudera Data Warehouse Engine!
-
-To check the status of your installation run:
-
-  helm list {{ .Release.Name }}
-
-Also to double check that all the pods are 'READY':
-
-  kubectl get pods -l 'name in (impala-catalog-{{ .Values.name }},impala-statestore-{{ .Values.name }},impala-coordinator-{{ .Values.name }},impala-worker-{{ .Values.name }})'
-
-You should be able to execute queries via:
-
-  kubectl port-forward svc/impala-coordinator-{{ .Values.name }} 21000:21000 &
-  impala-shell -i localhost
-
-Or directly go into a worker node:
-
-  export WORKER_POD=$(kubectl get pods -l name=impala-worker-{{ .Values.name }} -o jsonpath="{.items[0].metadata.name"})
-  echo kubectl exec -it $WORKER_POD bash
-
-Then launch the shell:
-  
-  impala-shell -i impala-coordinator-{{ .Values.name }}
-
-And type some queries
-
-  SHOW TABLES;

+ 0 - 52
tools/kubernetes/helm/impala-engine/templates/configmap-impala.yaml

@@ -1,52 +0,0 @@
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: impala-config
-data:
-  core-site.xml: |
-    <?xml version="1.0"?>
-    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
-    <configuration>
-      <property>
-        <name>fs.defaultFS</name>
-        <value>hdfs://{{ .Values.hdfs.namenode.host }}:{{ .Values.hdfs.namenode.port }}</value>
-      </property>
-    </configuration>
-
-  hdfs-site.xml: |
-    <?xml version="1.0"?>
-    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
-    <configuration>
-      {{ if .Values.aws.accessKeyId }}
-      <property>
-        <name>fs.s3a.access.key</name>
-        <value>{{ .Values.aws.accessKeyId }}</value>
-      </property>
-      {{ end }}
-      {{ if .Values.aws.accessKeyId }}
-      <property>
-        <name>fs.s3a.secret.key</name>
-        <value>{{ .Values.aws.secretAccessKey }}</value>
-      </property>
-      {{ end }}
-      <property>
-        <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
-        <value>true</value>
-        <description>Needed for Impala</description>
-      </property>
-      <property>
-        <name>dfs.client.file-block-storage-locations.timeout.millis</name>
-        <value>10000</value>
-      </property>
-    </configuration>
-
-  hive-site.xml: |
-    <configuration>
-      <property>
-        <name>hive.metastore.uris</name>
-        <value>{{ .Values.metastore.uris }}</value>
-        <description>IP address (or fully-qualified domain name) and port of the metastore host</description>
-      </property>
-    </configuration>

+ 0 - 34
tools/kubernetes/helm/impala-engine/templates/deployment-statestore.yaml

@@ -1,34 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: impala-statestore-{{ .Values.name }}
-  labels:
-    name: impala-statestore-{{ .Values.name }}
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: impala-statestore-{{ .Values.name }}
-  template:
-    metadata:
-      labels:
-        name: impala-statestore-{{ .Values.name }}
-    spec:
-      containers:
-        - name: impala-statestore-{{ .Values.name }}
-          image: {{ .Values.registry }}/impala-statestore:{{ .Values.tag }}
-          imagePullPolicy: Always
-          resources:
-            requests:
-              memory: "512Mi"
-              cpu: "1000m"
-            limits:
-              memory: "512Mi"
-              cpu: "1000m"
-          ports:
-            - containerPort: 24000
-              name: im-ss
-              protocol: TCP
-            - containerPort: 25010
-              name: im-ss-web
-              protocol: TCP

+ 0 - 65
tools/kubernetes/helm/impala-engine/templates/deployment-worker.yaml

@@ -1,65 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: impala-worker-{{ .Values.name }}
-spec:
-  replicas: {{ .Values.worker.replicas }}
-  selector:
-    matchLabels:
-      name: impala-worker-{{ .Values.name }}
-  template:
-    metadata:
-      labels:
-        name: impala-worker-{{ .Values.name }}
-    spec:
-      hostname: impala-worker-{{ .Values.name }}
-      subdomain: impala-worker-{{ .Values.name }}
-      containers:
-        - name: impala-worker-{{ .Values.name }}
-          image: {{ .Values.registry }}/impala-server:{{ .Values.tag }}
-          command: ['bash', '-c', '/opt/hadoop/run-worker.sh']
-          imagePullPolicy: Always
-          resources:
-            requests:
-              memory: "{{ .Values.worker.memoryRequest }}"
-              cpu: "{{ .Values.worker.cpuRequest }}"
-            limits:
-              memory: "{{ .Values.worker.memoryRequest }}"
-              cpu: "{{ .Values.worker.cpuRequest }}"
-          ports:
-            - containerPort: 21000
-              name: im-sr
-              protocol: TCP
-            - containerPort: 22000
-              name: im-sr2
-              protocol: TCP
-            - containerPort: 23000
-              name: im-sr3
-              protocol: TCP
-            - containerPort: 25000
-              name: im-sr-web
-              protocol: TCP
-          env:
-            - name: IMPALA_STATESTORE
-              value: impala-statestore-{{ .Values.name }}
-            - name: IMPALA_CATALOG
-              value: impala-catalog-{{ .Values.name }}
-            - name: POD_NAME
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.name
-          lifecycle:
-           preStop:
-             exec:
-               command: ["/opt/hadoop/run-graceful-decommission.sh", "impala-coordinator-{{ .Values.name }}", "$POD_NAME"]
-          volumeMounts:
-            - name: config-volume
-              mountPath: /etc/alternatives/impala-conf/hdfs-site.xml
-              subPath: hdfs-site.xml
-            - name: config-volume
-              mountPath: /etc/alternatives/impala-conf/core-site.xml
-              subPath: core-site.xml
-      volumes:
-        - name: config-volume
-          configMap:
-            name: impala-config

+ 0 - 21
tools/kubernetes/helm/impala-engine/templates/service-catalog.yaml

@@ -1,21 +0,0 @@
-kind: Service
-apiVersion: v1
-metadata:
-  name: impala-catalog-{{ .Values.name }}
-spec:
-  clusterIP: None
-  ports:
-    - port: 26000
-      targetPort: 26000
-      name: im-ct
-      protocol: TCP
-    - port: 24000
-      targetPort: 24000
-      name: im-ss
-      protocol: TCP
-    - port: 25020
-      targetPort: 25020
-      name: im-ct-web
-      protocol: TCP
-  selector:
-    name: impala-catalog-{{ .Values.name }}

+ 0 - 24
tools/kubernetes/helm/impala-engine/templates/service-coordinator.yaml

@@ -1,24 +0,0 @@
-kind: Service
-apiVersion: v1
-metadata:
-  name: impala-coordinator-{{ .Values.name }}
-spec:
-  clusterIP: None
-  ports:
-    - port: 21000
-      name: im-sr
-      protocol: TCP
-    - port: 21050
-      name: thrift
-      protocol: TCP
-    - port: 22000
-      name: state-heartbeat
-      protocol: TCP
-    - port: 23000
-      name: state-heartbeat-topic
-      protocol: TCP
-    - port: 25000
-      name: im-sr-web
-      protocol: TCP
-  selector:
-    name: impala-coordinator-{{ .Values.name }}

+ 0 - 16
tools/kubernetes/helm/impala-engine/templates/service-statestore.yaml

@@ -1,16 +0,0 @@
-kind: Service
-apiVersion: v1
-metadata:
-  name: impala-statestore-{{ .Values.name }}
-spec:
-  ports:
-    - port: 24000
-      targetPort: 24000
-      name: im-ss
-      protocol: TCP
-    - port: 25010
-      targetPort: 25010
-      name: im-ss-web
-      protocol: TCP
-  selector:
-    name: impala-statestore-{{ .Values.name }}

+ 0 - 22
tools/kubernetes/helm/impala-engine/templates/service-worker.yaml

@@ -1,22 +0,0 @@
-kind: Service
-apiVersion: v1
-metadata:
-  name: impala-worker-{{ .Values.name }}
-spec:
-  clusterIP: None
-  ports:
-    - port: 21000
-      targetPort: 21000
-      name: im-sr
-      protocol: TCP
-    - port: 21050
-      name: thrift
-      protocol: TCP
-    - port: 22000
-      name: state-heartbeat
-      protocol: TCP
-    - port: 23000
-      name: state-heartbeat-topic
-      protocol: TCP
-  selector:
-    name: impala-worker-{{ .Values.name }}

+ 0 - 56
tools/kubernetes/helm/impala-engine/templates/statefulset-coordinator.yaml

@@ -1,56 +0,0 @@
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
-  name: impala-coordinator-{{ .Values.name }}
-spec:
-  serviceName: impala-coordinator-{{ .Values.name }}
-  replicas: 1
-  selector:
-    matchLabels:
-      name: impala-coordinator-{{ .Values.name }}
-  template:
-    metadata:
-      labels:
-        name: impala-coordinator-{{ .Values.name }}
-    spec:
-      containers:
-        - name: impala-coordinator-{{ .Values.name }}
-          image: {{ .Values.registry }}/impala-server:{{ .Values.tag }}
-          imagePullPolicy: Always
-          command: ['bash', '-c', '/opt/hadoop/run-coordinator.sh']
-          resources:
-            requests:
-              memory: "4096Mi"
-              cpu: "2000m"
-            limits:
-              memory: "4096Mi"
-              cpu: "2000m"
-          ports:
-            - containerPort: 21000
-              name: im-sr
-              protocol: TCP
-            - containerPort: 22000
-              name: im-sr2x
-              protocol: TCP
-            - containerPort: 23000
-              name: im-sr3
-              protocol: TCP
-            - containerPort: 25000
-              name: im-sr-web
-              protocol: TCP
-          env:
-            - name: IMPALA_STATESTORE
-              value: impala-statestore-{{ .Values.name }}
-            - name: IMPALA_CATALOG
-              value: impala-catalog-{{ .Values.name }}
-          volumeMounts:
-            - name: config-volume
-              mountPath: /etc/alternatives/impala-conf/hdfs-site.xml
-              subPath: hdfs-site.xml
-            - name: config-volume
-              mountPath: /etc/alternatives/impala-conf/core-site.xml
-              subPath: core-site.xml
-      volumes:
-        - name: config-volume
-          configMap:
-            name: impala-config

+ 0 - 53
tools/kubernetes/helm/impala-engine/templates/statefulset-statestore.yaml

@@ -1,53 +0,0 @@
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
-  name: impala-catalog-{{ .Values.name }}
-spec:
-  serviceName: impala-catalog-{{ .Values.name }}
-  replicas: 1
-  selector:
-    matchLabels:
-      name: impala-catalog-{{ .Values.name }}
-  template:
-    metadata:
-      labels:
-        name: impala-catalog-{{ .Values.name }}
-    spec:
-      containers:
-        - name: impala-catalog-{{ .Values.name }}
-          image: {{ .Values.registry }}/impala-catalog:{{ .Values.tag }}
-          imagePullPolicy: Always
-          resources:
-            requests:
-              memory: "1024Mi"
-              cpu: "1500m"
-            limits:
-              memory: "1024Mi"
-              cpu: "1500m"
-          ports:
-            - containerPort: 26000
-              name: im-ct
-              protocol: TCP
-            - containerPort: 24000
-              name: im-ss
-              protocol: TCP
-            - containerPort: 25020
-              name: im-ct-web
-              protocol: TCP
-          env:
-          - name: IMPALA_STATESTORE
-            value: impala-statestore-{{ .Values.name }}
-          volumeMounts:
-            - name: config-volume
-              mountPath: /etc/alternatives/impala-conf/hdfs-site.xml
-              subPath: hdfs-site.xml
-            - name: config-volume
-              mountPath: /etc/alternatives/impala-conf/core-site.xml
-              subPath: core-site.xml
-            - name: config-volume
-              mountPath: /etc/alternatives/impala-conf/hive-site.xml
-              subPath: hive-site.xml
-      volumes:
-        - name: config-volume
-          configMap:
-            name: impala-config

+ 0 - 17
tools/kubernetes/helm/impala-engine/values.yaml

@@ -1,17 +0,0 @@
-registry: "docker-registry.infra.cloudera.com/cloudera/datawarehouse"
-tag: "v2"
-name: "datawarehouse"
-worker:
-  cpuRequest: "1000m"
-  memoryRequest: "2048Mi"
-  replicas: 1
-metastore:
-  uris: thrift://hive:9083
-hdfs:
-  namenode:
-    host: "hdfs-namenode"
-    port: 8020
-aws:
-  accessKeyId: ""
-  secretAccessKey: ""
-  region: "us-east-1"

+ 0 - 1
tools/kubernetes/helm/mock-storage/.helmignore

@@ -1 +0,0 @@
-.DS_Store

+ 0 - 5
tools/kubernetes/helm/mock-storage/Chart.yaml

@@ -1,5 +0,0 @@
-name: mock-storage
-description: Cloudera Data Warehouse storage
-version: 1.0.0
-appVersion: "1.0"
-apiVersion: v1

+ 0 - 27
tools/kubernetes/helm/mock-storage/README.md

@@ -1,27 +0,0 @@
-# Mock Storage Chart
-
-This chart schedules a mini HDFS and Hive to run inside the K8s cluster.
-
-## Install
-
-To boot a the Web UI:
-
-```
-helm install mock-storage
-```
-
-Or copy [values.yaml](values.yaml) and edit and run:
-
-```
-helm install mock-storage -f values.yaml
-```
-
-And now you will get:
-
-```
-kubectl get pods
-NAME                                          READY   STATUS    RESTARTS   AGE
-hdfs-datanode-1-0                             1/1     Running   0          13d
-hdfs-namenode-1-0                             1/1     Running   0          13d
-hive-697f49cdbd-ghr6w                         2/2     Running   0          13d
-```

+ 0 - 83
tools/kubernetes/helm/mock-storage/templates/hdfs.yaml

@@ -1,83 +0,0 @@
-kind: Service
-apiVersion: v1
-metadata:
-  name: hdfs-namenode
-spec:
-  ports:
-    - port: 9820
-      targetPort: 9820
-      protocol: TCP
-      name: nm-rpc
-    - port: 9870
-      targetPort: 9870
-      protocol: TCP
-      name: nm-web-ui
-  selector:
-    name: hdfs-namenode-{{ .Values.hdfs.nametag }}
----
-kind: StatefulSet
-apiVersion: apps/v1beta1
-metadata:
-  name: hdfs-namenode-{{ .Values.hdfs.nametag }}
-spec:
-  serviceName: hdfs-namenode-{{ .Values.hdfs.nametag }}
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        name: hdfs-namenode-{{ .Values.hdfs.nametag }}
-    spec:
-#      initContainers:
-#        - name: hdfs-namenode-format-{{ .Values.hdfs.nametag }}
-#          image: kubernetes:5000/cloudera/hdfs-namenode:{{ .Values.hdfs.tag }}
-#          imagePullPolicy: Always
-#          command: ['bash', '-c', 'if [ ! -f /dfs/nn/current/VERSION ]; then java -Dnn.hostname=`hostname` org.apache.hadoop.hdfs.server.namenode.NameNode -format; fi']
-#          volumeMounts:
-#            - name: hdfs-namenode
-#              mountPath: /dfs/nn
-      containers:
-        - name: hdfs-namenode-{{ .Values.hdfs.nametag }}
-          image: {{ .Values.registry }}/hdfs-namenode:{{ .Values.hdfs.tag }}
-          imagePullPolicy: Always
-          ports:
-              - containerPort: 9820
-                protocol: TCP
-                name: nm-rpc
-              - containerPort: 9870
-                protocol: TCP
-                name: nm-web-ui
-#          volumeMounts:
-#            - name: hdfs-namenode
-#              mountPath: /dfs/nn
-#      volumes:
-#        - name: hdfs-namenode
-#          hostPath:
-#            path: /tmp/dfs/nn-{{ .Values.hdfs.nametag }}
----
-kind: StatefulSet
-apiVersion: apps/v1beta1
-metadata:
-  name: hdfs-datanode-{{ .Values.hdfs.nametag }}
-spec:
-  serviceName: hdfs-datanode-{{ .Values.hdfs.nametag }}
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        name: hdfs-datanode-{{ .Values.hdfs.nametag }}
-    spec:
-      containers:
-        - name: hdfs-datanode-{{ .Values.hdfs.nametag }}
-          image: {{ .Values.registry }}/hdfs-datanode:{{ .Values.hdfs.tag }}
-          imagePullPolicy: Always
-          ports:
-          - containerPort: 9864
-            protocol: TCP
-            name: dn-web-ui
-          volumeMounts:
-            - name: hdfs-datanode
-              mountPath: /dfs/dn
-      volumes:
-        - name: hdfs-datanode
-          hostPath:
-            path: /tmp/dfs/dn-{{ .Values.hdfs.nametag }}

+ 0 - 94
tools/kubernetes/helm/mock-storage/templates/hive.yaml

@@ -1,94 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-  name: hive-metastore-db-secrets
-type: Opaque
-data:
-  hive-metastore-db-root-password: {{ .Values.hive.metastore_db_root_password | b64enc }}
-  hive-metastore-db-user: {{ .Values.hive.metastore_db_user | b64enc }}
-  hive-metastore-db-password: {{ .Values.hive.metastore_db_password | b64enc }}
-  hive-metastore-db-name: {{ .Values.hive.metastore_db_name | b64enc }}
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: hive
-spec:
-  selector:
-    hadoop: hive
-  ports:
-  - protocol: TCP
-    port: 3306
-    targetPort: 3306
-    name: a
-  - protocol: TCP
-    port: 9083
-    targetPort: 9083
-    name: b
----
-apiVersion: extensions/v1beta1
-kind: Deployment
-metadata:
-  labels:
-    hadoop: hive
-  name: hive
-  namespace: default
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      hadoop: hive
-  template:
-    metadata:
-      labels:
-        hadoop: hive
-    spec:
-      restartPolicy: Always
-      containers:
-      - image: mysql:5.7
-        imagePullPolicy: Always
-        name: mysql
-        resources: {}
-        env:
-        - name: MYSQL_ROOT_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: hive-metastore-db-secrets
-              key: hive-metastore-db-root-password
-        - name: MYSQL_USER
-          value: metastore
-        - name: MYSQL_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: hive-metastore-db-secrets
-              key: hive-metastore-db-password
-        - name: MYSQL_DATABASE
-          valueFrom:
-            secretKeyRef:
-              name: hive-metastore-db-secrets
-              key: hive-metastore-db-name
-      - image: {{ .Values.registry }}/hive-server:{{ .Values.hive.tag }}
-        imagePullPolicy: Always
-        name: hive-server
-        resources: {}
-        command: ['bash', '-c', '/opt/hive/init.sh; hive --service metastore --hiveconf hive.root.logger=INFO,console']
-        env:
-        - name: MYSQL_USER
-          valueFrom:
-            secretKeyRef:
-              name: hive-metastore-db-secrets
-              key: hive-metastore-db-user
-        - name: MYSQL_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: hive-metastore-db-secrets
-              key: hive-metastore-db-password
-        - name: MYSQL_DATABASE
-          valueFrom:
-            secretKeyRef:
-              name: hive-metastore-db-secrets
-              key: hive-metastore-db-name
-      volumes:
-        - name: hive-metastore-mysql
-          hostPath:
-            path: /tmp/hive-metastore-mysql2

+ 0 - 10
tools/kubernetes/helm/mock-storage/values.yaml

@@ -1,10 +0,0 @@
-registry: "docker-registry.infra.cloudera.com/cloudera/datawarehouse"
-hdfs:
-  tag: "v2"
-  nametag: "1"
-hive:
-  tag: "v2"
-  metastore_db_name: "metastore"
-  metastore_db_user: "root"
-  metastore_db_password: "hive"
-  metastore_db_root_password: "hive"