Browse Source

HUE-8730 [docker] Refactor and organize the image building

Romain Rigaux 6 năm trước cách đây
mục cha
commit
89589f7710

+ 2 - 2
README.md

@@ -46,8 +46,8 @@ Start Hue in a single click with the [Docker Guide](https://github.com/cloudera/
 
 Kubernetes
 ----------
-One Helm chart to start a Hue server is available at ``tools/kubernetes/helm/frontend``. The Kubernetes [hue.yaml](tools/kubernetes/helm/frontend/templates/hue.yaml)
-configuration is available in the template directoy.
+Configurations to start a Hue server are available at [``tools/kubernetes``](tools/kubernetes/). The Kubernetes [hue.yaml](tools/kubernetes/helm/hue/templates/hue.yaml)
+configuration is available in the template directory.
 
 
 Community

+ 2 - 0
desktop/conf/pseudo-distributed.ini.tmpl

@@ -942,6 +942,7 @@
     #   ## If 'user' and 'password' are omitted, they will be prompted in the UI.
     #   ## Option 'impersonation_property' used to configure outbound impersonation, e.g. "impersonation_property": "hive.server2.proxy.user".
     #   options='{"url": "jdbc:mysql://localhost:3306/hue", "driver": "com.mysql.jdbc.Driver", "user": "root", "password": "root", "impersonation_property": ""}'
+
     # [[[presto]]]
     #   name=Presto SQL
     #   interface=presto
@@ -957,6 +958,7 @@
     #   ## The JDBC driver clickhouse-jdbc.jar and its related jars need to be in the CLASSPATH environment variable.
     #   options='{"url": "jdbc:clickhouse://localhost:8123", "driver": "ru.yandex.clickhouse.ClickHouseDriver", "user": "readonly", "password": ""}'
 
+
 ###########################################################################
 # Settings to configure your Analytics Dashboards
 ###########################################################################

+ 21 - 7
package-lock.json

@@ -4628,12 +4628,14 @@
         "balanced-match": {
           "version": "1.0.0",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "brace-expansion": {
           "version": "1.1.11",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "balanced-match": "^1.0.0",
             "concat-map": "0.0.1"
@@ -4648,17 +4650,20 @@
         "code-point-at": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "concat-map": {
           "version": "0.0.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "console-control-strings": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "core-util-is": {
           "version": "1.0.2",
@@ -4775,7 +4780,8 @@
         "inherits": {
           "version": "2.0.3",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "ini": {
           "version": "1.3.5",
@@ -4787,6 +4793,7 @@
           "version": "1.0.0",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "number-is-nan": "^1.0.0"
           }
@@ -4801,6 +4808,7 @@
           "version": "3.0.4",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "brace-expansion": "^1.1.7"
           }
@@ -4808,12 +4816,14 @@
         "minimist": {
           "version": "0.0.8",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "minipass": {
           "version": "2.3.5",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "safe-buffer": "^5.1.2",
             "yallist": "^3.0.0"
@@ -4832,6 +4842,7 @@
           "version": "0.5.1",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "minimist": "0.0.8"
           }
@@ -4912,7 +4923,8 @@
         "number-is-nan": {
           "version": "1.0.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "object-assign": {
           "version": "4.1.1",
@@ -4924,6 +4936,7 @@
           "version": "1.4.0",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "wrappy": "1"
           }
@@ -5045,6 +5058,7 @@
           "version": "1.0.2",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "code-point-at": "^1.0.0",
             "is-fullwidth-code-point": "^1.0.0",

+ 5 - 68
tools/docker/README.md

@@ -1,80 +1,17 @@
 ![alt text](https://raw.githubusercontent.com/cloudera/hue/master/docs/images/hue_logo.png "Hue Logo")
 
-
 # Welcome to the repository for the Hue Dockerfiles
 
-
-Hue is an open source Web interface for analyzing data with any Apache Hadoop: [gethue.com](http://gethue.com)
-
-Here you can find the Dockerfiles for Hue and Livy, the Spark REST server.
-
-You can catch us on [Docker Hub](https://hub.docker.com/u/gethue/) as well.
+Availalable on [Docker Hub](https://hub.docker.com/u/gethue/) as well.
 
 [![DockerPulls](https://img.shields.io/docker/pulls/gethue/hue.svg)](https://registry.hub.docker.com/u/gethue/hue/)
 [![DockerStars](https://img.shields.io/docker/stars/gethue/hue.svg)](https://registry.hub.docker.com/u/gethue/hue/)
 
-## Get the docker image
-
-Just pull the latest from the Internet or build it yourself from the Hue repository.
-
-### Pull the image from Docker Hub
-```
-sudo docker pull gethue/hue:latest
-```
-
-### Build the image
-```
-cd tools/docker/hue-base
-sudo docker build --rm -t gethue/hue:latest .
-```
-
-## Running the image
-```
-docker run -it -p 8888:8888 gethue/hue:latest bash
-```
-This opens a bash to the root of the project. From there you can run the development version of Hue with the command
-
-```
-./build/env/bin/hue runserver_plus 0.0.0.0:8888
-```
-
-or
-
-## Running with docker-compose
-```
-cd tools/docker/hue-base
-cp docker-compose.yml.sample docker-compose.yml
-cp ../../../desktop/conf.dist/hue.ini .
-docker-compose up -d
-docker exec -ti development_hue bash
-```
-
-Hue should then be up and running on your default Docker IP on the port 8888, so usually [http://192.168.99.100:8888](http://192.168.99.100:8888).
-
-**Note**
-If 192.168.99.100 does not work, get the IP of the docker container with:
-```
-sudo docker ps
-CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                            NAMES
-b7950388c1db        gethue/hue:latest   "bash"              10 minutes ago      Up 10 minutes       22/tcp, 0.0.0.0:8888->8888/tcp   agitated_mccarthy
-```
+## Services
 
-Then get ``inet addr``, so in our case [http://172.17.0.1:8888](http://172.17.0.1:8888):
-```
-sudo docker exec -it b7950388c1db /sbin/ifconfig eth0
-eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:01
-          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
-          inet6 addr: fe80::42:acff:fe11:1/64 Scope:Link
-          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
-          RX packets:67 errors:0 dropped:0 overruns:0 frame:0
-          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
-          collisions:0 txqueuelen:0
-          RX bytes:10626 (10.6 KB)  TX bytes:648 (648.0 B)
-```
+* [Hue](hue)
 
-## Next
 
-You can then configure Hue and start using it!
+## Kubernetes
 
-Read more about configuring Hue on our [blog](http://gethue.com/how-to-configure-hue-in-your-hadoop-cluster/).
-The development version uses the configuration file [``desktop/conf/pseudo-distributed.ini``.](/desktop/conf/pseudo-distributed.ini).
+See [tools/kubernetes](/tools/kubernetes).

+ 0 - 2
tools/docker/hue-base/.gitignore

@@ -1,2 +0,0 @@
-docker-compose.yml
-hue.ini

+ 0 - 43
tools/docker/hue-base/Dockerfile

@@ -1,43 +0,0 @@
-# Welcome to the official Hue (http://gethue.com) developer Dockerfile
-FROM ubuntu:trusty
-MAINTAINER The Hue Team "https://github.com/cloudera/hue"
-
-RUN apt-get update -y
-
-RUN apt-get install -y software-properties-common
-RUN add-apt-repository -y ppa:webupd8team/java
-RUN apt-get update -y
-
-RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections
-RUN apt-get install -y oracle-java8-installer
-
-RUN apt-get update -y
-RUN apt-get install --fix-missing -q -y \
-  git \
-  ant \
-  gcc \
-  g++ \
-  libkrb5-dev \
-  libmysqlclient-dev \
-  libssl-dev \
-  libsasl2-dev \
-  libsasl2-modules-gssapi-mit \
-  libsqlite3-dev \
-  libtidy-0.99-0 \
-  libxml2-dev \
-  libxslt-dev \
-  libffi-dev \
-  make \
-  maven \
-  libldap2-dev \
-  python-dev \
-  python-setuptools \
-  libgmp3-dev \
-  libz-dev
-
-RUN git clone https://github.com/cloudera/hue.git
-WORKDIR hue
-RUN make apps
-EXPOSE 8888
-VOLUME /hue/desktop/
-CMD ["build/env/bin/hue", "runserver_plus", "0.0.0.0:8888"]

+ 0 - 13
tools/docker/hue-base/docker-compose.yml.sample

@@ -1,13 +0,0 @@
-cloudera:
-    image: gethue/hue:latest
-    hostname: hue
-    container_name: development_hue
-    dns: 8.8.8.8
-    ports:
-     - "8888:8888"
-    #command: ./build/env/bin/hue runserver_plus 0.0.0.0:8888
-    command: tail -f /dev/null
-    volumes:
-      - ../../../apps/hbase/src:/hue/apps/hbase/src
-      - ./hue.ini:/hue/desktop/conf/hue.ini
-

+ 14 - 6
tools/kubernetes/services/hue/Dockerfile → tools/docker/hue/Dockerfile

@@ -1,4 +1,8 @@
+# Welcome to Hue (http://gethue.com) Dockerfile
+# Build an image from a remote github or local cloned Hue repository.
+
 FROM ubuntu:16.04
+MAINTAINER Hue Project "https://github.com/cloudera/hue"
 
 RUN apt-get update -y && apt-get install -y \
   build-essential \
@@ -29,13 +33,17 @@ RUN apt-get update -y
 RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections
 RUN apt-get install -y oracle-java8-installer
 
-ADD hue /hue
+ADD . /hue
+
 RUN cd /hue && PREFIX=/usr/share make install
-COPY conf /usr/share/hue/desktop/conf
-EXPOSE 8888
 RUN useradd -ms /bin/bash hue && chown -R hue /usr/share/hue
+
+COPY tools/docker/hue/conf /usr/share/hue/desktop/conf
+COPY tools/docker/hue/startup.sh /usr/share/hue
+
 WORKDIR  /usr/share/hue
 RUN ./build/env/bin/pip install psycopg2-binary
-# dev tools
-COPY ./startup.sh /usr/share/hue
-CMD ["./startup.sh"]
+
+
+EXPOSE 8888
+CMD ["./startup.sh"]

+ 112 - 0
tools/docker/hue/README.md

@@ -0,0 +1,112 @@
+# Hue Server
+
+
+## Get the docker image
+
+Just pull the latest from the Internet or build it yourself from the Hue repository.
+
+
+### Pull the image from Docker Hub
+```
+sudo docker pull gethue/hue:latest
+```
+
+### Build the image
+
+Directly from Github source:
+
+```
+sudo docker build https://github.com/cloudera/hue.git#master -t hue -f tools/docker/hue/Dockerfile
+```
+
+Or from a cloned local Hue:
+
+```
+sudo docker build . -t hue -f tools/docker/hue/Dockerfile
+```
+
+**Note**
+
+Feel free to replace `-t hue` in all the commands by your own docker repository and image tag, e.g. `gethue/hue:latest`
+
+**Tag and push the image to the container registry**
+
+```
+docker build . -t docker-registry.gethue.com/gethue/hue:v4.4
+docker push docker-registry.gethue.com/gethue/hue:v4.4
+```
+
+
+## Run the image
+
+### Docker
+
+Directly boot the image:
+
+```
+docker run -it -p 8888:8888 gethue/hue:latest
+```
+
+Hue should then be up and running on your default Docker IP on the port 8888, so usually [http://192.168.99.100:8888](http://192.168.99.100:8888).
+
+
+#### Configuration
+
+By default the Hue container is using 
+[``tools/docker/hue/conf/defaults.ini``.](/tools/docker/hue/conf/defaults.ini) on top of [``desktop/conf/hue.ini``.](/desktop/conf/hue.ini)
+which assumes localhost for all the data services.
+
+In order to point to the services, use the simplified ini [``hue.ini``.](/tools/docker/hue/hue.ini), edit the values before starting it via:
+
+```
+docker run -it -p 8888:8888 -v $PWD/tools/docker/hue/hue.ini:/usr/share/hue/desktop/conf/z-hue.ini gethue/hue
+```
+
+or copy the full one:
+
+```
+cp /desktop/conf.dist/hue.ini .
+
+docker run -it -p 8888:8888 -v $PWD/hue.ini:/usr/share/hue/desktop/conf/z-hue.ini gethue/hue
+```
+
+Read more about configuring Hue on the [blog](http://gethue.com/how-to-configure-hue-in-your-hadoop-cluster/).
+
+
+#### Docker Compose
+
+Assuming we have a local ``hue.ini`` as shown in the previous section:
+
+```
+cd tools/docker/hue
+cp docker-compose.yml.sample docker-compose.yml
+```
+
+Then:
+
+```
+docker-compose up -d
+```
+
+
+**Note**
+If 192.168.99.100 does not work, get the IP of the docker container with:
+```
+sudo docker ps
+CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                            NAMES
+b7950388c1db        gethue/hue:latest   "bash"              10 minutes ago      Up 10 minutes       22/tcp, 0.0.0.0:8888->8888/tcp   agitated_mccarthy
+```
+
+Then get ``inet addr``, so in our case [http://172.17.0.1:8888](http://172.17.0.1:8888):
+```
+sudo docker exec -it b7950388c1db /sbin/ifconfig eth0
+eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:01
+          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
+          inet6 addr: fe80::42:acff:fe11:1/64 Scope:Link
+          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+          RX packets:67 errors:0 dropped:0 overruns:0 frame:0
+          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
+          collisions:0 txqueuelen:0
+          RX bytes:10626 (10.6 KB)  TX bytes:648 (648.0 B)
+```
+

+ 0 - 0
tools/kubernetes/services/hue/conf/log.conf → tools/docker/hue/conf/log.conf


+ 89 - 91
tools/kubernetes/services/hue/conf/hue.ini → tools/docker/hue/conf/z-defaults.ini

@@ -15,9 +15,6 @@
 ###########################################################################
 
 [desktop]
-  # Custom flags for Cloudera DW
-  is_multicluster_only=true
-  is_k8s_only=true
 
   # Set this to a random string, the longer the better.
   # This is used for secure hashing in the session store.
@@ -629,12 +626,14 @@
     # Note for Oracle, options={"threaded":true} must be set in order to avoid crashes.
     # Note for Oracle, you can use the Oracle Service Name by setting "host=" and "port=" and then "name=<host>:<port>/<service_name>".
     # Note for MariaDB use the 'mysql' engine.
-    engine=postgresql_psycopg2
-    host=hue-postgres
-    port=5432
-    user=hue
-    password=hue
-    name=hue
+
+    # engine=postgresql_psycopg2
+    # host=hue-postgres
+    # port=5432
+    # user=hue
+    # password=hue
+    # name=hue
+
     # conn_max_age option to make database connection persistent value in seconds
     # https://docs.djangoproject.com/en/1.9/ref/databases/#persistent-connections
     ## conn_max_age=0
@@ -783,19 +782,6 @@
    ## collection_interval=30000
 
 
-[[clusters]]
-    [[[engines]]]
-      type=altus-engines
-      hostname=...
-      auth_key_id=...
-      auth_key_secret=...
-      
-    [[[AltusV2]]]
-      type=altusv2
-      hostname=...
-      auth_key_id=...
-      auth_key_secret=...
-
 ###########################################################################
 # Settings to configure the snippets available in the Notebook
 ###########################################################################
@@ -839,11 +825,11 @@
   [[interpreters]]
     # Define the name and how to connect and execute the language.
 
-    # [[[hive]]]
-    #   # The name of the snippet.
-    #   name=Hive
-    #   # The backend connection to use to communicate with the server.
-    #   interface=hiveserver2
+    [[[hive]]]
+      # The name of the snippet.
+      name=Hive
+      # The backend connection to use to communicate with the server.
+      interface=hiveserver2
 
     [[[impala]]]
       name=Impala
@@ -853,83 +839,83 @@
     #   name=SparkSql
     #   interface=hiveserver2
 
-    # [[[spark]]]
-    #  name=Scala
-    #  interface=livy
+    [[[spark]]]
+      name=Scala
+      interface=livy
 
-    # [[[pyspark]]]
-    #  name=PySpark
-    #  interface=livy
+    [[[pyspark]]]
+      name=PySpark
+      interface=livy
 
-    # [[[r]]]
-    #  name=R
-    #  interface=livy
+    [[[r]]]
+      name=R
+      interface=livy
 
-    # [[[jar]]]
-    # name=Spark Submit Jar
-    # interface=livy-batch
+    [[[jar]]]
+      name=Spark Submit Jar
+      interface=livy-batch
 
-    # [[[py]]]
-    # name=Spark Submit Python
-    # interface=livy-batch
+    [[[py]]]
+      name=Spark Submit Python
+      interface=livy-batch
 
-    # [[[text]]]
-    # name=Text
-    # interface=text
+    [[[text]]]
+      name=Text
+      interface=text
 
-    # [[[markdown]]]
-    #  name=Markdown
-    #  interface=text
+    [[[markdown]]]
+      name=Markdown
+      interface=text
 
-    # [[[mysql]]]
-    #  name = MySQL
-    #  interface=rdbms
+    [[[mysql]]]
+      name = MySQL
+      interface=rdbms
 
-    # [[[sqlite]]]
-    #  name = SQLite
-    #  interface=rdbms
+    [[[sqlite]]]
+      name = SQLite
+      interface=rdbms
 
-    # [[[postgresql]]]
-    #  name = PostgreSQL
-    #  interface=rdbms
+    [[[postgresql]]]
+      name = PostgreSQL
+      interface=rdbms
 
-    # [[[oracle]]]
-    #  name = Oracle
-    #  interface=rdbms
+    [[[oracle]]]
+      name = Oracle
+      interface=rdbms
 
-    # [[[solr]]]
-    #  name = Solr SQL
-    #  interface=solr
-    #  ## Name of the collection handler
-    #  # options='{"collection": "default"}'
+    [[[solr]]]
+      name = Solr SQL
+      interface=solr
+      ## Name of the collection handler
+      # options='{"collection": "default"}'
 
-    # [[[pig]]]
-    #  name=Pig
-    #  interface=oozie
+    [[[pig]]]
+      name=Pig
+      interface=oozie
 
-    # [[[java]]]
-    #  name=Java
-    #  interface=oozie
+    [[[java]]]
+      name=Java
+      interface=oozie
 
-    # [[[spark2]]]
-    #  name=Spark
-    #  interface=oozie
+    [[[spark2]]]
+      name=Spark
+      interface=oozie
 
-    # [[[mapreduce]]]
-    #  name=MapReduce
-    #  interface=oozie
+    [[[mapreduce]]]
+      name=MapReduce
+      interface=oozie
 
-    # [[[sqoop1]]]
-    #  name=Sqoop1
-    #  interface=oozie
+    [[[sqoop1]]]
+      name=Sqoop1
+      interface=oozie
 
-    # [[[distcp]]]
-    #  name=Distcp
-    #  interface=oozie
+    [[[distcp]]]
+      name=Distcp
+      interface=oozie
 
-    # [[[shell]]]
-    #  name=Shell
-    #  interface=oozie
+    [[[shell]]]
+      name=Shell
+      interface=oozie
 
     # [[[mysql]]]
     #   name=MySql JDBC
@@ -940,6 +926,21 @@
     #   ## Option 'impersonation_property' used to configure outbound impersonation, e.g. "impersonation_property": "hive.server2.proxy.user".
     #   options='{"url": "jdbc:mysql://localhost:3306/hue", "driver": "com.mysql.jdbc.Driver", "user": "root", "password": "root", "impersonation_property": ""}'
 
+    # [[[presto]]]
+    #   name=Presto SQL
+    #   interface=presto
+    #   ## Specific options for connecting to the Presto server.
+    #   ## The JDBC driver presto-jdbc.jar need to be in the CLASSPATH environment variable.
+    #   ## If 'user' and 'password' are omitted, they will be prompted in the UI.
+    #   options='{"url": "jdbc:presto://localhost:8080/catalog/schema", "driver": "io.prestosql.jdbc.PrestoDriver", "user": "root", "password": "root"}'
+
+    # [[[clickhouse]]]
+    #   name=ClickHouse
+    #   interface=jdbc
+    #   ## Specific options for connecting to the ClickHouse server.
+    #   ## The JDBC driver clickhouse-jdbc.jar and its related jars need to be in the CLASSPATH environment variable.
+    #   options='{"url": "jdbc:clickhouse://localhost:8123", "driver": "ru.yandex.clickhouse.ClickHouseDriver", "user": "readonly", "password": ""}'
+
 
 ###########################################################################
 # Settings to configure your Analytics Dashboards
@@ -951,7 +952,7 @@
   ## is_enabled=true
 
   # Activate the SQL Dashboard (beta).
-  ## has_sql_enabled=false
+  has_sql_enabled=true
 
   # Activate the Query Builder (beta).
   ## has_query_builder_enabled=false
@@ -1025,7 +1026,7 @@
       ## resourcemanager_port=8032
 
       # Whether to submit jobs to this cluster
-      submit_to=False
+      submit_to=True
 
       # Resource Manager logical name (required for HA)
       ## logical_name=
@@ -1834,9 +1835,6 @@
 ###########################################################################
 
 [metadata]
-  # TODO: shoud come from Helm
-  [[k8s]]
-  api_url=http://provisioner.107.178.211.187.nip.io
 
   [[optimizer]]
     # Hostname to Optimizer API or compatible service.

+ 9 - 0
tools/docker/hue/docker-compose.yml.sample

@@ -0,0 +1,9 @@
+cloudera:
+    image: gethue/hue:latest
+    hostname: hue
+    container_name: hue
+    dns: 8.8.8.8
+    ports:
+     - "8888:8888"
+    volumes:
+      - ./hue.ini:/hue/desktop/conf/hue.ini

+ 450 - 0
tools/docker/hue/hue.ini

@@ -0,0 +1,450 @@
+# Lightweight Hue configuration file
+# ==================================
+
+[desktop]
+
+  # Set this to a random string, the longer the better.
+  secret_key=kasdlfjknasdfl3hbaksk3bwkasdfkasdfba23asdf
+
+  # Webserver listens on this address and port
+  http_host=0.0.0.0
+  http_port=8888
+
+  # Time zone name
+  time_zone=America/Los_Angeles
+
+  # Configuration options for specifying the Desktop Database. For more info,
+  # see http://docs.djangoproject.com/en/1.11/ref/settings/#database-engine
+  # ------------------------------------------------------------------------
+  [[database]]
+    # Database engine is typically one of:
+    # postgresql_psycopg2, mysql, sqlite3 or oracle.
+    #
+    # Note that for sqlite3, 'name', below is a path to the filename. For other backends, it is the database name
+    # Note for Oracle, options={"threaded":true} must be set in order to avoid crashes.
+    # Note for Oracle, you can use the Oracle Service Name by setting "host=" and "port=" and then "name=<host>:<port>/<service_name>".
+    # Note for MariaDB use the 'mysql' engine.
+
+    # engine=postgresql_psycopg2
+    # host=hue-postgres
+    # port=5432
+    # user=hue
+    # password=hue
+    # name=hue
+
+###########################################################################
+# Settings to configure the snippets available in the Notebook
+###########################################################################
+
+[notebook]
+
+  # One entry for each type of snippet.
+  [[interpreters]]
+    # Define the name and how to connect and execute the language.
+
+    [[[hive]]]
+      # The name of the snippet.
+      name=Hive
+      # The backend connection to use to communicate with the server.
+      interface=hiveserver2
+
+    [[[impala]]]
+      name=Impala
+      interface=hiveserver2
+
+    # [[[sparksql]]]
+    #   name=SparkSql
+    #   interface=hiveserver2
+
+    [[[spark]]]
+      name=Scala
+      interface=livy
+
+    [[[pyspark]]]
+      name=PySpark
+      interface=livy
+
+    [[[r]]]
+      name=R
+      interface=livy
+
+    [[[jar]]]
+      name=Spark Submit Jar
+      interface=livy-batch
+
+    [[[py]]]
+      name=Spark Submit Python
+      interface=livy-batch
+
+    [[[text]]]
+      name=Text
+      interface=text
+
+    [[[markdown]]]
+      name=Markdown
+      interface=text
+
+    [[[mysql]]]
+      name = MySQL
+      interface=rdbms
+
+    [[[sqlite]]]
+      name = SQLite
+      interface=rdbms
+
+    [[[postgresql]]]
+      name = PostgreSQL
+      interface=rdbms
+
+    [[[oracle]]]
+      name = Oracle
+      interface=rdbms
+
+    [[[solr]]]
+      name = Solr SQL
+      interface=solr
+      ## Name of the collection handler
+      # options='{"collection": "default"}'
+
+    [[[pig]]]
+      name=Pig
+      interface=oozie
+
+    [[[java]]]
+      name=Java
+      interface=oozie
+
+    [[[spark2]]]
+      name=Spark
+      interface=oozie
+
+    [[[mapreduce]]]
+      name=MapReduce
+      interface=oozie
+
+    [[[sqoop1]]]
+      name=Sqoop1
+      interface=oozie
+
+    [[[distcp]]]
+      name=Distcp
+      interface=oozie
+
+    [[[shell]]]
+      name=Shell
+      interface=oozie
+
+    # [[[mysql]]]
+    #   name=MySql JDBC
+    #   interface=jdbc
+    #   ## Specific options for connecting to the server.
+    #   ## The JDBC connectors, e.g. mysql.jar, need to be in the CLASSPATH environment variable.
+    #   ## If 'user' and 'password' are omitted, they will be prompted in the UI.
+    #   ## Option 'impersonation_property' used to configure outbound impersonation, e.g. "impersonation_property": "hive.server2.proxy.user".
+    #   options='{"url": "jdbc:mysql://localhost:3306/hue", "driver": "com.mysql.jdbc.Driver", "user": "root", "password": "root", "impersonation_property": ""}'
+
+    # [[[presto]]]
+    #   name=Presto SQL
+    #   interface=presto
+    #   ## Specific options for connecting to the Presto server.
+    #   ## The JDBC driver presto-jdbc.jar need to be in the CLASSPATH environment variable.
+    #   ## If 'user' and 'password' are omitted, they will be prompted in the UI.
+    #   options='{"url": "jdbc:presto://localhost:8080/catalog/schema", "driver": "io.prestosql.jdbc.PrestoDriver", "user": "root", "password": "root"}'
+
+    # [[[clickhouse]]]
+    #   name=ClickHouse
+    #   interface=jdbc
+    #   ## Specific options for connecting to the ClickHouse server.
+    #   ## The JDBC driver clickhouse-jdbc.jar and its related jars need to be in the CLASSPATH environment variable.
+    #   options='{"url": "jdbc:clickhouse://localhost:8123", "driver": "ru.yandex.clickhouse.ClickHouseDriver", "user": "readonly", "password": ""}'
+
+
+[hadoop]
+
+  # Configuration for HDFS NameNode
+  # ------------------------------------------------------------------------
+  [[hdfs_clusters]]
+    # HA support by using HttpFs
+
+    [[[default]]]
+      # Enter the filesystem uri
+      fs_defaultfs=hdfs://localhost:8020
+
+      # Use WebHdfs/HttpFs as the communication mechanism.
+      # Domain should be the NameNode or HttpFs host.
+      # Default port is 14000 for HttpFs.
+      ## webhdfs_url=http://localhost:50070/webhdfs/v1
+
+  # Configuration for YARN (MR2)
+  # ------------------------------------------------------------------------
+  [[yarn_clusters]]
+
+    [[[default]]]
+      # Enter the host on which you are running the ResourceManager
+      ## resourcemanager_host=localhost
+
+      # The port where the ResourceManager IPC listens on
+      ## resourcemanager_port=8032
+
+
+      # URL of the ResourceManager API
+      ## resourcemanager_api_url=http://localhost:8088
+
+      # URL of the ProxyServer API
+      ## proxy_api_url=http://localhost:8088
+
+      # URL of the HistoryServer API
+      ## history_server_api_url=http://localhost:19888
+
+      # URL of the Spark History Server
+      ## spark_history_server_url=http://localhost:18088
+
+
+###########################################################################
+# Settings to configure Beeswax with Hive
+###########################################################################
+
+[beeswax]
+
+  # Host where HiveServer2 is running.
+  # If Kerberos security is enabled, use fully-qualified domain name (FQDN).
+  ## hive_server_host=localhost
+
+  # Port where HiveServer2 Thrift server runs on.
+  ## hive_server_port=10000
+
+
+###########################################################################
+# Settings to configure Impala
+###########################################################################
+
+[impala]
+  # Host of the Impala Server (one of the Impalad)
+  ## server_host=localhost
+
+  # Port of the Impala Server
+  ## server_port=21050
+
+
+###########################################################################
+# Settings to configure the Spark application.
+###########################################################################
+
+[spark]
+  # The Livy Server URL.
+  ## livy_server_url=http://localhost:8998
+
+  # Configure Livy to start in local 'process' mode, or 'yarn' workers.
+  ## livy_server_session_kind=yarn
+
+  # Whether Livy requires client to perform Kerberos authentication.
+  ## security_enabled=false
+
+  # Host of the Sql Server
+  ## sql_server_host=localhost
+
+  # Port of the Sql Server
+  ## sql_server_port=10000
+
+  # Choose whether Hue should validate certificates received from the server.
+  ## ssl_cert_ca_verify=true
+
+
+###########################################################################
+# Settings to configure HBase Browser
+###########################################################################
+
+[hbase]
+  # Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'.
+  ## hbase_clusters=(Cluster|localhost:9090)
+
+
+###########################################################################
+# Settings to configure Solr Search
+###########################################################################
+
+[search]
+
+  # URL of the Solr Server
+  ## solr_url=http://localhost:8983/solr/
+
+
+###########################################################################
+# Settings to configure liboozie
+###########################################################################
+
+[liboozie]
+  # The URL where the Oozie service runs on. This is required in order for
+  # users to submit jobs. Empty value disables the config check.
+  ## oozie_url=http://localhost:11000/oozie
+
+
+###########################################################################
+# Settings for the AWS lib
+###########################################################################
+
+[aws]
+  [[aws_accounts]]
+    # Default AWS account
+    ## [[[default]]]
+      # AWS credentials
+      ## access_key_id=
+      ## secret_access_key=
+      ## security_token=
+
+      # Execute this script to produce the AWS access key ID.
+      ## access_key_id_script=/path/access_key_id.sh
+
+      # Execute this script to produce the AWS secret access key.
+      ## secret_access_key_script=/path/secret_access_key.sh
+
+      # Allow to use either environment variables or
+      # EC2 InstanceProfile to retrieve AWS credentials.
+      ## allow_environment_credentials=yes
+
+      # AWS region to use, if no region is specified, will attempt to connect to standard s3.amazonaws.com endpoint
+      ## region=us-east-1
+
+      # Endpoint overrides
+      ## host=
+
+      # Proxy address and port
+      ## proxy_address=
+      ## proxy_port=8080
+      ## proxy_user=
+      ## proxy_pass=
+
+      # Secure connections are the default, but this can be explicitly overridden:
+      ## is_secure=true
+
+
+###########################################################################
+# Settings for the Azure lib
+###########################################################################
+[azure]
+  [[azure_accounts]]
+    # Default Azure account
+    [[[default]]]
+      # Azure credentials
+      ## client_id=
+      # Execute this script to produce the ADLS client id.
+      ## client_id_script=/path/client_id.sh
+      ## client_secret=
+      # Execute this script to produce the ADLS client secret.
+      ## client_secret_script=/path/client_secret.sh
+      ## tenant_id=
+      # Execute this script to produce the ADLS tenant id.
+      ## tenant_id_script=/path/tenant_id.sh
+
+  [[adls_clusters]]
+    # Default ADLS cluster
+    [[[default]]]
+      ## fs_defaultfs=adl://<account_name>.azuredatalakestore.net
+      ## webhdfs_url=https://<account_name>.azuredatalakestore.net/webhdfs/v1
+
+
+###########################################################################
+# Settings to configure the ZooKeeper Lib
+###########################################################################
+
+[libzookeeper]
+  # ZooKeeper ensemble. Comma separated list of Host/Port.
+  # e.g. localhost:2181,localhost:2182,localhost:2183
+  ## ensemble=localhost:2181
+
+
+###########################################################################
+# Settings for the RDBMS application
+###########################################################################
+
+[librdbms]
+  # The RDBMS app can have any number of databases configured in the databases
+  # section. A database is known by its section name
+  # (IE sqlite, mysql, psql, and oracle in the list below).
+
+  [[databases]]
+    # sqlite configuration.
+    ## [[[sqlite]]]
+      # Name to show in the UI.
+      ## nice_name=SQLite
+
+      # For SQLite, name defines the path to the database.
+      ## name=/tmp/sqlite.db
+
+      # Database backend to use.
+      ## engine=sqlite
+
+      # Database options to send to the server when connecting.
+      # https://docs.djangoproject.com/en/1.4/ref/databases/
+      ## options={}
+
+    # mysql, oracle, or postgresql configuration.
+    ## [[[mysql]]]
+      # Name to show in the UI.
+      ## nice_name="My SQL DB"
+
+      # For MySQL and PostgreSQL, name is the name of the database.
+      # For Oracle, Name is instance of the Oracle server. For express edition
+      # this is 'xe' by default.
+      ## name=mysqldb
+
+      # Database backend to use. This can be:
+      # 1. mysql
+      # 2. postgresql
+      # 3. oracle
+      ## engine=mysql
+
+      # IP or hostname of the database to connect to.
+      ## host=localhost
+
+      # Port the database server is listening to. Defaults are:
+      # 1. MySQL: 3306
+      # 2. PostgreSQL: 5432
+      # 3. Oracle Express Edition: 1521
+      ## port=3306
+
+      # Username to authenticate with when connecting to the database.
+      ## user=example
+
+      # Password matching the username to authenticate with when
+      # connecting to the database.
+      ## password=example
+
+      # Database options to send to the server when connecting.
+      # https://docs.djangoproject.com/en/1.4/ref/databases/
+      ## options={}
+
+
+###########################################################################
+# Settings to configure Kafka
+###########################################################################
+
+[kafka]
+
+  [[kafka]]
+    # Enable the Kafka integration.
+    ## is_enabled=false
+
+    # Base URL of Kafka REST API.
+    ## api_url=http://localhost:8082
+
+
+###########################################################################
+# Settings to configure Metadata
+###########################################################################
+
+[metadata]
+
+  [[navigator]]
+    # Navigator API URL (without version suffix).
+    ## api_url=http://localhost:7187/api
+
+    # Which authentication to use: CM or external via LDAP or SAML.
+    ## navmetadataserver_auth_type=CMDB
+
+    # Username of the CM user used for authentication.
+    ## navmetadataserver_cmdb_user=hue
+    # CM password of the user used for authentication.
+    ## navmetadataserver_cmdb_password=
+    # Execute this script to produce the CM password. This will be used when the plain password is not set.
+    # navmetadataserver_cmdb_password_script=

+ 1 - 1
tools/kubernetes/services/hue/startup.sh → tools/docker/hue/startup.sh

@@ -2,4 +2,4 @@
 
 ./build/env/bin/hue syncdb --noinput
 ./build/env/bin/hue migrate
-./build/env/bin/supervisor
+./build/env/bin/supervisor

+ 0 - 42
tools/docker/livy/Dockerfile

@@ -1,42 +0,0 @@
-# Welcome to the official Livy (http://gethue.com) developer Dockerfile
-FROM ubuntu:trusty
-MAINTAINER The Hue Team "https://github.com/cloudera/hue"
-
-RUN apt-get install -y software-properties-common
-RUN add-apt-repository -y ppa:webupd8team/java
-RUN apt-get update -y
-
-RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections
-RUN apt-get install -y oracle-java8-installer
-
-RUN apt-get install -q -y \
-  git \
-  maven \
-  curl \
-  m4 \
-  ruby \
-  texinfo \
-  libbz2-dev \
-  libcurl4-openssl-dev \
-  libexpat-dev \
-  libncurses-dev \
-  zlib1g-dev
-
-RUN useradd -ms /bin/bash ellie
-
-USER ellie
-WORKDIR /home/ellie
-RUN yes |ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/linuxbrew/go/install)"
-ENV PATH="/home/ellie/.linuxbrew/bin:$PATH"
-ENV SPARK_HOME="/home/ellie/.linuxbrew/Cellar/apache-spark/1.5.1"
-RUN brew update
-RUN brew install apache-spark
-
-RUN git clone https://github.com/cloudera/hue.git
-
-WORKDIR hue/apps/spark/java
-
-RUN mvn clean install -DskipTests
-
-EXPOSE 8998
-CMD ["./bin/livy-server"]

+ 11 - 2
tools/kubernetes/README.md

@@ -10,10 +10,19 @@ Assuming you have a Kubernetes cluster configured with Helm installed and images
 * [Helm](helm)
    * [Hue](helm/hue)
 * YAML
+   * Hue (TBD)
    * NGINX (TBD)
    * Task Server (TBD)
-* Containers
-   * [Hue](services/hue)
+* [Container Images](/tools/docker)
+   * [Hue](/tools/docker/hue)
+
+## Hue
+
+Update the Hue pods:
+
+```
+kubectl delete pods `kubectl get pods | grep hue | grep -v postgres | cut -d" " -f1`
+```
 
 ## K8s Cluster
 

+ 1 - 1
tools/kubernetes/helm/hue/README.md

@@ -17,7 +17,7 @@ helm install hue -f values.yaml
 
 And follow-up the instructions printed on the screen.
 
-E.g. by default it comes with the UI, a temporary DB and a REST API:
+E.g. by default it comes with a temporary DB and ingres load balancer:
 
 ```
 kubectl get pods

+ 0 - 17
tools/kubernetes/services/hue/README.md

@@ -1,17 +0,0 @@
-# Hue Docker
-
-TODO: unify with current [Hue docker](/tools/docker).
-
-
-Build and push:
-
-```
-sudo docker build . -t gethue/hue
-sudo docker push gethue/hue
-```
-
-Update the Hue pods:
-
-```
-kubectl delete pods `kubectl get pods | grep hue | grep -v postgres | cut -d" " -f1`
-```