2014-07-24-tutorial-how-to-run-the-hue-integration-tests.md 8.1 KB


title: 'Tutorial: How to run the Hue unit and integration tests?' author: admin type: post date: 2014-07-24T17:47:08+00:00 url: /tutorial-how-to-run-the-hue-integration-tests/ sf_thumbnail_type:

  • none sf_thumbnail_link_type:
  • link_to_post sf_detail_type:
  • none sf_page_title:
  • 1 sf_page_title_style:
  • standard sf_no_breadcrumbs:
  • 1 sf_page_title_bg:
  • none sf_page_title_text_style:
  • light sf_background_image_size:
  • cover sf_social_sharing:
  • 1 sf_sidebar_config:
  • left-sidebar sf_left_sidebar:
  • Sidebar-2 sf_right_sidebar:
  • Sidebar-1 sf_caption_position:
  • caption-right sf_remove_promo_bar:
  • 1 slide_template:
  • default categories:
  • Development

After the Review Board post, here is another tutorial about becoming an awesome Hue developer: how to run the Hue tests!

Hue has two types of tests:

  • unit tests
  • integration tests

First, clone the Hue repository and make sure that you have all the pre-requisite packages:

{{< highlight bash >}}git clone git@github.com:cloudera/hue.git{{< /highlight >}}

Unit Tests

The regular unit tests do not require all this setup! Just run them directly:

{{< highlight bash >}}build/env/bin/hue test specific beeswax.tests:test_split_statements &> a; vim a{{< /highlight >}}

Note

This requires to have done an initial build of Hue with:

{{< highlight bash >}}make apps{{< /highlight >}}

Integration Tests

Live cluster

The tests will run against the cluster configured in your hue.ini one if you specify:

{{< highlight bash >}}export LIVE_CLUSTER=true{{< /highlight >}}

Mini cluster

The test will run in a mini cluster (mini Hadoop, Oozie, Sqoop2 and Hive) created and destroyed after each run.

Here is how to get started:

{{< highlight bash >}}./tools/jenkins/jenkins.sh slow{{< /highlight >}}

Note

You might have lost all the changes in your local pseudo hue.ini because of the `git clean` done by the script.

In order to avoid this, add this into your `~/.bashrc`:

{{< highlight bash >}}export SKIP_CLEAN=true{{< /highlight >}}

Note

To point to an Impalad and trigger the Impala tests:

{{< highlight bash >}}export TEST_IMPALAD_HOST=impalad-01.gethue.com{{< /highlight >}}

or

{{< highlight bash >}}./build/env/bin/hue test impalaimpalad-01.gethue.com{{< /highlight >}}

 

It is then going to download the 4 latest Hadoop, Oozie, Sqoop2 and Hive and pre-install them for you.

You can CTRL+C and kill the script when you see:

{{< highlight bash >}}INFO: Oozie webconsole disabled, ExtJS library not specified

New Oozie WAR file with added 'JARs' at /home/romain/projects/hue-master/ext/oozie/oozie-4.0.0-cdh5.1.0/oozie-server/webapps/oozie.war

INFO: Oozie is ready to be started

  • /home/romain/projects/hue-master/ext/oozie/oozie-4.0.0-cdh5.1.0/bin/ooziedb.sh create -sqlfile oozie.sql -run

setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

Validate DB Connection

DONE

Check DB schema does not exist

DONE

Check OOZIE_SYS table does not exist

DONE

Create SQL schema

DONE

Create OOZIE_SYS table

DONE

Oozie DB has been created for Oozie version '4.0.0-cdh5.1.0'

The SQL commands have been written to: oozie.sql

++ curl -Is http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.3-cdh5.1.0.tar.gz

++ awk 'BEGIN {FS=":"} { if ($1 == "Last-Modified") { print substr($2,2) } }'

  • MTIME='Tue, 15 Jul 2014 20'

  • echo 'Tue, 15 Jul 2014 20'

  • diff /home/romain/.hue_cache/.sqoop_mtime -

  • return 0

  • '[' '!' -f /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz ']'

  • SQOOP_DIR=/home/romain/projects/hue-master/ext/sqoop

  • export SQOOP_HOME=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0

  • SQOOP_HOME=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0

  • mkdir -p /home/romain/projects/hue-master/ext/sqoop

  • rm -rf /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0

  • echo 'Unpacking /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz to /home/romain/projects/hue-master/ext/sqoop'

Unpacking /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz to /home/romain/projects/hue-master/ext/sqoop

  • tar -C /home/romain/projects/hue-master/ext/sqoop -xzf /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz

  • export SQOOP_CONF_DIR=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf

  • SQOOP_CONF_DIR=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf

  • rm -rf /home/romain/projects/hue-master/ext/sqoop/sqoop

  • ln -sf /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0 /home/romain/projects/hue-master/ext/sqoop/sqoop

  • sed -i.bk 's/12000/${test.port.http}/g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/server.xml

  • sed -i.bk 's/12001/${test.port.shutdown}/g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/server.xml

  • sed -i.bk 's/localhost/${test.host.local}/g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/server.xml

  • sed -i.bk 's|(common.loader.*$)|\1,/home/romain/projects/hue-master/ext/hadoop/hadoop/share/hadoop/common/lib/*.jar,/home/romain/projects/hue-master/ext/hadoop/hadoop/share/hadoop/mapreduce1/*.jar,/home/romain/projects/hue-master/ext/hadoop/hadoop/share/hadoop/mapreduce1/lib/*.jar|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/catalina.properties

  • sed -i.bk 's|${catalina.base}/logs|${test.log.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/logging.properties

  • sed -i.bk 's|\@LOGDIR\@|${test.log.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/sqoop.properties

  • sed -i.bk 's|\@BASEDIR\@|${test.base.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/sqoop.properties

  • sed -i.bk 's|/etc/hadoop/conf|${test.hadoop.conf.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/sqoop.properties

  • make apps

cd /home/romain/projects/hue-master/maven && mvn install

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------

[INFO] Building Hue Maven Parent POM 3.6.0-SNAPSHOT

[INFO] ------------------------

[INFO]

[INFO] - maven-enforcer-plugin:1.0:enforce (default) @ hue-parent -

[INFO]

[INFO] - maven-install-plugin:2.3:install (default-install) @ hue-parent -

[INFO] Installing /home/romain/projects/hue-master/maven/pom.xml to /home/romain/.m2/repository/com/cloudera/hue/hue-parent/3.6.0-SNAPSHOT/hue-parent-3.6.0-SNAPSHOT.pom

[INFO] ------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------

[INFO] Total time: 1.488s

[INFO] Finished at: Thu Jul 24 10:46:56 PDT 2014

[INFO] Final Memory: 7M/150M

[INFO] ------------------------

{{< /highlight >}}

And that's it! You can run all the tests or some parts with this syntax:

{{< highlight bash >}}build/env/bin/hue test specific filebrowser.views_test:test_listdir_sort_and_filter &> a; vim a{{< /highlight >}}

 

Note:

In some cases you might need to clear up your your cache with something like:

{{< highlight bash >}}rm /home/romain/.hue_cache/.*{{< /highlight >}}

then in

{{< highlight bash >}}ext/{{< /highlight >}}

delete everything but

{{< highlight bash >}}thirdparty/{{< /highlight >}}

 

Sum-up

We hope that is now easier to help you contribute to Hue 😉

As usual feel free to send feedback on the hue-user list or @gethue!