title: 'Tutorial: How to run the Hue unit and integration tests?' author: admin type: post date: 2014-07-24T17:47:08+00:00 url: /tutorial-how-to-run-the-hue-integration-tests/ sf_thumbnail_type:
After the Review Board post, here is another tutorial about becoming an awesome Hue developer: how to run the Hue tests!
Hue has two types of tests:
First, clone the Hue repository and make sure that you have all the pre-requisite packages:
{{< highlight bash >}}git clone git@github.com:cloudera/hue.git{{< /highlight >}}
The regular unit tests do not require all this setup! Just run them directly:
{{< highlight bash >}}build/env/bin/hue test specific beeswax.tests:test_split_statements &> a; vim a{{< /highlight >}}
Note
This requires to have done an initial build of Hue with:
{{< highlight bash >}}make apps{{< /highlight >}}
The tests will run against the cluster configured in your hue.ini one if you specify:
{{< highlight bash >}}export LIVE_CLUSTER=true{{< /highlight >}}
The test will run in a mini cluster (mini Hadoop, Oozie, Sqoop2 and Hive) created and destroyed after each run.
Here is how to get started:
{{< highlight bash >}}./tools/jenkins/jenkins.sh slow{{< /highlight >}}
Note
You might have lost all the changes in your local pseudo hue.ini because of the `git clean` done by the script.
In order to avoid this, add this into your `~/.bashrc`:
{{< highlight bash >}}export SKIP_CLEAN=true{{< /highlight >}}
Note
To point to an Impalad and trigger the Impala tests:
{{< highlight bash >}}export TEST_IMPALAD_HOST=impalad-01.gethue.com{{< /highlight >}}
or
{{< highlight bash >}}./build/env/bin/hue test impalaimpalad-01.gethue.com{{< /highlight >}}
It is then going to download the 4 latest Hadoop, Oozie, Sqoop2 and Hive and pre-install them for you.
You can CTRL+C and kill the script when you see:
{{< highlight bash >}}INFO: Oozie webconsole disabled, ExtJS library not specified
New Oozie WAR file with added 'JARs' at /home/romain/projects/hue-master/ext/oozie/oozie-4.0.0-cdh5.1.0/oozie-server/webapps/oozie.war
INFO: Oozie is ready to be started
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Validate DB Connection
DONE
Check DB schema does not exist
DONE
Check OOZIE_SYS table does not exist
DONE
Create SQL schema
DONE
Create OOZIE_SYS table
DONE
Oozie DB has been created for Oozie version '4.0.0-cdh5.1.0'
The SQL commands have been written to: oozie.sql
build_sqoop
check_mtime /home/romain/.hue_cache/.sqoop_mtime http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.3-cdh5.1.0.tar.gz
MTIME_FILE=/home/romain/.hue_cache/.sqoop_mtime
++ curl -Is http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.3-cdh5.1.0.tar.gz
++ awk 'BEGIN {FS=":"} { if ($1 == "Last-Modified") { print substr($2,2) } }'
MTIME='Tue, 15 Jul 2014 20'
echo 'Tue, 15 Jul 2014 20'
diff /home/romain/.hue_cache/.sqoop_mtime -
return 0
'[' '!' -f /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz ']'
SQOOP_DIR=/home/romain/projects/hue-master/ext/sqoop
export SQOOP_HOME=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0
SQOOP_HOME=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0
mkdir -p /home/romain/projects/hue-master/ext/sqoop
rm -rf /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0
echo 'Unpacking /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz to /home/romain/projects/hue-master/ext/sqoop'
Unpacking /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz to /home/romain/projects/hue-master/ext/sqoop
tar -C /home/romain/projects/hue-master/ext/sqoop -xzf /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz
export SQOOP_CONF_DIR=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf
SQOOP_CONF_DIR=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf
rm -rf /home/romain/projects/hue-master/ext/sqoop/sqoop
ln -sf /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0 /home/romain/projects/hue-master/ext/sqoop/sqoop
sed -i.bk 's/12000/${test.port.http}/g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/server.xml
sed -i.bk 's/12001/${test.port.shutdown}/g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/server.xml
sed -i.bk 's/localhost/${test.host.local}/g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/server.xml
sed -i.bk 's|(common.loader.*$)|\1,/home/romain/projects/hue-master/ext/hadoop/hadoop/share/hadoop/common/lib/*.jar,/home/romain/projects/hue-master/ext/hadoop/hadoop/share/hadoop/mapreduce1/*.jar,/home/romain/projects/hue-master/ext/hadoop/hadoop/share/hadoop/mapreduce1/lib/*.jar|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/catalina.properties
sed -i.bk 's|${catalina.base}/logs|${test.log.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/logging.properties
sed -i.bk 's|\@LOGDIR\@|${test.log.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/sqoop.properties
sed -i.bk 's|\@BASEDIR\@|${test.base.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/sqoop.properties
sed -i.bk 's|/etc/hadoop/conf|${test.hadoop.conf.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/sqoop.properties
make apps
cd /home/romain/projects/hue-master/maven && mvn install
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------
[INFO] Building Hue Maven Parent POM 3.6.0-SNAPSHOT
[INFO] ------------------------
[INFO]
[INFO] - maven-enforcer-plugin:1.0:enforce (default) @ hue-parent -
[INFO]
[INFO] - maven-install-plugin:2.3:install (default-install) @ hue-parent -
[INFO] Installing /home/romain/projects/hue-master/maven/pom.xml to /home/romain/.m2/repository/com/cloudera/hue/hue-parent/3.6.0-SNAPSHOT/hue-parent-3.6.0-SNAPSHOT.pom
[INFO] ------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------
[INFO] Total time: 1.488s
[INFO] Finished at: Thu Jul 24 10:46:56 PDT 2014
[INFO] Final Memory: 7M/150M
[INFO] ------------------------
{{< /highlight >}}
And that's it! You can run all the tests or some parts with this syntax:
{{< highlight bash >}}build/env/bin/hue test specific filebrowser.views_test:test_listdir_sort_and_filter &> a; vim a{{< /highlight >}}
Note:
In some cases you might need to clear up your your cache with something like:
{{< highlight bash >}}rm /home/romain/.hue_cache/.*{{< /highlight >}}
then in
{{< highlight bash >}}ext/{{< /highlight >}}
delete everything but
{{< highlight bash >}}thirdparty/{{< /highlight >}}
We hope that is now easier to help you contribute to Hue 😉
As usual feel free to send feedback on the hue-user list or @gethue!