title: "API" date: 2019-03-13T18:28:09-07:00 draft: false
Hue can easily be extended to natively support other databases or be enriched via other data catalogs. Some components of Hue like the SQL parsers or Editor Scratchpad can also be imported independently as dependencies into your own apps.
The parsers are the flagship part of the Hue and power extremely advanced autocompletes or other SQL functionalities. They are running on the client side and comes with just a few megabytes of JavaScript that are cached by the browser. This provides a very reactive experience to the end user and allows to import them as classic JavaScript modules for your own development needs.
While the dynamic content like the list of tables, columns is obviously fetched via remote endpoints, all the SQL knowledge of the statements is available.
See the currently shipped SQL dialects.
What if I only want to use only the autocomplete as a JavaScript module in my own app?
Importing the Parser can be simply done as a npm package. Here is an example on how to use the parser in a node.js app:
cd tools/parser/hue_dep
npm install
npm run webpack
npm run app
In package.json there’s a dependency on Hue:
"dependencies": {
"hue": "file:../../.."
},
Note that it can also be a GitHub link, e.g. "hue": "git://github.com/cloudera/hue.git” but it takes a bit longer to do npm install.
Now let's import the Hive parser and run it on an SQL statement:
import sqlAutocompleteParser from 'hue/desktop/core/src/desktop/js/parse/sql/hive/hiveAutocompleteParser';
const beforeCursor = 'SELECT col1, col2, tbl2.col3 FROM tbl; '; // Note extra space at end
const afterCursor = '';
const dialect = 'hive';
const debug = false;
console.log(
JSON.stringify(
sqlAutocompleteParser.parseSql(beforeCursor, afterCursor, dialect, debug),
null,
2
)
);
Which then will output keywords suggestions and all the known locations:
{ locations:
[ { type: 'statement', location: [Object] },
{ type: 'statementType',
location: [Object],
identifier: 'SELECT' },
{ type: 'selectList', missing: false, location: [Object] },
{ type: 'column',
location: [Object],
identifierChain: [Array],
qualified: false,
tables: [Array] },
{ type: 'column',
location: [Object],
identifierChain: [Array],
qualified: false,
tables: [Array] },
{ type: 'column',
location: [Object],
identifierChain: [Array],
qualified: false,
tables: [Array] },
{ type: 'table', location: [Object], identifierChain: [Array] },
{ type: 'whereClause', missing: true, location: [Object] },
{ type: 'limitClause', missing: true, location: [Object] } ],
lowerCase: false,
suggestKeywords:
[ { value: 'ABORT', weight: -1 },
{ value: 'ALTER', weight: -1 },
{ value: 'ANALYZE TABLE', weight: -1 },
{ value: 'CREATE', weight: -1 },
{ value: 'DELETE', weight: -1 },
{ value: 'DESCRIBE', weight: -1 },
{ value: 'DROP', weight: -1 },
{ value: 'EXPLAIN', weight: -1 },
{ value: 'EXPORT', weight: -1 },
{ value: 'FROM', weight: -1 },
{ value: 'GRANT', weight: -1 },
{ value: 'IMPORT', weight: -1 },
{ value: 'INSERT', weight: -1 },
{ value: 'LOAD', weight: -1 },
{ value: 'MERGE', weight: -1 },
{ value: 'MSCK', weight: -1 },
{ value: 'RELOAD FUNCTION', weight: -1 },
{ value: 'RESET', weight: -1 },
{ value: 'REVOKE', weight: -1 },
{ value: 'SELECT', weight: -1 },
{ value: 'SET', weight: -1 },
{ value: 'SHOW', weight: -1 },
{ value: 'TRUNCATE', weight: -1 },
{ value: 'UPDATE', weight: -1 },
{ value: 'USE', weight: -1 },
{ value: 'WITH', weight: -1 } ],
definitions: [] }
The lightweight SQL Editor also called "Quick Query" comes as a Web component.
REST APIs are not all public yet but this is work in progress in HUE-1450.
Hue is Ajax based and has a REST API used by the browser to communicate (e.g. submit a query or workflow, list some S3 files, export a document...). Currently this API is private and subject to change but can be easily reused. You would need to GET /accounts/login to get the CSRF token and POST it back along username and password and reuse the sessionid cookie in next communication calls.
Hue is based on the Django Web Framework. Django comes with user authentication system. Django uses sessions and middleware to hook the authentication system into request object. Hue uses stock auth form which uses username and password and csrftoken form variables to authenticate.
In this code snippet, we will use well-known python requests library. We will first acquire csrftoken by GET login_url and then create a dictionary of form data which contains username, password and csrftoken and the next_url and another dictionary for header which contains the Referer url and an empty dictionary for the cookies. After the POST request to login_url we will check the reponse code, which should be r.status_code == 200.
Once the request is successful then capture headers and cookies for subsequent requests. Subsequent request.session calls can be made by providing cookies=session.cookies and headers=session.headers.
import requests
next_url = "/"
login_url = "http://localhost:8888/accounts/login"
session = requests.Session()
response = session.get(login_url)
form_data = {
'username': '[your Hue username]',
'password': '[your Hue password]',
'csrfmiddlewaretoken': session.cookies['csrftoken'],
'next': next_url
}
response = session.post(login_url, data=form_data, cookies={}, headers={'Referer': login_url})
print('Logged in successfully: %s %s' % (response.status_code == 200, response.status_code))
cookies = session.cookies
headers = session.headers
response = session.get('http://localhost:8888/metastore/databases/default/metadata')
print(response.status_code)
print(response.text)
The metadata API is powering the external Catalog integrations.
Additional catalogs can be integrated via some connectors.
$.post("/metadata/api/catalog/search_entities_interactive/", {
query_s: ko.mapping.toJSON("*sample"),
sources: ko.mapping.toJSON(["sql", "hdfs", "s3"]),
field_facets: ko.mapping.toJSON([]),
limit: 10
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
Searching for entities with the dummy backend:
$.post("/metadata/api/catalog/search_entities_interactive/", {
query_s: ko.mapping.toJSON("*sample"),
interface: "dummy"
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.get("/metadata/api/navigator/find_entity", {
type: "table",
database: "default",
name: "sample_07",
interface: "dummy"
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
Adding/updating a comment with the dummy backend:
$.post("/metadata/api/catalog/update_properties/", {
id: "22",
properties: ko.mapping.toJSON({"description":"Adding a description"}),
interface: "dummy"
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/metadata/api/catalog/add_tags/", {
id: "22",
tags: ko.mapping.toJSON(["usage"]),
interface: "dummy"
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/metadata/api/catalog/delete_metadata_properties/", {
"id": "32",
"keys": ko.mapping.toJSON(["project", "steward"])
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/metadata/api/catalog/delete_metadata_properties/", {
"id": "32",
"keys": ko.mapping.toJSON(["project", "steward"])
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.get("/metadata/api/catalog/models/properties/mappings/", function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/metadata/api/catalog/namespace/", {
namespace: 'huecatalog'
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/metadata/api/catalog/namespace/create/", {
"namespace": "huecatalog",
"description": "my desc"
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/metadata/api/catalog/namespace/property/create/", {
"namespace": "huecatalog",
"properties": ko.mapping.toJSON({
"name" : "relatedEntities2",
"displayName" : "Related objects",
"description" : "My desc",
"multiValued" : true,
"maxLength" : 50,
"pattern" : ".*",
"enumValues" : null,
"type" : "TEXT"
})
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/metadata/api/catalog/namespace/property/map/", {
"class": "hv_view",
"properties": ko.mapping.toJSON([{
namespace: "huecatalog",
name: "relatedQueries"
}])
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/notebook/api/autocomplete/", {
"snippet": ko.mapping.toJSON({
type: "hive"
})
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/notebook/api/autocomplete/<DB>", {
"snippet": ko.mapping.toJSON({
type: "hive"
})
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/notebook/api/autocomplete/<DB>/<TABLE>", {
"snippet": ko.mapping.toJSON({
type: "hive"
})
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
$.post("/notebook/api/autocomplete/<DB>/<TABLE>/<COL1>", {
"snippet": ko.mapping.toJSON({
type: "hive"
})
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
For nested columns:
$.post("/notebook/api/autocomplete/<DB>/<TABLE>/<COL1>/<COL2>", {
"snippet": ko.mapping.toJSON({
type: "hive"
})
}, function(data) {
console.log(ko.mapping.toJSON(data));
});
On the command line:
./build/env/bin/hue shell
If using Cloudera Manager, as a root user launch the shell.
Export the configuration directory:
export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE_SERVER | tail -1 | awk '{print $9}'`"
echo $HUE_CONF_DIR
> /var/run/cloudera-scm-agent/process/2061-hue-HUE_SERVER
Get the process id:
lsof -i :8888|grep -m1 hue|awk '{ print $2 }'
> 14850
In order to export all Hue's env variables:
for line in `strings /proc/$(lsof -i :8888|grep -m1 hue|awk '{ print $2 }')/environ|egrep -v "^HOME=|^TERM=|^PWD="`;do export $line;done
And finally launch the shell by:
HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=1 /opt/cloudera/parcels/CDH/lib/hue/build/env/bin/hue shell
> ALERT: This appears to be a CM Managed environment
> ALERT: HUE_CONF_DIR must be set when running hue commands in CM Managed environment
> ALERT: Please run 'hue <command> --cm-managed'
Then use the Python code to access a certain user information:
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
Type "copyright", "credits" or "license" for more information.
IPython 5.2.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
from django.contrib.auth.models import User
from desktop.models import Document2
user = User.objects.get(username='demo')
Document2.objects.documents(user=user).count()
In [8]: Document2.objects.documents(user=user).count()
Out[8]: 1167
In [10]: Document2.objects.documents(user=user, perms='own').count()
Out[10]: 1166
In [11]: Document2.objects.documents(user=user, perms='own', include_history=True).count()
Out[11]: 7125
In [12]: Document2.objects.documents(user=user, perms='own', include_history=True, include_trashed=True).count()
Out[12]: 7638
In [13]: Document2.objects.documents(user=user, perms='own', include_history=True, include_trashed=True, include_managed=True).count()
Out[13]: 31408
Out[14]:
(85667L,
{u'desktop.Document': 18524L,
u'desktop.Document2': 31409L,
u'desktop.Document2Permission': 556L,
u'desktop.Document2Permission_groups': 277L,
u'desktop.Document2Permission_users': 0L,
u'desktop.Document2_dependencies': 15087L,
u'desktop.DocumentPermission': 1290L,
u'desktop.DocumentPermission_groups': 0L,
u'desktop.DocumentPermission_users': 0L,
u'desktop.Document_tags': 18524L})
Building a brand new application is more work but is ideal for creating a custom solution.
Note It is now more recommended to integrate external services (e.g. integrate a new SQL Datatase with the Editor, add a new visualization...) to the core Hue APIs instead of building brand new application. This page gives good content in both cases. Feel free to contact the community for advice.
Hue leverages the browser to provide users with an environment for exploring and analyzing data.
Build on top of the Hue SDK to enable your application to interact efficiently with Hadoop and the other Hue services.
By building on top of Hue SDK, you get, out of the box:
This document will orient you with the general structure of Hue and will walk you through adding a new application using the SDK.
Run "create_desktop_app" to Set up a New Source Tree
./build/env/bin/hue create_desktop_app calculator
find calculator -type f
calculator/setup.py # distutils setup file
calculator/src/calculator/__init__.py # main src module
calculator/src/calculator/forms.py
calculator/src/calculator/models.py
calculator/src/calculator/settings.py # app metadata setting
calculator/src/calculator/urls.py # url mapping
calculator/src/calculator/views.py # app business logic
calculator/src/calculator/templates/index.mako
calculator/src/calculator/templates/shared_components.mako
# Static resources
calculator/src/static/calculator/art/calculator.png # logo
calculator/src/static/calculator/css/calculator.css
calculator/src/static/calculator/js/calculator.js
As you'll discover if you look at calculator's setup.py, Hue uses a distutils entrypoint to register applications. By installing the calculator package into Hue's python virtual environment, you'll install a new app. The "app_reg.py" tool manages the applications that are installed. Note that in the following example, the value after the "--install" option is the path to the root directory of the application you want to install. In this example, it is a relative path to "/Users/philip/src/hue/calculator".
./build/env/bin/python tools/app_reg/app_reg.py --install calculator --relative-paths
=== Installing app at calculator
Updating registry with calculator (version 0.1)
--- Making egg-info for calculator
Congrats, you've added a new app!
You can now browse the new application.
# If you haven't killed the old process, do so now.
build/env/bin/hue runserver
And then visit http://localhost:8000/ to check it out! You should see the app in the left menu.
Now that your app has been installed, you'll want to customize it.
As you may have guessed, we're going to build a small calculator
application. Edit calculator/src/calculator/templates/index.mako
to include a simple form and a Knockout viewmodel:
<%!from desktop.views import commonheader, commonfooter %>
<%namespace name="shared" file="shared_components.mako" />
%if not is_embeddable:
${commonheader("Calculator", "calculator", user, "100px") | n,unicode}
%endif
## Main body
<div class="container-fluid calculator-components">
<div class="row">
<div class="span6 offset3 margin-top-30 text-center">
<form class="form-inline">
<input type="text" class="input-mini margin-right-10" placeholder="A" data-bind="value: a">
<!-- ko foreach: operations -->
<label class="radio margin-left-5">
<input type="radio" name="op" data-bind="checkedValue: $data, checked: $parent.chosenOperation" /><span data-bind="text: $data"></span>
</label>
<!-- /ko -->
<input type="text" class="input-mini margin-left-10" placeholder="B" data-bind="value: b">
<button class="btn" data-bind="click: calculate">Calculate</button>
</form>
<h2 data-bind="visible: result() !== null">The result is <strong data-bind="text: result"></strong></h2>
</div>
</div>
</div>
<script>
(function() {
var CalculatorViewModel = function () {
var self = this;
self.operations = ko.observableArray(['+', '-', '*', '/']);
self.a = ko.observable();
self.b = ko.observable();
self.chosenOperation = ko.observable('+');
self.result = ko.observable(null);
self.calculate = function () {
var a = parseFloat(self.a());
var b = parseFloat(self.b());
var result = null;
switch (self.chosenOperation()) {
case '+':
result = a + b;
break;
case '-':
result = a - b;
break;
case '*':
result = a * b;
break;
case '/':
result = a / b;
}
self.result(result);
}
};
$(document).ready(function () {
ko.applyBindings(new CalculatorViewModel(), $('.calculator-components')[0]);
});
})();
</script>
%if not is_embeddable:
${commonfooter(messages) | n,unicode}
%endif
The template language here is Mako,
which is flexible and powerful. If you use the ".html" extension, Hue
will render your page using
Django templates
instead.
Note that we use Knockout.js to do the heavy lifting of this app.
Let's edit calculator/src/calculator/views.py to simply render the page:
#!/usr/bin/env python
from desktop.lib.django_util import render
def index(request):
return render('index.mako', request, {
'is_embeddable': request.GET.get('is_embeddable', False),
})
You can now go and try the calculator.