|
|
@@ -11,7 +11,7 @@ The REST API is not properly public yet and can (will) be simplified in the curr
|
|
|
|
|
|
Hue is Ajax based and has a REST API used by the browser to communicate (e.g. submit an SQL query, list some S3 files in a buck...) with the Hue server which will then perform the operation with the remote services. Currently this API is private and subject to change but can already be reused for prove of concepts until the proper public API is released.
|
|
|
|
|
|
-In general we want to authenticate with a username and password, retrieve a cookie and [CSRF](https://docs.djangoproject.com/en/3.1/ref/csrf/) token and provide them in the follow-up calls so that Hue knows who we are and won't block you.
|
|
|
+In general we want to authenticate with a username and password, retrieve a `session` and [CSRF](https://docs.djangoproject.com/en/3.1/ref/csrf/) cookies and provide them in the follow-up calls so that Hue knows who we are and won't block you.
|
|
|
|
|
|
The [API documentation](https://docs.gethue.com/developer/api/) has been refreshed and list new endpoints.
|
|
|
|
|
|
@@ -30,17 +30,22 @@ It is important to note the `csrftoken` and `sessionid` values from the response
|
|
|
set-cookie: csrftoken=XUFgN1WPZNlaJtBeBDtBvwzrOFqRXIaMlNJv4mdvsS2bIE2Lb8LRmCh5cPUBnBdk; expires=Mon, 13-Sep-2021 20:43:47 GMT; httponly; Max-Age=31449600; Path=/
|
|
|
set-cookie: sessionid=9cdltfee1q1zmt8b7slsjcomtxzgvgfz; expires=Mon, 14-Sep-2020 21:43:47 GMT; httponly; Max-Age=3600; Path=/
|
|
|
|
|
|
-
|
|
|
Then authenticate with the credendials and provide the `csrftoken` and `sessionid` as well (note: `CRSF` currently needs to be repeated twice):
|
|
|
|
|
|
curl -i -X POST https://demo.gethue.com/hue/accounts/login/?fromModal=true -d 'username=demo&password=demo' -o /dev/null -D - --cookie "csrftoken=XUFgN1WPZNlaJtBeBDtBvwzrOFqRXIaMlNJv4mdvsS2bIE2Lb8LRmCh5cPUBnBdk;sessionid=9cdltfee1q1zmt8b7slsjcomtxzgvgfz" -H "X-CSRFToken: XUFgN1WPZNlaJtBeBDtBvwzrOFqRXIaMlNJv4mdvsS2bIE2Lb8LRmCh5cPUBnBdk"
|
|
|
|
|
|
-Then you can perform any operation as the `demo` user by providing the latest `CSRF` and the `sessionid` tokens. Here for example we list all the databases of the `hive` connector:
|
|
|
+Then you can perform any operation as the `demo` user by providing the latest `CSRF` and the `sessionid` tokens with these parameters:
|
|
|
+
|
|
|
+ --cookie "csrftoken=tkuKwiyRsEmt7zCqIfym4GCwPxAPQfnTm4Y97gUSszkCRih067sC2GA2mCrH6Lmu;sessionid=z6pjze6j6zrv3uoe2hxa8dfbhbn9bynp" -H "X-CSRFToken: tkuKwiyRsEmt7zCqIfym4GCwPxAPQfnTm4Y97gUSszkCRih067sC2GA2mCrH6Lmu"
|
|
|
+
|
|
|
+Here for example we list all the databases of the `hive` connector:
|
|
|
|
|
|
curl -X POST https://demo.gethue.com/notebook/api/autocomplete/ --data 'snippet={"type":"hive"}' --cookie "csrftoken=XUFgN1WPZNlaJtBeBDtBvwzrOFqRXIaMlNJv4mdvsS2bIE2Lb8LRmCh5cPUBnBdk;sessionid=9cdltfee1q1zmt8b7slsjcomtxzgvgfz" -H "X-CSRFToken: XUFgN1WPZNlaJtBeBDtBvwzrOFqRXIaMlNJv4mdvsS2bIE2Lb8LRmCh5cPUBnBdk"
|
|
|
|
|
|
{"status": 0, "databases": ["a0817", "arunso", "ath", "athlete", "beingdatum_db", "bharath_practice", "chungu", "darth", "default", "demo", "diwakar", "emp", "hadooptest", "hebe", "hello", "hivedataset", "hivemap", "hivetesting", "hr_db", "icedb", "lib", "libdemo", "lti", "m", "m1", "movie", "movie1", "movielens", "mscda", "my_database", "mydb", "mydemo", "noo", "prizesh", "rajeev", "ram", "retail", "ruban", "sept9", "sept9_2020", "student", "test", "test21", "test123", "ticktick", "userdb", "vidu"]}
|
|
|
|
|
|
+**Note** The `cookie` and `CSRF` tokens are omitted in the following API description to keep it simpler. Those parameters will be integrated into one in the future.
|
|
|
+
|
|
|
### Python
|
|
|
|
|
|
In this code snippet, we will use well-known python *requests* library. We will first acquire *csrftoken* by GET *login_url* and then create a dictionary of form data which contains *username*, *password* and *csrftoken* and the *next_url* and another dictionary for header which contains the *Referer* url and an empty dictionary for the cookies. After the POST request to *login_url* we will check the response code, which should be *r.status_code == 200*.
|
|
|
@@ -127,7 +132,7 @@ Login to a Hue and open up the browser developer console and just type the JavaS
|
|
|
|
|
|
### Logout
|
|
|
|
|
|
- curl -v -X POST https://demo.gethue.com/hue/accounts/logout/ -H "X-CSRFToken: CZOt9VFK7X6OUpqrLk4URW9DI69FxVKnsRnuaDgsDqZS9GpXzuVehxMnH1gvheWU" --cookie "csrftoken=CZOt9VFK7X6OUpqrLk4URW9DI69FxVKnsRnuaDgsDqZS9GpXzuVehxMnH1gvheWU"
|
|
|
+ curl -v -X POST https://demo.gethue.com/hue/accounts/logout/
|
|
|
|
|
|
## SQL Querying
|
|
|
|
|
|
@@ -139,25 +144,25 @@ Until Editor v2 is out [HUE-8768](https://issues.cloudera.org/browse/HUE-8768),
|
|
|
|
|
|
For a `SHOW TABLES`, first we send the query statement:
|
|
|
|
|
|
- curl -X POST https://demo.gethue.com/notebook/api/execute/hive --data 'executable={"statement":"SHOW TABLES","database":"default"}¬ebook={"type":"query","snippets":[{"id":1,"statement_raw":"SHOW TABLES","type":"hive","variables":[]}],"name":"","isSaved":false,"sessions":[]}&snippet={"id":1,"type":"hive","result":{},"statement":"SHOW TABLES","properties":{}}' --cookie "csrftoken=lFpBt6uaa8isgjIthEFZdUbofKPI7wJaXpWS0q54YKORs8zWvKgvrKzwUnTjsFt3;sessionid=5msr9s45o4emem1zt009kw64ni6uso3e" -H "X-CSRFToken: lFpBt6uaa8isgjIthEFZdUbofKPI7wJaXpWS0q54YKORs8zWvKgvrKzwUnTjsFt3"
|
|
|
+ curl -X POST https://demo.gethue.com/notebook/api/execute/hive --data 'executable={"statement":"SHOW TABLES","database":"default"}¬ebook={"type":"query","snippets":[{"id":1,"statement_raw":"SHOW TABLES","type":"hive","variables":[]}],"name":"","isSaved":false,"sessions":[]}&snippet={"id":1,"type":"hive","result":{},"statement":"SHOW TABLES","properties":{}}'
|
|
|
|
|
|
{"status": 0, "history_id": 17880, "handle": {"statement_id": 0, "session_type": "hive", "has_more_statements": false, "guid": "EUI32vrfTkSOBXET6Eaa+A==\n", "previous_statement_hash": "3070952e55d733fb5bef249277fb8674989e40b6f86c5cc8b39cc415", "log_context": null, "statements_count": 1, "end": {"column": 10, "row": 0}, "session_id": 63, "start": {"column": 0, "row": 0}, "secret": "RuiF0LEkRn+Yok/gjXWSqg==\n", "has_result_set": true, "session_guid": "c845bb7688dca140:859a5024fb284ba2", "statement": "SHOW TABLES", "operation_type": 0, "modified_row_count": null}, "history_uuid": "63ce87ba-ca0f-4653-8aeb-e9f5c1781b78"}
|
|
|
|
|
|
Then check the operation status until it is available for fetching its result:
|
|
|
|
|
|
- curl -X POST https://demo.gethue.com/notebook/api/check_status --data 'notebook={"type":"hive"}&snippet={"history_id": 17886,"type":"hive","result":{"handle":{"guid": "0J6PwGcSQaCJjagzYUBHzA==\n","secret": "uiP3IS4fR/mxkLJER5wRCg==\n","has_result_set": true}},"status":""}' --cookie "csrftoken=lFpBt6uaa8isgjIthEFZdUbofKPI7wJaXpWS0q54YKORs8zWvKgvrKzwUnTjsFt3;sessionid=5msr9s45o4emem1zt009kw64ni6uso3e" -H "X-CSRFToken: lFpBt6uaa8isgjIthEFZdUbofKPI7wJaXpWS0q54YKORs8zWvKgvrKzwUnTjsFt3"
|
|
|
+ curl -X POST https://demo.gethue.com/notebook/api/check_status --data 'notebook={"type":"hive"}&snippet={"history_id": 17886,"type":"hive","result":{"handle":{"guid": "0J6PwGcSQaCJjagzYUBHzA==\n","secret": "uiP3IS4fR/mxkLJER5wRCg==\n","has_result_set": true}},"status":""}'
|
|
|
|
|
|
{"status": 0, "query_status": {"status": "available", "has_result_set": true}}
|
|
|
|
|
|
And now ask for the resultset of the statement:
|
|
|
|
|
|
- curl -X POST https://demo.gethue.com/notebook/api/fetch_result_data --data 'notebook={"type":"hive"}&snippet={"history_id": 17886,"type":"hive","result":{"handle":{"guid": "0J6PwGcSQaCJjagzYUBHzA==\n","secret": "uiP3IS4fR/mxkLJER5wRCg==\n","has_result_set": true}},"status":""}' --cookie "csrftoken=lFpBt6uaa8isgjIthEFZdUbofKPI7wJaXpWS0q54YKORs8zWvKgvrKzwUnTjsFt3;sessionid=5msr9s45o4emem1zt009kw64ni6uso3e" -H "X-CSRFToken: lFpBt6uaa8isgjIthEFZdUbofKPI7wJaXpWS0q54YKORs8zWvKgvrKzwUnTjsFt3"
|
|
|
+ curl -X POST https://demo.gethue.com/notebook/api/fetch_result_data --data 'notebook={"type":"hive"}&snippet={"history_id": 17886,"type":"hive","result":{"handle":{"guid": "0J6PwGcSQaCJjagzYUBHzA==\n","secret": "uiP3IS4fR/mxkLJER5wRCg==\n","has_result_set": true}},"status":""}'
|
|
|
|
|
|
{"status": 0, "result": {"has_more": true, "type": "table", "meta": [{"comment": "from deserializer", "type": "STRING_TYPE", "name": "tab_name"}], "data": [["adavi"], ["adavi1"], ["adavi2"], ["ambs_feed"], ["apx_adv_deduction_data_process_total"], ["avro_table"], ["avro_table1"], ["bb"], ["bharath_info1"], ["bucknew"], ["bucknew1"], ["chungu"], ["cricket3"], ["cricket4"], ["cricket5_view"], ["cricketer"], ["cricketer_view"], ["cricketer_view1"], ["demo1"], ["demo12345"], ["dummy"], ["embedded"], ["emp"], ["emp1_sept9"], ["emp_details"], ["emp_sept"], ["emp_tbl1"], ["emp_tbl2"], ["empdtls"], ["empdtls_ext"], ["empdtls_ext_v2"], ["employee"], ["employee1"], ["employee_ins"], ["empppp"], ["events"], ["final"], ["flight_data"], ["gopalbhar"], ["guruhive_internaltable"], ["hell"], ["info1"], ["lost_messages"], ["mnewmyak"], ["mortality"], ["mscda"], ["myak"], ["mysample"], ["mysample1"], ["mysample2"], ["network"], ["ods_t_exch_recv_rel_wfz_stat_szy"], ["olympicdata"], ["p_table"], ["partition_cricket"], ["partitioned_user"], ["s"], ["sample"], ["sample_07"], ["sample_08"], ["score"], ["stg_t_exch_recv_rel_wfz_stat_szy"], ["stocks"], ["students"], ["studentscores"], ["studentscores2"], ["t1"], ["table_name"], ["tablex"], ["tabley"], ["temp"], ["test1"], ["test2"], ["test21"], ["test_info"], ["topage"], ["txnrecords"], ["u_data"], ["udata"], ["user_session"], ["user_test"], ["v_empdtls"], ["v_empdtls_ext"], ["v_empdtls_ext_v2"], ["web_logs"]], "isEscaped": true}}
|
|
|
|
|
|
And if we wanted to get the execution log for this statement:
|
|
|
|
|
|
- curl -X POST https://demo.gethue.com/notebook/api/get_logs --data 'notebook={"type":"hive","sessions":[]}&snippet={"history_id": 17886,"type":"hive","result":{"handle":{"guid": "0J6PwGcSQaCJjagzYUBHzA==\n","secret": "uiP3IS4fR/mxkLJER5wRCg==\n","has_result_set": true}},"status":"","properties":{},"sessions":[]}' --cookie "csrftoken=lFpBt6uaa8isgjIthEFZdUbofKPI7wJaXpWS0q54YKORs8zWvKgvrKzwUnTjsFt3;sessionid=5msr9s45o4emem1zt009kw64ni6uso3e" -H "X-CSRFToken: lFpBt6uaa8isgjIthEFZdUbofKPI7wJaXpWS0q54YKORs8zWvKgvrKzwUnTjsFt3"
|
|
|
+ curl -X POST https://demo.gethue.com/notebook/api/get_logs --data 'notebook={"type":"hive","sessions":[]}&snippet={"history_id": 17886,"type":"hive","result":{"handle":{"guid": "0J6PwGcSQaCJjagzYUBHzA==\n","secret": "uiP3IS4fR/mxkLJER5wRCg==\n","has_result_set": true}},"status":"","properties":{},"sessions":[]}'
|
|
|
|
|
|
{"status": 0, "progress": 5, "jobs": [], "logs": "", "isFullLogs": false}
|
|
|
|
|
|
@@ -224,7 +229,6 @@ Default functions:
|
|
|
console.log(ko.mapping.toJSON(data));
|
|
|
});
|
|
|
|
|
|
-
|
|
|
For a specific database:
|
|
|
|
|
|
$.post("/notebook/api/autocomplete/<DB>", {
|
|
|
@@ -236,7 +240,6 @@ For a specific database:
|
|
|
console.log(ko.mapping.toJSON(data));
|
|
|
});
|
|
|
|
|
|
-
|
|
|
For a specific function/UDF details (e.g. trunc):
|
|
|
|
|
|
$.post("/notebook/api/autocomplete/<function_name>", {
|
|
|
@@ -248,6 +251,127 @@ For a specific function/UDF details (e.g. trunc):
|
|
|
console.log(ko.mapping.toJSON(data));
|
|
|
});
|
|
|
|
|
|
+## File Browsing
|
|
|
+
|
|
|
+### List
|
|
|
+
|
|
|
+Hue's [File Browser](https://docs.gethue.com/user/browsing/#data) offer uploads, downloads and listing of data in HDFS, S3, ADLS storages.
|
|
|
+
|
|
|
+Here is how to list the content of a path, here the S3 bucket `S3A://gethue-demo`:
|
|
|
+
|
|
|
+ curl -X GET "https://demo.gethue.com/filebrowser/view=S3A://gethue-demo?pagesize=45&pagenum=1&filter=&sortby=name&descending=false&format=json"
|
|
|
+
|
|
|
+ {
|
|
|
+ ...........
|
|
|
+ "files": [
|
|
|
+ {
|
|
|
+ "humansize": "0\u00a0bytes",
|
|
|
+ "url": "/filebrowser/view=s3a%3A%2F%2Fdemo-hue",
|
|
|
+ "stats": {
|
|
|
+ "size": 0,
|
|
|
+ "aclBit": false,
|
|
|
+ "group": "",
|
|
|
+ "user": "",
|
|
|
+ "mtime": null,
|
|
|
+ "path": "s3a://demo-hue",
|
|
|
+ "atime": null,
|
|
|
+ "mode": 16895
|
|
|
+ },
|
|
|
+ "name": "demo-hue",
|
|
|
+ "mtime": "",
|
|
|
+ "rwx": "drwxrwxrwx",
|
|
|
+ "path": "s3a://demo-hue",
|
|
|
+ "is_sentry_managed": false,
|
|
|
+ "type": "dir",
|
|
|
+ "mode": "40777"
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "humansize": "0\u00a0bytes",
|
|
|
+ "url": "/filebrowser/view=S3A%3A%2F%2F",
|
|
|
+ "stats": {
|
|
|
+ "size": 0,
|
|
|
+ "aclBit": false,
|
|
|
+ "group": "",
|
|
|
+ "user": "",
|
|
|
+ "mtime": null,
|
|
|
+ "path": "S3A://",
|
|
|
+ "atime": null,
|
|
|
+ "mode": 16895
|
|
|
+ },
|
|
|
+ "name": ".",
|
|
|
+ "mtime": "",
|
|
|
+ "rwx": "drwxrwxrwx",
|
|
|
+ "path": "S3A://",
|
|
|
+ "is_sentry_managed": false,
|
|
|
+ "type": "dir",
|
|
|
+ "mode": "40777"
|
|
|
+ }
|
|
|
+ ],
|
|
|
+ ...........
|
|
|
+ }
|
|
|
+
|
|
|
+### Get file content
|
|
|
+
|
|
|
+How to get the file content and its metadata. Here with the public file of demo.gethue.com [s3a://demo-hue/web_log_data/index_data.csv](https://demo.gethue.com/hue/filebrowser/view=s3a%3A%2F%2Fdemo-hue%2Fweb_log_data%2Findex_data.csv):
|
|
|
+
|
|
|
+**Note** It needs the `XMLHttpRequest` header to return the data in json:
|
|
|
+
|
|
|
+ curl -X GET "https://demo.gethue.com/filebrowser/view=s3a://demo-hue/web_log_data/index_data.csv?offset=0&length=204800&compression=none&mode=text" -H "X-requested-with: XMLHttpRequest"
|
|
|
+
|
|
|
+ {
|
|
|
+ "show_download_button": true,
|
|
|
+ "is_embeddable": false,
|
|
|
+ "editable": false,
|
|
|
+ "mtime": "October 31, 2016 03:34 PM",
|
|
|
+ "rwx": "-rw-rw-rw-",
|
|
|
+ "path": "s3a://demo-hue/web_log_data/index_data.csv",
|
|
|
+ "stats": {
|
|
|
+ "size": 6199593,
|
|
|
+ "aclBit": false,
|
|
|
+ ...............
|
|
|
+ "contents": "code,protocol,request,app,user_agent_major,region_code,country_code,id,city,subapp,latitude,method,client_ip, user_agent_family,bytes,referer,country_name,extension,url,os_major,longitude,device_family,record,user_agent,time,os_family,country_code3
|
|
|
+ 200,HTTP/1.1,GET /metastore/table/default/sample_07 HTTP/1.1,metastore,,00,SG,8836e6ce-9a21-449f-a372-9e57641389b3,Singapore,table,1.2931000000000097,GET,128.199.234.236,Other,1041,-,Singapore,,/metastore/table/default/sample_07,,103.85579999999999,Other,"demo.gethue.com:80 128.199.234.236 - - [04/May/2014:06:35:49 +0000] ""GET /metastore/table/default/sample_07 HTTP/1.1"" 200 1041 ""-"" ""Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)""
|
|
|
+ ",Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org),2014-05-04T06:35:49Z,Other,SGP
|
|
|
+ 200,HTTP/1.1,GET /metastore/table/default/sample_07 HTTP/1.1,metastore,,00,SG,6ddf6e38-7b83-423c-8873-39842dca2dbb,Singapore,table,1.2931000000000097,GET,128.199.234.236,Other,1041,-,Singapore,,/metastore/table/default/sample_07,,103.85579999999999,Other,"demo.gethue.com:80 128.199.234.236 - - [04/May/2014:06:35:50 +0000] ""GET /metastore/table/default/sample_07 HTTP/1.1"" 200 1041 ""-"" ""Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)""
|
|
|
+ ",Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org),2014-05-04T06:35:50Z,Other,SGP
|
|
|
+ ...............
|
|
|
+ }
|
|
|
+
|
|
|
+## Data Importer
|
|
|
+
|
|
|
+### File import
|
|
|
+
|
|
|
+First guessing the format of the file `s3a://demo-hue/web_log_data/index_data.csv`:
|
|
|
+
|
|
|
+ curl -X POST https://demo.gethue.com/indexer/api/indexer/guess_format --data 'fileFormat={"inputFormat":"file","path":"s3a://demo-hue/web_log_data/index_data.csv"}'
|
|
|
+
|
|
|
+ {"status": 0, "fieldSeparator": ",", "hasHeader": true, "quoteChar": "\"", "recordSeparator": "\\n", "type": "csv"}
|
|
|
+
|
|
|
+Then getting some data sample as well as the column types (column names will be picked from the header line if present):
|
|
|
+
|
|
|
+ curl -X POST https://demo.gethue.com/indexer/api/indexer/guess_field_types --data 'fileFormat={"inputFormat":"file","path":"s3a://demo-hue/web_log_data/index_data.csv","format":{"type":"csv","fieldSeparator":",","recordSeparator":"\\n","quoteChar":"\"","hasHeader":true,"status":0}}'
|
|
|
+
|
|
|
+ {
|
|
|
+ "sample": [["200", "HTTP/1.1", "GET /metastore/table/default/sample_07 HTTP/1.1", "metastore", "", "00", "SG", "8836e6ce-9a21-449f-a372-9e57641389b3", "Singapore", "table", "1.2931000000000097", "GET", "128.199.234.236", "Other", "1041", "-", "Singapore", "", "/metastore/table/default/sample_07", "", "103.85579999999999", "Other", "demo.gethue.com:80 128.199.234.236 - - [04/May/2014:06:35:49 +0000] \"GET /metastore/table/default/sample_07 HTTP/1.1\" 200 1041 \"-\" \"Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)\"\n", "Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)", "2014-05-04T06:35:49Z", "Other", "SGP"],
|
|
|
+ ....
|
|
|
+ "columns": [{"operations": [], "comment": "", "nested": [], "name": "code", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "long", "showProperties": false, "scale": 0}, {"operations": [], "comment": "", "nested": [], "name": "protocol", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "string", "showProperties": false, "scale": 0},
|
|
|
+ .....
|
|
|
+ }
|
|
|
+
|
|
|
+Then we submit via `https://demo.gethue.com/indexer/api/importer/submit` and provide the `source` and `destination` parameters. We get back an `operation id` (i.e. some SQL Editor query history id).
|
|
|
+
|
|
|
+If the `show_command` parameter is given, the API call will instead return the generated SQL queries that will import the data.
|
|
|
+
|
|
|
+ {"status": 0, "history_id": 17820, "handle": {"statement_id": 0, "session_type": "hive", "has_more_statements": true, "guid": "uu6K3SSWSY6mx/fbh0nm2w==\n", "previous_statement_hash": "4bee3a62b3c7142c60475021469483bff81ba09bd07b8e527179e617", "log_context": null, "statements_count": 4, "end": {"column": 53, "row": 0}, "session_id": 55, "start": {"column": 0, "row": 0}, "secret": "8mKu1bhdRtWXu82DXjDZdg==\n", "has_result_set": false, "session_guid": "fd4c667f3a5e4507:0335af7716db3d9e", "statement": "DROP TABLE IF EXISTS `default`.`hue__tmp_index_data`", "operation_type": 0, "modified_row_count": null}, "history_uuid": "bf5804f5-6f12-47a8-8ba6-0ed7032ebe93"}
|
|
|
+
|
|
|
+## Connectors
|
|
|
+
|
|
|
+### List
|
|
|
+
|
|
|
+Get the list of configured [connectors](/administrator/configuration/connectors/):
|
|
|
+
|
|
|
+ curl -L -X POST demo.gethue.com/desktop/connectors/api/instances
|
|
|
+
|
|
|
## Data Catalog
|
|
|
|
|
|
The [metadata API](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata) is powering the external [Catalog integrations](/user/browsing/#data-catalogs).
|
|
|
@@ -378,118 +502,6 @@ Adding/updating a comment with the dummy backend:
|
|
|
});
|
|
|
|
|
|
|
|
|
-## File Browsing
|
|
|
-
|
|
|
-### List
|
|
|
-
|
|
|
-Hue's [File Browser](https://docs.gethue.com/user/browsing/#data) offer uploads, downloads and listing of data in HDFS, S3, ADLS storages.
|
|
|
-
|
|
|
-Here is how to list the content of a path, here the S3 bucket `S3A://gethue-demo`:
|
|
|
-
|
|
|
- curl -X GET "https://demo.gethue.com/filebrowser/view=S3A://gethue-demo?pagesize=45&pagenum=1&filter=&sortby=name&descending=false&format=json" --cookie "csrftoken=oT8C5cQCbmpuoKcUZ2YaxybfLhtRShEO9UcvRWetx4HVatLuf6qicgJnbEHxfJNI;sessionid=nkblu68xfofabfsjctdwseaubfbkiwlg" -H "X-CSRFToken: oT8C5cQCbmpuoKcUZ2YaxybfLhtRShEO9UcvRWetx4HVatLuf6qicgJnbEHxfJNI"
|
|
|
-
|
|
|
- {
|
|
|
- ...........
|
|
|
- "files": [
|
|
|
- {
|
|
|
- "humansize": "0\u00a0bytes",
|
|
|
- "url": "/filebrowser/view=s3a%3A%2F%2Fdemo-hue",
|
|
|
- "stats": {
|
|
|
- "size": 0,
|
|
|
- "aclBit": false,
|
|
|
- "group": "",
|
|
|
- "user": "",
|
|
|
- "mtime": null,
|
|
|
- "path": "s3a://demo-hue",
|
|
|
- "atime": null,
|
|
|
- "mode": 16895
|
|
|
- },
|
|
|
- "name": "demo-hue",
|
|
|
- "mtime": "",
|
|
|
- "rwx": "drwxrwxrwx",
|
|
|
- "path": "s3a://demo-hue",
|
|
|
- "is_sentry_managed": false,
|
|
|
- "type": "dir",
|
|
|
- "mode": "40777"
|
|
|
- },
|
|
|
- {
|
|
|
- "humansize": "0\u00a0bytes",
|
|
|
- "url": "/filebrowser/view=S3A%3A%2F%2F",
|
|
|
- "stats": {
|
|
|
- "size": 0,
|
|
|
- "aclBit": false,
|
|
|
- "group": "",
|
|
|
- "user": "",
|
|
|
- "mtime": null,
|
|
|
- "path": "S3A://",
|
|
|
- "atime": null,
|
|
|
- "mode": 16895
|
|
|
- },
|
|
|
- "name": ".",
|
|
|
- "mtime": "",
|
|
|
- "rwx": "drwxrwxrwx",
|
|
|
- "path": "S3A://",
|
|
|
- "is_sentry_managed": false,
|
|
|
- "type": "dir",
|
|
|
- "mode": "40777"
|
|
|
- }
|
|
|
- ],
|
|
|
- ...........
|
|
|
- }
|
|
|
-
|
|
|
-### Get file content
|
|
|
-
|
|
|
-How to get the file content and its metadata. Here with the public file of demo.gethue.com [s3a://demo-hue/web_log_data/index_data.csv](https://demo.gethue.com/hue/filebrowser/view=s3a%3A%2F%2Fdemo-hue%2Fweb_log_data%2Findex_data.csv):
|
|
|
-
|
|
|
-**Note** It needs the `XMLHttpRequest` header to return the data in json:
|
|
|
-
|
|
|
- curl -X GET "https://demo.gethue.com/filebrowser/view=s3a://demo-hue/web_log_data/index_data.csv?offset=0&length=204800&compression=none&mode=text" --cookie "csrftoken=oT8C5cQCbmpuoKcUZ2YaxybfLhtRShEO9UcvRWetx4HVatLuf6qicgJnbEHxfJNI;sessionid=nkblu68xfofabfsjctdwseaubfbkiwlg" -H "X-CSRFToken: oT8C5cQCbmpuoKcUZ2YaxybfLhtRShEO9UcvRWetx4HVatLuf6qicgJnbEHxfJNI" -H "X-requested-with: XMLHttpRequest"
|
|
|
-
|
|
|
- {
|
|
|
- "show_download_button": true,
|
|
|
- "is_embeddable": false,
|
|
|
- "editable": false,
|
|
|
- "mtime": "October 31, 2016 03:34 PM",
|
|
|
- "rwx": "-rw-rw-rw-",
|
|
|
- "path": "s3a://demo-hue/web_log_data/index_data.csv",
|
|
|
- "stats": {
|
|
|
- "size": 6199593,
|
|
|
- "aclBit": false,
|
|
|
- ...............
|
|
|
- "contents": "code,protocol,request,app,user_agent_major,region_code,country_code,id,city,subapp,latitude,method,client_ip, user_agent_family,bytes,referer,country_name,extension,url,os_major,longitude,device_family,record,user_agent,time,os_family,country_code3
|
|
|
- 200,HTTP/1.1,GET /metastore/table/default/sample_07 HTTP/1.1,metastore,,00,SG,8836e6ce-9a21-449f-a372-9e57641389b3,Singapore,table,1.2931000000000097,GET,128.199.234.236,Other,1041,-,Singapore,,/metastore/table/default/sample_07,,103.85579999999999,Other,"demo.gethue.com:80 128.199.234.236 - - [04/May/2014:06:35:49 +0000] ""GET /metastore/table/default/sample_07 HTTP/1.1"" 200 1041 ""-"" ""Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)""
|
|
|
- ",Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org),2014-05-04T06:35:49Z,Other,SGP
|
|
|
- 200,HTTP/1.1,GET /metastore/table/default/sample_07 HTTP/1.1,metastore,,00,SG,6ddf6e38-7b83-423c-8873-39842dca2dbb,Singapore,table,1.2931000000000097,GET,128.199.234.236,Other,1041,-,Singapore,,/metastore/table/default/sample_07,,103.85579999999999,Other,"demo.gethue.com:80 128.199.234.236 - - [04/May/2014:06:35:50 +0000] ""GET /metastore/table/default/sample_07 HTTP/1.1"" 200 1041 ""-"" ""Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)""
|
|
|
- ",Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org),2014-05-04T06:35:50Z,Other,SGP
|
|
|
- ...............
|
|
|
- }
|
|
|
-
|
|
|
-## Data Importer
|
|
|
-
|
|
|
-### File import
|
|
|
-
|
|
|
-First guessing the format of the file `s3a://demo-hue/web_log_data/index_data.csv`:
|
|
|
-
|
|
|
- curl -X POST https://demo.gethue.com/indexer/api/indexer/guess_format --cookie "csrftoken=oT8C5cQCbmpuoKcUZ2YaxybfLhtRShEO9UcvRWetx4HVatLuf6qicgJnbEHxfJNI;sessionid=nkblu68xfofabfsjctdwseaubfbkiwlg" -H "X-CSRFToken: oT8C5cQCbmpuoKcUZ2YaxybfLhtRShEO9UcvRWetx4HVatLuf6qicgJnbEHxfJNI" --data 'fileFormat={"inputFormat":"file","path":"s3a://demo-hue/web_log_data/index_data.csv"}'
|
|
|
-
|
|
|
- {"status": 0, "fieldSeparator": ",", "hasHeader": true, "quoteChar": "\"", "recordSeparator": "\\n", "type": "csv"}
|
|
|
-
|
|
|
-Then getting some data sample as well as the column types (column names will be picked from the header line if present):
|
|
|
-
|
|
|
- curl -X POST https://demo.gethue.com/indexer/api/indexer/guess_field_types --cookie "csrftoken=oT8C5cQCbmpuoKcUZ2YaxybfLhtRShEO9UcvRWetx4HVatLuf6qicgJnbEHxfJNI;sessionid=nkblu68xfofabfsjctdwseaubfbkiwlg" -H "X-CSRFToken: oT8C5cQCbmpuoKcUZ2YaxybfLhtRShEO9UcvRWetx4HVatLuf6qicgJnbEHxfJNI" --data 'fileFormat={"inputFormat":"file","path":"s3a://demo-hue/web_log_data/index_data.csv","format":{"type":"csv","fieldSeparator":",","recordSeparator":"\\n","quoteChar":"\"","hasHeader":true,"status":0}}'
|
|
|
-
|
|
|
- {
|
|
|
- "sample": [["200", "HTTP/1.1", "GET /metastore/table/default/sample_07 HTTP/1.1", "metastore", "", "00", "SG", "8836e6ce-9a21-449f-a372-9e57641389b3", "Singapore", "table", "1.2931000000000097", "GET", "128.199.234.236", "Other", "1041", "-", "Singapore", "", "/metastore/table/default/sample_07", "", "103.85579999999999", "Other", "demo.gethue.com:80 128.199.234.236 - - [04/May/2014:06:35:49 +0000] \"GET /metastore/table/default/sample_07 HTTP/1.1\" 200 1041 \"-\" \"Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)\"\n", "Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)", "2014-05-04T06:35:49Z", "Other", "SGP"],
|
|
|
- ....
|
|
|
- "columns": [{"operations": [], "comment": "", "nested": [], "name": "code", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "long", "showProperties": false, "scale": 0}, {"operations": [], "comment": "", "nested": [], "name": "protocol", "level": 0, "keyType": "string", "required": false, "precision": 10, "keep": true, "isPartition": false, "length": 100, "partitionValue": "", "multiValued": false, "unique": false, "type": "string", "showProperties": false, "scale": 0},
|
|
|
- .....
|
|
|
- }
|
|
|
-
|
|
|
-Then we submit via `https://demo.gethue.com/indexer/api/importer/submit` and provide the `source` and `destination` parameters. We get back an `operation id` (i.e. some SQL Editor query history id).
|
|
|
-
|
|
|
-If the `show_command` parameter is given, the API call will instead return the generated SQL queries that will import the data.
|
|
|
-
|
|
|
- {"status": 0, "history_id": 17820, "handle": {"statement_id": 0, "session_type": "hive", "has_more_statements": true, "guid": "uu6K3SSWSY6mx/fbh0nm2w==\n", "previous_statement_hash": "4bee3a62b3c7142c60475021469483bff81ba09bd07b8e527179e617", "log_context": null, "statements_count": 4, "end": {"column": 53, "row": 0}, "session_id": 55, "start": {"column": 0, "row": 0}, "secret": "8mKu1bhdRtWXu82DXjDZdg==\n", "has_result_set": false, "session_guid": "fd4c667f3a5e4507:0335af7716db3d9e", "statement": "DROP TABLE IF EXISTS `default`.`hue__tmp_index_data`", "operation_type": 0, "modified_row_count": null}, "history_uuid": "bf5804f5-6f12-47a8-8ba6-0ed7032ebe93"}
|
|
|
|
|
|
# Python
|
|
|
|