Przeglądaj źródła

HUE-8888 [api] Documenting how to import parsers

Romain 5 lat temu
rodzic
commit
8eca90edbc

+ 103 - 1
docs/docs-site/content/developer/api/_index.md

@@ -5,7 +5,109 @@ draft: false
 weight: 5
 ---
 
-Hue can be accessed directly via a Django Python Shell or by its REST API.
+Hue can be accessed directly via a Python Shell or by its REST API. Some components of Hue like the SQL parsers can also be imported independently.
+
+
+## SQL autocompletion
+
+The parser is running on the client side and comes with just a few MBs of Javascript that are cached by the browser. This provides a very reactive experience to the end user and allow to import it as a module.
+
+While the dynamic content like the list of tables, columns is obviously fetched via a remote endpoint, all the SQL knowledge of the statements is available.
+
+See the currently shipped [SQL dialects](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/js/parse/sql).
+
+### npm package
+
+What if I only want to use only the autocomplete as a Javascript module in my own app?
+
+Importing the Parser can be simply done as a npm package. Here is an example on how to use the parser in a node.js app:
+
+    cd tools/parser/hue_dep
+    npm install
+    npm run webpack
+    npm run app
+
+In `package.json` there’s a dependency on Hue:
+
+    "dependencies": {
+      "hue": "file:../../.."
+    },
+
+Note that it can also be a GitHub link, e.g. "hue": "git://github.com/cloudera/hue.git” but it takes a bit longer to do `npm install`.
+
+Now let's import the Hive parser and run it on an SQL statement:
+
+    import sqlAutocompleteParser from 'hue/desktop/core/src/desktop/js/parse/sql/hive/hiveAutocompleteParser';
+
+    const beforeCursor = 'SELECT col1, col2, tbl2.col3 FROM tbl; '; // Note extra space at end
+    const afterCursor = '';
+    const dialect = 'hive';
+    const debug = false;
+
+    console.log(
+      JSON.stringify(
+        sqlAutocompleteParser.parseSql(beforeCursor, afterCursor, dialect, debug),
+        null,
+        2
+      )
+    );
+
+Which then will output keywords suggestions and all the known locations:
+
+    { locations:
+      [ { type: 'statement', location: [Object] },
+        { type: 'statementType',
+          location: [Object],
+          identifier: 'SELECT' },
+        { type: 'selectList', missing: false, location: [Object] },
+        { type: 'column',
+          location: [Object],
+          identifierChain: [Array],
+          qualified: false,
+          tables: [Array] },
+        { type: 'column',
+          location: [Object],
+          identifierChain: [Array],
+          qualified: false,
+          tables: [Array] },
+        { type: 'column',
+          location: [Object],
+          identifierChain: [Array],
+          qualified: false,
+          tables: [Array] },
+        { type: 'table', location: [Object], identifierChain: [Array] },
+        { type: 'whereClause', missing: true, location: [Object] },
+        { type: 'limitClause', missing: true, location: [Object] } ],
+      lowerCase: false,
+      suggestKeywords:
+      [ { value: 'ABORT', weight: -1 },
+        { value: 'ALTER', weight: -1 },
+        { value: 'ANALYZE TABLE', weight: -1 },
+        { value: 'CREATE', weight: -1 },
+        { value: 'DELETE', weight: -1 },
+        { value: 'DESCRIBE', weight: -1 },
+        { value: 'DROP', weight: -1 },
+        { value: 'EXPLAIN', weight: -1 },
+        { value: 'EXPORT', weight: -1 },
+        { value: 'FROM', weight: -1 },
+        { value: 'GRANT', weight: -1 },
+        { value: 'IMPORT', weight: -1 },
+        { value: 'INSERT', weight: -1 },
+        { value: 'LOAD', weight: -1 },
+        { value: 'MERGE', weight: -1 },
+        { value: 'MSCK', weight: -1 },
+        { value: 'RELOAD FUNCTION', weight: -1 },
+        { value: 'RESET', weight: -1 },
+        { value: 'REVOKE', weight: -1 },
+        { value: 'SELECT', weight: -1 },
+        { value: 'SET', weight: -1 },
+        { value: 'SHOW', weight: -1 },
+        { value: 'TRUNCATE', weight: -1 },
+        { value: 'UPDATE', weight: -1 },
+        { value: 'USE', weight: -1 },
+        { value: 'WITH', weight: -1 } ],
+      definitions: [] }
+
 
 ## REST
 

+ 10 - 3
docs/docs-site/content/developer/parsers/_index.md

@@ -7,14 +7,21 @@ weight: 2
 
 This guide goes you through the steps necessary to create an autocompleter for any [SQL dialect](/administrator/configuration/connectors/#databases) in Hue. The major benefits are:
 
-* Showing only valid syntax in the autocomplete
-* Getting the list of tables, columns automatically
-* Proper syntax highlighting of the keywords
+* Proposing only valid syntax in the autocomplete
+* Getting the list of tables, columns, UDFs... automatically
+* Suggesting fixes
 
 ## Parser Theory
 
 There are several parsers in Hue already (e.g. one for Impala, one for Hive..) and a generic SQL that is used for other dialects. The parsers are written using a [bison](https://www.gnu.org/software/bison/) grammar and are generated with [jison](https://github.com/zaach/jison). They arere 100% Javascript and live on the client side, this gives the performance of a desktop editor in your browser.
 
+Building a dedicated work is more effort but it then allows a very rich end user experience, e.g.:
+
+* Handle invalid/imcomplete queries and propose suggestions/fixes
+* date_column = <Date compatible UDF ...>
+* Language reference or data samples just by pointing the cursor on SQL identifiers
+* Leverage the parser for risk alerts (e.g. adding automatic LIMIT) or proper re-formatting
+
 ### Structure
 
 Normally parsers generate a parse tree but for our purposes we don’t really care about the statement itself but rather about what should happen when parts of a particular statement is encountered. During parsing the state is kept outside the parse tree and in case of syntax errors this enables us to provide some results up to the point of the error. There are two ways that incomplete/erroneous statements are handled, first we try to define most of the incomplete grammar and secondly we rely on the “error” token which allows the parser to recover.

+ 1 - 1
tools/parser/hue_dep/package.json

@@ -11,7 +11,7 @@
   "author": "Queso",
   "license": "ISC",
   "dependencies": {
-    "hue": "file:../../../hue"
+    "hue": "file:../../.."
   },
   "devDependencies": {
     "@babel/cli": "^7.2.3",

+ 1 - 1
tools/parser/hue_dep/src/app.js

@@ -3,7 +3,7 @@ import sqlAutocompleteParser from 'hue/desktop/core/src/desktop/js/parse/sql/hiv
 
 const beforeCursor = 'SELECT col1, col2, tbl2.col3 FROM tbl; '; // Note extra space at end
 const afterCursor = '';
-const dialect = 'impala';
+const dialect = 'hive';
 const debug = false;
 
 console.log(