_index.md 7.7 KB


title: "Data Catalog" date: 2019-03-13T18:28:09-07:00 draft: false

weight: 4

Hue Browsers power the Data Catalog. They let you easily search, glance and perform actions on data or jobs in Cloud or on premise clusters.

The browsers can be "enriched" with Search and Tagging by metadata services.

SQL Tables

The Table Browser enables you to manage the databases, tables, and partitions of the metastore shared by the Hive and Impala. You can perform the following operations:

  • Search and display metadata like tags and additional description from Catalog backends.

  • Databases

    • Select a database
    • Create a database
    • Drop databases
  • Tables

Data Importer

The goal of the importer is to allow ad-hoc queries on data not yet in the clusters and simplifies self-service analytics.

If you want to import your own data instead of installing the sample tables, open the importer from the left menu or from the little + in the left assist.

To learn more, watch the video on Data Import Wizard.

Note Files can be dragged & dropped, selected from HDFS or S3 (if configured), and their formats are automatically detected. The wizard also assists when performing advanced functionalities like table partitioning, Kudu tables, and nested types.

Traditional Databases

Read more about ingesting data from traditional databases.

Indexing

In the past, indexing data into Solr to then explore it with a Dynamic Dashboard has been quite difficult. The task involved writing a Solr schema and a Morphlines file then submitting a job to YARN to do the indexing. Often times getting this correct for non trivial imports could take a few days of work. Now with Hue's new feature you can start your YARN indexing job in minutes. This tutorial offers a step by step guide on how to do it.

Read more about it here.

Dashboards

Dashboards are an interactive way to explore your data quickly and easily. No programming is required and the analysis is done by drag & drops and clicks.

Read more about Dashboards.

Concepts

Simply drag & drop widgets that are interconnected together. This is great for exploring new datasets or monitoring without having to type.

Querying

The search box support live prefix filtering of field data and comes with a Solr syntax autocomplete in order to make the querying intuitive and quick. Any field can be inspected for its top values of statistic. This analysis happens very fast as the data is indexed.

Autocomplete

The top search bar offers a full autocomplete on all the values of the index.

More Like This

The “More like This” feature lets you selected fields you would like to use to find similar records. This is a great way to find similar issues, customers, people... with regard to a list of attributes.

Files

The File Browser application lets you interact with these file systems HDFS, S3 or ADLS:

  • Create files and directories, upload and download files, upload zip archives and extract them, rename, move, and delete files and directories.
  • Change a file's or directory's owner, group, and permissions.
  • View and edit files as text or binary.
  • Create external tables or export query results

HDFS

Hue is fully compatible with HDFS and is handy for browsing, peeking at file content, upload or downloading data.

S3

Hue can be setup to read and write to a configured S3 account, and users get autocomplete capabilities and can directly query from and save data to S3 without any intermediate moving/copying to HDFS.

Read more about it.

ADLS

Learn more about it on the ADLS integration post.

Note ADLS gen2 is currently not supported.

GFS

Google file system is currently not supported.

Solr Indexes / Collections

Solr indexes can be created and are listed in the interface.

Sentry Permissions

Sentry roles and privileges can directly be edited in the Security interface.

Note Sentry is going to be replaced by Apache Ranger in HUE-8748.

SQL

Hive UI.

Solr

Solr privileges can be edited directly via the interface.

For listing collections, query and creating collection:

Admin=*->action=*
Collection=*->action=*
Schema=*->action=*
Config=*->action=*

Kafka

Kafka topics can be listed.

Note This is currently an experimental feature.

Jobs

The Job Browser application lets you to examine multiple types of jobs jobs running in the cluster. Job Browser presents the job and tasks in layers for quick access to the logs and troubleshooting.

YARN (Spark, MapReduce, Tez)

Any job running on the Resource Manager will be automatically listed. The information will be fetched accordingly if the job got moved to one of the history servers.

Impala Queries

There are three ways to access the Query browser:

  • Best: Click on the query ID after executing a SQL query in the editor. This will open the mini job browser overlay at the current query. Having the query execution information side by side the SQL editor is especially helpful to understand the performance characteristics of your queries.
  • Open the mini job browser overlay and navigate to the queries tab.
  • Open the job browser and navigate to the queries tab.

Query capabilities

  • Display the list of currently running queries on the user's current Impala coordinator and a certain number of completed queries based on your configuration (25 by default).
  • Display the summary report which shows physical timing and memory information of each operation of the explain plan. You can quickly find bottlenecks in the execution of the query which you can resolve by replacing expensive operations, repartitioning, changing file format or moving data.
  • Display the query plan which is a condensed version of the summary report in graphical form
  • Display the memory profile which contains information about the memory usage during the execution of the query. You can use this to determine if the memory available to your query is sufficient.
  • Display the profile which gives you physical execution of the query in great detail. This view is used to analyze data exchange between the various operator and the performance of the IO (disk, network, CPU). You can use this to reorganize the location of your data (on disk, in memory, different partitions or file formats).
  • Manually close an opened query.

Read more about it on Browsing Impala Query Execution within the SQL Editor .

Workflow / Schedules (Oozie)

List submitted workflows, schedules and bundles.

Livy / Spark

List Livy sessions and submitted statements.