--- title: Introducing Amazon S3 Support in Hue author: admin type: post date: 2016-08-25T08:15:20+00:00 url: /introducing-s3-support-in-hue/ sf_no_breadcrumbs: - 1 sf_page_title_style: - standard sf_page_title: - 1 sf_detail_type: - none sf_thumbnail_type: - none sf_thumbnail_link_type: - link_to_post sf_page_title_bg: - none sf_page_title_text_style: - light sf_background_image_size: - cover sf_social_sharing: - 1 sf_sidebar_config: - left-sidebar sf_left_sidebar: - Sidebar-2 sf_right_sidebar: - Sidebar-1 sf_caption_position: - caption-right sf_remove_promo_bar: - 1 categories: - Browsing ---
We’re very excited to officially introduce Amazon S3 (Amazon Simple Storage Service) integration in Hue with Hue’s 3.11 release. Hue can be setup to read and write to a configured S3 account, and users can directly query from and save data to S3 without any intermediate moving/copying to HDFS.
{{< youtube P-7XH78aT9s >}} ## S3 Configuration in Hue {.p3}Hue’s filebrowser can now allow users to explore, manage, and upload data in an S3 account, in addition to HDFS.
In order to add an S3 account to Hue, you’ll need to configure Hue with valid S3 credentials, including the access key ID and secret access key: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html
These keys can securely stored in a script that outputs the actual access key and secret key to stdout to be read by Hue (this is similar to how Hue reads password scripts). In order to use script files, add the following section to your hue.ini configuration file:
[aws]
[[aws_accounts]]
[[[default]]]
access_key_id_script=/path/to/access_key_script
secret_access_key_script= /path/to/secret_key_script
allow_environment_credentials=false
region=us-east-1
Alternatively (but not recommended for production or secure environments), you can set the access_key_id and secret_access_key values to the plain-text values of your keys:
[aws]
[[aws_accounts]]
[[[default]]]
access_key_id=s3accesskeyid
secret_access_key=s3secretaccesskey
allow_environment_credentials=false
region=us-east-1
The region should be set to the AWS region corresponding to the S3 account. By default, this region will be set to ‘us-east-1’.
### Integrating Hadoop with S3 {.p3}
In addition to configuring Hue with your S3 credentials, Hadoop will also need to be configured with the S3 authentication credentials in order to read from and save to S3. This can be done by setting the following properties in your core-site.xml file:
fs.s3a.awsAccessKeyId
AWS access key ID
fs.s3a.awsSecretAccessKey
AWS secret key
For more information see http://wiki.apache.org/hadoop/AmazonS3
With Hue and Hadoop configured, we can verify that Hue is able to successfully connect to your S3 account by restarting Hue and checking the configuration page. You should not see any errors related to AWS, and you should notice an additional dropdown option in the Filebrowser menu from the main navigation:
[
][1]
## {.p3}
## Exploring S3 in Hue’s Filebrowser {.p3}
Once Hue is successfully configured to connect to S3, we can view all accessible buckets within the account by clicking on the S3 root.
Users can also create new buckets or delete existing buckets from this view.
#### NOTE: Unique Bucket Names ❗️ S3 bucket names must be unique _across all regions_. Hue will raise an error if you attempt to create or rename a bucket with a reserved name.However, in most cases users will be working directly with keys within a bucket. From the buckets-view, users can click on a bucket to expand its contents. From here, we can view the existing keys (both directories and files) and create, rename, move, copy, or delete existing directories and files. Additionally, we can directly upload files to S3