Browse Source

[oozie] Don't encode request body with latin-1 for HDFS calls in Py3 for workflow shell script having Korean chars (#3159)

## What changes were proposed in this pull request?

- For Py3 Hue, when the browser language is set to Korean and someone tries to run an oozie workflow for a shell script containing Korean chars, it was failing with the below error:

```
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 134-135: Body ('작업') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
```

- Because of i18n, the xml data is having Korean chars when the browser language is Korean which was not utf-8 encoded. So we are encoding it properly and sending the xml data as bytes instead of str type.

NOTE: This issue is not seen in Py2 Hue.

## Steps to reproduce the issue or verify the fix

- Change the chrome browser language to Korean
- Create a new workflow with shell action
- Use any shell script with a simple "echo 안녕하세요"
- Add a workflow description in English or skip so that it takes a default description
- Submit the workflow

## How was this patch tested?

- E2E manual testing in Hue of a Py3 enabled cluster.
Harsh Gupta 2 years ago
parent
commit
decf994b4b
1 changed files with 8 additions and 2 deletions
  1. 8 2
      desktop/libs/liboozie/src/liboozie/submission2.py

+ 8 - 2
desktop/libs/liboozie/src/liboozie/submission2.py

@@ -616,10 +616,16 @@ STORED AS TEXTFILE %s""" % (self.properties.get('send_result_path'),
 
   def _create_file(self, deployment_dir, file_name, data, do_as=False):
     file_path = self.fs.join(deployment_dir, file_name)
+
+    # In Py3 because of i18n, the xml data is not properly utf-8 encoded for some languages.
+    # This can later throw UnicodeEncodeError exception for request body in HDFS or other FS API calls. To tackle this,
+    # We are converting the data into bytes by utf-8 encoding instead of str type.
+    data = smart_str(data).encode('utf-8') if sys.version_info[0] > 2 else smart_str(data)
+
     if do_as:
-      self.fs.do_as_user(self.user, self.fs.create, file_path, overwrite=True, permission=0o644, data=smart_str(data))
+      self.fs.do_as_user(self.user, self.fs.create, file_path, overwrite=True, permission=0o644, data=data)
     else:
-      self.fs.create(file_path, overwrite=True, permission=0o644, data=smart_str(data))
+      self.fs.create(file_path, overwrite=True, permission=0o644, data=data)
     LOG.debug("Created/Updated %s" % (file_path,))
 
   def _generate_altus_action_script(self, service, command, arguments, auth_key_id, auth_key_secret):