Browse Source

[sparksql] Clean unused session from /fetch_result_data call (#2867)

- /fetch_result_data is called only once after /execute call (unlike /check_status or /get_logs).
- We can leverage the call for cleaning up any unused sessions specially the dead ones which are created after cancelling the executed query statement and retrying.
Harsh Gupta 3 years ago
parent
commit
fdf927d55e
1 changed files with 6 additions and 0 deletions
  1. 6 0
      desktop/libs/notebook/src/notebook/connectors/spark_shell.py

+ 6 - 0
desktop/libs/notebook/src/notebook/connectors/spark_shell.py

@@ -234,6 +234,12 @@ class SparkApi(Api):
     session = self._handle_session_health_check(session)
 
     response = self._fetch_result(api, session, cell, start_over)
+
+    # Close unused sessions if there are any.
+    # Clean here since /fetch_result_data is called only once after the /execute call
+    if self._get_session_info_from_user():
+      self._close_unused_sessions()
+
     return response