Quellcode durchsuchen

[lint] Remove pylint and switch to ruff for Hue backend linting (#3703)

## What changes were proposed in this pull request?

- Current pylint setup in Hue was linting staged changes only by checking for 3 rules.
- Also pylint is comparatively slow and GPL licensed.


- Switching to ruff has its own perks which are mentioned on: https://docs.astral.sh/ruff/ and it's MIT licensed.
- For starters, it's waaayyy faster, can auto-fix errors and work as both linter and formatter replacing a lot of latest packages (like a all-in-one solution).
- It is also supporting pylint rules which is WIP: https://github.com/astral-sh/ruff/issues/970


- Currently this PR only targets the minimal transition from pylint to ruff. Iteratively, we will extend the rules to improve the Hue codebase, add its pre-commit and play around auto-fixing errors capability. This will also help in making our codebase more PEP8 compliant.

#### Note
- Ruff still does not support pylint's `W0311(bad-indentation)` rule which was present earlier with pylint in Hue. We will update it later when the support lands in a future release.

## How was this patch tested?

- Manually in local setup.
Harsh Gupta vor 1 Jahr
Ursprung
Commit
264a267b22

+ 0 - 1
.circleci/config.yml

@@ -54,7 +54,6 @@ commands:
       - run:
           name: run python lints
           command: |
-            ./build/env/bin/pip install pylint==2.5.3 pylint-django==2.3.0 configparser==5.3.0
             ./tools/ci/check_for_python_lint.sh
 
       - run:

+ 0 - 1
.github/workflows/commitflow-py3.yml

@@ -60,7 +60,6 @@ jobs:
 
     - name: run python lints
       run: |
-        ./build/env/bin/pip install pylint==2.5.3 pylint-django==2.3.0 configparser==5.3.0
         ./tools/ci/check_for_python_lint.sh
 
     - name: run documentation lints

+ 0 - 380
.pylintrc

@@ -1,380 +0,0 @@
-[MASTER]
-
-# Specify a configuration file.
-#rcfile=
-
-# Python code to execute, usually for sys.path manipulation such as
-# pygtk.require().
-#init-hook=
-
-# Add files or directories to the blacklist. They should be base names, not
-# paths.
-ignore=CVS
-
-# Pickle collected data for later comparisons.
-persistent=yes
-
-# List of plugins (as comma separated values of python modules names) to load,
-# usually to register additional checkers.
-load-plugins=
-
-# Use multiple processes to speed up Pylint.
-jobs=1
-
-# Allow loading of arbitrary C extensions. Extensions are imported into the
-# active Python interpreter and may run arbitrary code.
-unsafe-load-any-extension=no
-
-# A comma-separated list of package or module names from where C extensions may
-# be loaded. Extensions are loading into the active Python interpreter and may
-# run arbitrary code
-extension-pkg-whitelist=
-
-
-[MESSAGES CONTROL]
-
-# Only show warnings with the listed confidence levels. Leave empty to show
-# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED
-confidence=
-
-# Enable the message, report, category or checker with the given id(s). You can
-# either give multiple identifier separated by comma (,) or put this option
-# multiple time. See also the "--disable" option for examples.
-
-# For some reason not picked-up, please keep in sync and see the `RULES` list in
-# https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/management/commands/runpylint.py#L30
-# enable=
-  # C0326(bad-whitespace)
-  # W0311(bad-indentation)
-  # C0301(line-too-long)
-
-
-# Disable the message, report, category or checker with the given id(s). You
-# can either give multiple identifiers separated by comma (,) or put this
-# option multiple times (only on the command line, not in the configuration
-# file where it should appear only once).You can also use "--disable=all" to
-# disable everything first and then reenable specific checks. For example, if
-# you want to run only the similarities checker, you can use "--disable=all
-# --enable=similarities". If you want to run only the classes checker, but have
-# no Warning level messages displayed, use"--disable=all --enable=classes
-# --disable=W"
-disable=all
-
-  #attribute-defined-outside-init,
-  #duplicate-code,
-  #invalid-name,
-  #missing-docstring,
-  #protected-access,
-  #too-few-public-methods,
-  # handled by black
-  #format
-
-
-[REPORTS]
-
-# Set the output format. Available formats are text, parseable, colorized, msvs
-# (visual studio) and html. You can also give a reporter class, eg
-# mypackage.mymodule.MyReporterClass.
-output-format=text
-
-# Put messages in a separate file for each module / package specified on the
-# command line instead of printing them on stdout. Reports (if any) will be
-# written in a file name "pylint_global.[txt|html]".
-files-output=no
-
-# Tells whether to display a full report or only the messages
-reports=no
-
-# Python expression which should return a note less than 10 (10 is the highest
-# note). You have access to the variables errors warning, statement which
-# respectively contain the number of errors / warnings messages and the total
-# number of statements analyzed. This is used by the global evaluation report
-# (RP0004).
-evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)
-
-# Template used to display messages. This is a python new-style format string
-# used to format the message information. See doc for all details
-#msg-template=
-
-
-[LOGGING]
-
-# Logging modules to check that the string format arguments are in logging
-# function parameter format
-logging-modules=logging
-
-
-[MISCELLANEOUS]
-
-# List of note tags to take in consideration, separated by a comma.
-notes=FIXME,XXX,TODO
-
-
-[SIMILARITIES]
-
-# Minimum lines number of a similarity.
-min-similarity-lines=4
-
-# Ignore comments when computing similarities.
-ignore-comments=yes
-
-# Ignore docstrings when computing similarities.
-ignore-docstrings=yes
-
-# Ignore imports when computing similarities.
-ignore-imports=no
-
-
-[VARIABLES]
-
-# Tells whether we should check for unused import in __init__ files.
-init-import=yes
-
-# A regular expression matching the name of dummy variables (i.e. expectedly
-# not used).
-dummy-variables-rgx=_$|dummy
-
-# List of additional names supposed to be defined in builtins. Remember that
-# you should avoid defining new builtins when possible.
-additional-builtins=
-
-# List of strings which can identify a callback function by name. A callback
-# name must start or end with one of those strings.
-callbacks=cb_,_cb
-
-
-[FORMAT]
-
-# Maximum number of characters on a single line.
-max-line-length=140
-
-# Regexp for a line that is allowed to be longer than the limit.
-ignore-long-lines=^\s*(# )?<?https?://\S+>?$
-
-# Allow the body of an if to be on the same line as the test if there is no
-# else.
-single-line-if-stmt=no
-
-# List of optional constructs for which whitespace checking is disabled
-# no-space-check=trailing-comma,dict-separator
-no-space-check=
-
-# Maximum number of lines in a module
-max-module-lines=2000
-
-# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
-# tab).
-indent-string='  '
-
-# Number of spaces of indent required inside a hanging or continued line.
-indent-after-paren=4
-
-# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.
-expected-line-ending-format=
-
-
-[BASIC]
-
-# List of builtins function names that should not be used, separated by a comma
-bad-functions=map,filter,input
-
-# Good variable names which should always be accepted, separated by a comma
-good-names=i,j,k,ex,Run,_
-
-# Bad variable names which should always be refused, separated by a comma
-bad-names=foo,bar,baz,toto,tutu,tata
-
-# Colon-delimited sets of names that determine each other's naming style when
-# the name regexes allow several styles.
-name-group=
-
-# Include a hint for the correct naming format with invalid-name
-include-naming-hint=yes
-
-# Regular expression matching correct function names
-function-rgx=[a-z_][a-z0-9_]{2,30}$
-
-# Naming hint for function names
-function-name-hint=[a-z_][a-z0-9_]{2,30}$
-
-# Regular expression matching correct variable names
-variable-rgx=[a-z_][a-z0-9_]{2,30}$
-
-# Naming hint for variable names
-variable-name-hint=[a-z_][a-z0-9_]{2,30}$
-
-# Regular expression matching correct constant names
-const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$
-
-# Naming hint for constant names
-const-name-hint=(([A-Z_][A-Z0-9_]*)|(__.*__))$
-
-# Regular expression matching correct attribute names
-attr-rgx=[a-z_][a-z0-9_]{2,}$
-
-# Naming hint for attribute names
-attr-name-hint=[a-z_][a-z0-9_]{2,}$
-
-# Regular expression matching correct argument names
-argument-rgx=[a-z_][a-z0-9_]{2,30}$
-
-# Naming hint for argument names
-argument-name-hint=[a-z_][a-z0-9_]{2,30}$
-
-# Regular expression matching correct class attribute names
-class-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$
-
-# Naming hint for class attribute names
-class-attribute-name-hint=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$
-
-# Regular expression matching correct inline iteration names
-inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$
-
-# Naming hint for inline iteration names
-inlinevar-name-hint=[A-Za-z_][A-Za-z0-9_]*$
-
-# Regular expression matching correct class names
-class-rgx=[A-Z_][a-zA-Z0-9]+$
-
-# Naming hint for class names
-class-name-hint=[A-Z_][a-zA-Z0-9]+$
-
-# Regular expression matching correct module names
-module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
-
-# Naming hint for module names
-module-name-hint=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
-
-# Regular expression matching correct method names
-method-rgx=[a-z_][a-z0-9_]{2,}$
-
-# Naming hint for method names
-method-name-hint=[a-z_][a-z0-9_]{2,}$
-
-# Regular expression which should only match function or class names that do
-# not require a docstring.
-no-docstring-rgx=__.*__
-
-# Minimum line length for functions/classes that require docstrings, shorter
-# ones are exempt.
-docstring-min-length=-1
-
-# List of decorators that define properties, such as abc.abstractproperty.
-property-classes=abc.abstractproperty
-
-
-[TYPECHECK]
-
-# Tells whether missing members accessed in mixin class should be ignored. A
-# mixin class is detected if its name ends with "mixin" (case insensitive).
-ignore-mixin-members=yes
-
-# List of module names for which member attributes should not be checked
-# (useful for modules/projects where namespaces are manipulated during runtime
-# and thus existing member attributes cannot be deduced by static analysis
-ignored-modules=
-
-# List of classes names for which member attributes should not be checked
-# (useful for classes with attributes dynamically set).
-ignored-classes=SQLObject, optparse.Values, thread._local, _thread._local
-
-# List of members which are set dynamically and missed by pylint inference
-# system, and so shouldn't trigger E1101 when accessed. Python regular
-# expressions are accepted.
-generated-members=REQUEST,acl_users,aq_parent
-
-# List of decorators that create context managers from functions, such as
-# contextlib.contextmanager.
-contextmanager-decorators=contextlib.contextmanager
-
-
-[SPELLING]
-
-# Spelling dictionary name. Available dictionaries: none. To make it working
-# install python-enchant package.
-spelling-dict=
-
-# List of comma separated words that should not be checked.
-spelling-ignore-words=
-
-# A path to a file that contains private dictionary; one word per line.
-spelling-private-dict-file=
-
-# Tells whether to store unknown words to indicated private dictionary in
-# --spelling-private-dict-file option instead of raising a message.
-spelling-store-unknown-words=no
-
-
-[DESIGN]
-
-# Maximum number of arguments for function / method
-max-args=10
-
-# Argument names that match this expression will be ignored. Default to name
-# with leading underscore
-ignored-argument-names=_.*
-
-# Maximum number of locals for function / method body
-max-locals=25
-
-# Maximum number of return / yield for function / method body
-max-returns=11
-
-# Maximum number of branch for function / method body
-max-branches=26
-
-# Maximum number of statements in function / method body
-max-statements=100
-
-# Maximum number of parents for a class (see R0901).
-max-parents=7
-
-# Maximum number of attributes for a class (see R0902).
-max-attributes=11
-
-# Minimum number of public methods for a class (see R0903).
-min-public-methods=2
-
-# Maximum number of public methods for a class (see R0904).
-max-public-methods=25
-
-
-[CLASSES]
-
-# List of method names used to declare (i.e. assign) instance attributes.
-defining-attr-methods=__init__,__new__,setUp,__post_init__
-
-# List of valid names for the first argument in a class method.
-valid-classmethod-first-arg=cls
-
-# List of valid names for the first argument in a metaclass class method.
-valid-metaclass-classmethod-first-arg=mcs
-
-# List of member names, which should be excluded from the protected access
-# warning.
-exclude-protected=_asdict,_fields,_replace,_source,_make
-
-
-[IMPORTS]
-
-# Deprecated modules which should not be used, separated by a comma
-deprecated-modules=regsub,TERMIOS,Bastion,rexec
-
-# Create a graph of every (i.e. internal and external) dependencies in the
-# given file (report RP0402 must not be disabled)
-import-graph=
-
-# Create a graph of external dependencies in the given file (report RP0402 must
-# not be disabled)
-ext-import-graph=
-
-# Create a graph of internal dependencies in the given file (report RP0402 must
-# not be disabled)
-int-import-graph=
-
-
-[EXCEPTIONS]
-
-# Exceptions that will emit a warning when being caught. Defaults to
-# "Exception"
-overgeneral-exceptions=Exception

+ 0 - 1
Makefile.tarball

@@ -58,7 +58,6 @@ define remove_devtree_exclusions
 		      -name '.gitignore' -o \
 		      -name '.*~' -o \
 		      -name '.#*' -o \
-		      -name '.pylintrc' -o \
 		      -name 'build' -o \
 		      -name 'logs' -o \
 		      -name 'tags' -o \

+ 1 - 2
desktop/core/base_requirements.txt

@@ -46,8 +46,6 @@ prompt-toolkit==3.0.39
 protobuf==3.20.3
 py==1.11.0
 pyformance==0.3.2
-pylint==2.6.0
-pylint-django==2.3.0
 pytest==8.1.1
 pytest-django==4.8.0
 python-dateutil==2.8.2
@@ -61,6 +59,7 @@ PyJWT==2.4.0
 PyYAML==6.0.1
 requests-kerberos==0.12.0
 rsa==4.7.2
+ruff==0.3.7
 sasl==0.3.1  # Move to https://pypi.org/project/sasl3/ ?
 slack-sdk==3.2.0
 SQLAlchemy==1.3.8

+ 0 - 106
desktop/core/src/desktop/management/commands/runpylint.py

@@ -1,106 +0,0 @@
-#!/usr/bin/env python
-# Licensed to Cloudera, Inc. under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  Cloudera, Inc. licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-import os.path
-import subprocess
-import sys
-
-from django.core.management.base import BaseCommand, CommandError
-from django.conf import settings
-
-if sys.version_info[0] > 2:
-  from django.utils.translation import gettext as _
-else:
-  from django.utils.translation import ugettext as _
-
-from desktop.lib import paths
-
-
-RULES = [
-  'C0326(bad-whitespace)',
-  'W0311(bad-indentation)',
-  'C0301(line-too-long)'
-]
-
-
-class Command(BaseCommand):
-  help = _("""
-  Runs pylint on desktop and app code.
-
-  With no arguments, or with "all", this will run pylint on all installed apps. Otherwise, specify a list of files
-  or modules to run, as well as other parameters to pylint.
-  Note that you'll want to preface the section of pylint arguments with "--" so Django's manage.py passes them along.
-
-  Examples:
-    python core/manage.py runpylint all -- -f parseable
-    python core/manage.py runpylint --files="apps/jobbrowser/src/jobbrowser/apis/base_api.py desktop/libs/notebook/src/notebook/api.py"
-    python core/manage.py runpylint filebrowser
-    python core/manage.py runpylint
-  """)
-
-  def valid_app(self):
-    from desktop import appmanager
-    apps = ["desktop"]
-    for app in appmanager.DESKTOP_APPS:
-      apps.append(app.name)
-    return apps
-
-  def add_arguments(self, parser):
-    parser.add_argument('-f', '--force', dest='force', default='true', action="store_true")
-    parser.add_argument('--output-format', action='store', dest='outputformat', default='parseable')
-    parser.add_argument('-a', '--app', dest='app', action='store', default='all', choices=self.valid_app())
-    parser.add_argument('-F', '--files', dest='files', action='store', default=None)
-
-  def handle(self, *args, **options):
-    """Check the source code using PyLint."""
-
-    # Note that get_build_dir() is suitable for testing use only.
-    pylint_prog = paths.get_build_dir('env', 'bin', 'pylint')
-    pylint_args = [
-      pylint_prog,
-      "--rcfile=" + settings.PYLINTRC,
-      "--disable=all",
-      "--enable=%s" % ','.join([rule.split('(', 1)[0] for rule in RULES]),
-      "--load-plugins=pylint_django"
-    ]
-
-    if options['force']:
-      pylint_args.append('-f')
-
-    if options['outputformat']:
-      pylint_args.append(options['outputformat'])
-
-    if options['files'] is not None:
-      pylint_args.extend(options['files'].split())
-    elif options['app'] == 'all':
-      pylint_args.extend(self.valid_app())
-    else:
-      pylint_args.append(options['app'])
-
-    if not os.path.exists(pylint_prog):
-      msg = _("Cannot find pylint at '%(path)s'. Please install pylint first.") % {'path': pylint_prog}
-      logging.error(msg)
-      raise CommandError(msg)
-
-    logging.info("Running pylint with args: %s" % (" ".join(pylint_args),))
-
-    # We exec pylint directly due to a "maximum recursion depth" bug when doing
-    # pylint.lint(...) programmatically.
-    ret = subprocess.call(pylint_args)
-    if ret != 0:
-      sys.exit(1)

+ 16 - 18
desktop/core/src/desktop/settings.py

@@ -31,6 +31,8 @@ import sys
 import uuid
 
 import desktop.redaction
+from desktop.lib import conf
+from desktop import appmanager
 
 from desktop.lib.paths import get_desktop_root, get_run_root
 from desktop.lib.python_util import force_dict_to_strings
@@ -52,7 +54,7 @@ NICE_NAME = "Hue"
 
 ENV_HUE_PROCESS_NAME = "HUE_PROCESS_NAME"
 ENV_DESKTOP_DEBUG = "DESKTOP_DEBUG"
-LOGGING_CONFIG = None # We're handling our own logging config. Consider upgrading our logging infra to LOGGING_CONFIG
+LOGGING_CONFIG = None  # We're handling our own logging config. Consider upgrading our logging infra to LOGGING_CONFIG
 
 
 ############################################################
@@ -72,9 +74,6 @@ desktop.log.basic_logging(os.environ[ENV_HUE_PROCESS_NAME])
 
 logging.info("Welcome to Hue " + HUE_DESKTOP_VERSION)
 
-# Then we can safely import some more stuff
-from desktop import appmanager
-from desktop.lib import conf
 
 # Add fancy logging
 desktop.log.fancy_logging()
@@ -166,7 +165,7 @@ MIDDLEWARE = [
     'django.middleware.csrf.CsrfViewMiddleware',
     'desktop.middleware.CacheControlMiddleware',
     'django.middleware.http.ConditionalGetMiddleware',
-    #'axes.middleware.FailedLoginMiddleware',
+    # 'axes.middleware.FailedLoginMiddleware',
     'desktop.middleware.MimeTypeJSFileFixStreamingMiddleware',
     'crequest.middleware.CrequestMiddleware',
 ]
@@ -194,7 +193,7 @@ INSTALLED_APPS = [
     'django_extensions',
 
     # 'debug_toolbar',
-    #'south', # database migration tool
+    # 'south', # database migration tool
 
     # i18n support
     'django_babel',
@@ -207,7 +206,7 @@ INSTALLED_APPS = [
     'webpack_loader',
     'django_prometheus',
     'crequest',
-    #'django_celery_results',
+    # 'django_celery_results',
     'rest_framework',
     'rest_framework.authtoken',
 ]
@@ -270,9 +269,7 @@ TEMPLATES = [
 AUTH_PROFILE_MODULE = None
 
 LOGIN_REDIRECT_URL = "/"
-LOGOUT_REDIRECT_URL = "/" # For djangosaml2 bug.
-
-PYLINTRC = get_run_root('.pylintrc')
+LOGOUT_REDIRECT_URL = "/"  # For djangosaml2 bug.
 
 # Custom CSRF Failure View
 CSRF_FAILURE_VIEW = 'desktop.views.csrf_failure'
@@ -315,7 +312,7 @@ conf.initialize(_app_conf_modules, _config_dir)
 # Now that we've loaded the desktop conf, set the django DEBUG mode based on the conf.
 DEBUG = desktop.conf.DJANGO_DEBUG_MODE.get()
 GTEMPLATE_DEBUG = DEBUG
-if DEBUG: # For simplification, force all DEBUG when django_debug_mode is True and re-apply the loggers
+if DEBUG:  # For simplification, force all DEBUG when django_debug_mode is True and re-apply the loggers
   os.environ[ENV_DESKTOP_DEBUG] = 'True'
   desktop.log.basic_logging(os.environ[ENV_HUE_PROCESS_NAME])
   desktop.log.fancy_logging()
@@ -390,7 +387,7 @@ if os.getenv('DESKTOP_DB_CONFIG'):
       zip(["ENGINE", "NAME", "TEST_NAME", "USER", "PASSWORD", "HOST", "PORT"], conn_string.split(':'))
     )
   )
-  default_db['NAME'] = default_db['NAME'].replace('#', ':') # For is_db_alive command
+  default_db['NAME'] = default_db['NAME'].replace('#', ':')  # For is_db_alive command
 else:
   test_name = os.environ.get('DESKTOP_DB_TEST_NAME', get_desktop_root('desktop-test.db'))
   logging.debug("DESKTOP_DB_TEST_NAME SET: %s" % test_name)
@@ -439,7 +436,7 @@ if desktop.conf.QUERY_DATABASE.HOST.get():
 
 CACHES = {
     'default': {
-        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', # TODO: Parameterize here for all the caches
+        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',  # TODO: Parameterize here for all the caches
         'LOCATION': 'unique-hue'
     },
     'axes_cache': {
@@ -480,7 +477,7 @@ if desktop.conf.SESSION.TRUSTED_ORIGINS.get():
   TRUSTED_ORIGINS += desktop.conf.SESSION.TRUSTED_ORIGINS.get()
 
 # This is required for knox
-if desktop.conf.KNOX.KNOX_PROXYHOSTS.get(): # The hosts provided here don't have port. Add default knox port
+if desktop.conf.KNOX.KNOX_PROXYHOSTS.get():  # The hosts provided here don't have port. Add default knox port
   if desktop.conf.KNOX.KNOX_PORTS.get():
     hostport = []
     ports = [  # In case the ports are in hostname
@@ -488,7 +485,7 @@ if desktop.conf.KNOX.KNOX_PROXYHOSTS.get(): # The hosts provided here don't have
     ]
     for port in ports + desktop.conf.KNOX.KNOX_PORTS.get():
       if port == '80':
-        port = '' # Default port needs to be empty
+        port = ''  # Default port needs to be empty
       else:
         port = ':' + port
       hostport += [host.split(':')[0] + port for host in desktop.conf.KNOX.KNOX_PROXYHOSTS.get()]
@@ -586,6 +583,7 @@ for middleware in desktop.conf.MIDDLEWARE.get():
 def is_oidc_configured():
   return 'desktop.auth.backend.OIDCBackend' in AUTHENTICATION_BACKENDS
 
+
 if is_oidc_configured():
   INSTALLED_APPS.append('mozilla_django_oidc')
   if 'desktop.auth.backend.AllowFirstUserDjangoBackend' not in AUTHENTICATION_BACKENDS:
@@ -746,11 +744,11 @@ if desktop.conf.TASK_SERVER.ENABLED.get() or desktop.conf.TASK_SERVER.BEAT_ENABL
 
   CELERY_TASK_QUEUES = {
     'low_priority': {
-        'exchange': 'low_priority', # unused
+        'exchange': 'low_priority',  # unused
         'routing_key': 'low_priority',
     },
     'high_priority': {
-        'exchange': 'high_priority', # unused
+        'exchange': 'high_priority',  # unused
         'routing_key': 'high_priority',
     },
     'default': {
@@ -773,7 +771,7 @@ if desktop.conf.TASK_SERVER.ENABLED.get() or desktop.conf.TASK_SERVER.BEAT_ENABL
     USE_TZ = True
 
 
-PROMETHEUS_EXPORT_MIGRATIONS = False # Needs to be there even when enable_prometheus is not enabled
+PROMETHEUS_EXPORT_MIGRATIONS = False  # Needs to be there even when enable_prometheus is not enabled
 if desktop.conf.ENABLE_PROMETHEUS.get():
   MIDDLEWARE.insert(0, 'django_prometheus.middleware.PrometheusBeforeMiddleware')
   MIDDLEWARE.append('django_prometheus.middleware.PrometheusAfterMiddleware')

+ 32 - 1
pyproject.toml

@@ -6,4 +6,35 @@ python_files = "tests.py test_*.py *_tests.py tests_* *_test.py"
 markers = [
     "integration: live server based tests",
     "requires_hadoop: live hadoop cluster based tests"
-    ]
+]
+
+[tool.ruff]
+target-version = "py38"
+line-length = 140
+force-exclude = true
+extend-exclude = [
+  "*/ext-py3/*",
+  "desktop/core/src/desktop/lib/wsgiserver.py",
+  "*/migrations/*",
+  "apps/oozie/src/oozie/tests.py",
+  "tools/ops/",
+  "tools/ace-editor/",
+  "*/gen-py/*",
+  "*/org_migrations/*"
+]
+
+[tool.ruff.lint]
+preview = true
+select = [
+    "E", # pycodestyle errors
+    "W", # pycodestyle warnings
+]
+ignore = [
+    "E111",
+    "E114",
+    "W191",
+    "E902", # Skip it because of deleted staged files
+]
+
+[tool.ruff.format]
+docstring-code-format = true

+ 5 - 13
tools/ci/check_for_python_lint.sh

@@ -20,32 +20,24 @@
 HOME=${1:-"."}
 FOUND_ISSUE=-1
 
-files=`git diff --name-only origin/master --diff-filter=b | egrep .py$ | \
-  grep -v /ext-py/ | \
-  grep -v /ext-py3/ | \
-  grep -v wsgiserver.py | \
-  grep -v /migrations/ | \
-  grep -v apps/oozie/src/oozie/tests.py | \
-  grep -v tools/ops/ | \
-  grep -v apps/beeswax/gen-py/ | \
-  grep -v /org_migrations/`
+files=`git diff --name-only origin/master --diff-filter=b | egrep .py$` 
 
 cd $HOME
 
 if [ ! -z "$files" ];
 then
-  ./build/env/bin/hue runpylint --files "$files"
+  ./build/env/bin/ruff check $files
   FOUND_ISSUE=$?
 else
-  echo "No Python code files changed present"
+  echo "No Python code files changes present."
   FOUND_ISSUE=0
 fi
 
 if [ "$FOUND_ISSUE" -eq "0" ]
 then
-  echo "No Python code styling issues found"
+  echo "No Python code styling issues found."
 else
-  echo "Found some Python code styling issues"
+  echo "Found some Python code styling issues."
 fi
 
 exit $FOUND_ISSUE