ci-cd.md 2.6 KB

Goal: Automate the builds and validation of changes

Be fast and simple:

  • Cloud based
  • 100% automated
  • Below 5 minutes for basic unit tests (API and UI)
  • Prepare the ground for other components
  • Eventually lock master branch

What to validate

Python

Similar to the Docker Hue. Currently doing only Python 3.8 and 3.9, but we should test the Python 3.10+ build and run the tests.

  • Python 3.8
  • Python 3.9

The docker images could be forked for QA and comes with a series of flags.

Documentation

By updating the makefile to build the new Website. Would need to install Hugo and build it.

Step 2 would be to setup so that the documentation would get published automatically [done].

Step 3 would be to check and get warnings about dead links.

Release

make prod should work, and the tarball release should be automatically installed and checked if boots normally.

Docker images

Building the images: Docker. Ideally the images should then be sent to demo.gethue.com for deployment and reverted in case of issues.

Tests

Those could be broken into several categories and labelled accordingly:

  • Unit tests unit
  • Integration tests (testing a certain connector would involve a live URL or booting its k8s service) integration
    • Hive hive
    • Impala
    • HDFS
    • MySQL...
  • Frontend tests frontend
  • Migration tests migration

Those tests would run on:

  • master branch
  • testing branch
  • pull requests

More quality metrics

Several areas to track:

  • js
  • linting (and missing tests, warnings added)
  • Coverage
  • MyPy

Proposal

Investigate:

  • Cloud based: https://circleci.com currently (free for open source, easy to use)
  • Docker based CI (e.g. CI in Docker...)
  • Tox
  • Build and Makefile refactoring with standard requirement.txt and pypy package
  • git clone under 10s
  • Unit tests under 5min

Starting with these basic steps:

  • Report the runs via UI and github checks
  • Split the tests into several categories
    • unit Python
    • integrations
    • js

Items

Browser Testing CI

Goal: detect JavaScript changes completely breaking pages (e.g. lib upgrades, component change side effects...).

  • Basic page load of Editor
  • Integration to CI https://circleci.com/docs/2.0/browser-testing/
  • Basic page load of Dashboard, Oozie Wokflow Editor
  • Load of Hive, Impala, Dashboad, Oozie Wokflow examples
  • Add Table Browser, Job Browser, File Browser...
  • Add FF, IE as Browsers