---
title: Improving the developer productivity with some Continuous Integration
author: admin
type: post
date: 2019-06-12T17:16:31+00:00
url: /improving-the-developer-productivity-with-some-continuous-integration/
sf_thumbnail_type:
- none
sf_thumbnail_link_type:
- link_to_post
sf_detail_type:
- none
sf_page_title:
- 1
sf_page_title_style:
- standard
sf_no_breadcrumbs:
- 1
sf_page_title_bg:
- none
sf_page_title_text_style:
- light
sf_background_image_size:
- cover
sf_social_sharing:
- 1
sf_related_articles:
- 1
sf_sidebar_config:
- left-sidebar
sf_left_sidebar:
- Sidebar-2
sf_right_sidebar:
- Sidebar-1
sf_caption_position:
- caption-right
ampforwp-amp-on-off:
- default
categories:
- Administration
- Version 4
# - Version 4.5
---
Tooling and automation has been simplified dramatically with technologies like [Docker][1] and the Cloud. On top of this, continuous integration companies like [Circle CI][2] and [Travis CI][3] have added one more layer of simplicity.
## Background
Hue‘s scope has been pretty wide (ranging from SQL parsers to CSS to scheduling jobs or querying any type of databases…) and the increase of external contributions has made the previous Development process outdated. It was becoming frequent to brake the master branch with errors or not making easy for developer to contribute by asking to manual run the test suite. It was time to modernize the development experience!
## Goals
* Automate all the test runs and remove the manual steps burden of the contributors
* Forbid to push code that breaks tests
* Keep the overall runs below 10min and closer to 5min (the less friction, the better)
* Keep it simple
## Picking up a CI infrastructure
Continuous Integration (CI) is a mean to automate checks on developer changes. The team looked at three separate systems:
* Build our own in Docker images and Jenkins
* Travis CI
* Circle CI
The goal was to minimized the time spent on the setups while providing a maximum of automation to the developers. Influential open source projects like Kubernetes were also looked at for inspiration. We played with Travis CI and Circle CI and found Circle CI free open source plan clear and interface pretty easy to use (literally 5 minutes to get a first pipeline started). The documentation was good even if mostly only needed for more advanced automation points (e.g. automatically pushing commits to the master branch). The paying plans seemed also fair, even if we would not need it as Hue is open source.
## Integrating the CI to Github
On the Github side, these steps below were added to the project configuration. Circle CI was also authorized to access the repository.
* Protect master branch
* Require Circle CI check
* Require pushed branch to be up to date with master
On the Circle side:
For historical reason, traditional `pip install` of requirements is not possible yet in Hue. For the record, this will be cleaner after the migration to Python 3 support [HUE-8737][4]. In the meantime, the Hue docker base image is pulled, the repository is checked out and the Python and JavaScript code changes are applied to the image. Here is the [Circle CI configuration][5].
On the Hue side, here are the current checks:
[
][6]
### Python tests
>
][7]
## Automatic pushes
First the process required developers to push to a branch, then push again to the master branch. This was cumbersome as it was easy to forget to push to master, was a redundant steps and easier to get into conflicts as someone might push some other commits to master in the meantime.