title: Easily checking for deadlinks on docs.gethue.com author: Hue Team type: post date: 2019-10-17T15:34:31+00:00 url: /easily-checking-for-deadlinks-on-docs-gethue-com/ sf_thumbnail_type:
docs.gethue.com are getting some refreshed content continuously. In addition, a series of links not working (returning a 404) have been fixed. Here is how it was done.
First we used the muffet tool. muffet is a fast link checker crawler, very easy to use:
sudo snap install muffet
Then after booting the hugo documentation server, we point to its url. We also blacklist certain urls to avoid some noisy false positives:
muffet http://localhost:35741/ --exclude ".*releases.*" -f
And here is the output:
$ muffet http://localhost:35741/ --exclude ".*releases.*" -f http://localhost:35741/user/ 404 http://localhost:35741/administrator/configuration/editor/#connectors http://localhost:35741/developer/parsers/ 404 http://localhost:35741/administrator/configuration/editor/#postgresql http://localhost:35741/administrator/administration/reference/ dial tcp4 127.0.0.1:5555: connect: connection refused http://localhost:5555/tasks http://localhost:35741/user/querying/ 404 http://wiki.apache.org/hadoop/Hive/AdminManual/Configuration http://localhost:35741/administrator/installation/cloud/ dial tcp4 127.0.0.1:16686: connect: connection refused http://localhost:16686 lookup prometheus on 127.0.0.53:53: server misbehaving http://prometheus:9090/graph lookup prometheus on 127.0.0.53:53: server misbehaving http://prometheus:9090/targets http://localhost:35741/administrator/configuration/apps/ 404 http://localhost:35741/administrator/configuration/dashboard 404 http://localhost:35741/administrator/configuration/editor/ 404 http://localhost:35741/developer/editor/ 404 http://localhost:35741/user/editor/ dialing to the given TCP address timed out https://dev.mysql.com/downloads/connector/j/ http://localhost:35741/user/browsing/ 404 http://localhost:35741/administrator/configuration/external/ http://localhost:35741/developer/sdk/ 404 http://localhost:8000 404 https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/static/desktop/js/autocomplete/jison 404 https://github.com/cloudera/hue/tree/master/desktop/libs/metadata/catalog http://localhost:35741/administrator/configuration/connectors/ 404 http://localhost:35741/administrator/configuration/external/ 404 http://localhost:35741/user/browsers#adls 404 http://localhost:35741/user/browsers#s3 404 http://localhost:35741/user/browsers/ http://localhost:35741/developer/development/ 404 http://docs.python.org/library/hotshot.html 404 https://en.wikipedia.org/wiki/Hue_(Software 404 https://twitter.com/gethue! lookup developer on 127.0.0.53:53: server misbehaving http://developer/development/#javascript
Et voila! Then a few searches and replaces in the documentation content and we have a cleaner experience! Next action is to add the link checking to the Continuous Integration to fully automate the process and scale the developer productivity.
Romain from the Hue Team