Developing LAVA on Debian

LAVA no longer supports development on Ubuntu.

Packages for LAVA are available for:

  • Debian Stretch (stable) - with backports
  • Debian Buster (testing)
  • Debian Sid (unstable)

Packages will remain available for Debian Jessie (oldstable) with backports until June 2018 as security support for Jessie will end. The last production release with Jessie support was 2018.2. Developers need to use Debian Stretch or Buster for development.

When using the packages to develop LAVA, there is a change to the workflow compared to the old lava-deployment-tool buildouts.

Note

Changes to build dependencies between Debian versions can cause changes to the builds for each suite. Always ensure that you build packages for unstable using unstable and build packages for stable using a chroot or VM or other stable environment. If a package built on unstable does not install on stable, rebuild the same changes in a stable environment and re-install. Backports to stable in Debian are always built in a stable chroot or VM for this reason.

Preparing for LAVA development

LAVA provides a lava-dev package which supplies all the dependencies which are required to build local LAVA packages. This package is intended primarily for developers working on laptops and other systems where a full desktop environment is already installed:

$ sudo apt install lava-dev

If you want to build local packages on a headless box or a system with limited space, you can trim the set of dependencies by pre-installing pinentry-curses instead of the default pinentry-gtk2. QEMU is still required and will bring in some X11 dependencies but these are minimal compared to the full dependencies of pinentry-gtk2 which is brought in via gnupg2:

$ sudo apt install pinentry-curses
$ sudo apt-get --purge remove pinentry-gtk2
$ sudo apt-get --purge autoremove
$ sudo apt install lava-dev

Developer package build

Note

The supported suite for LAVA development is now Stretch. The developer package build now defaults to expecting Stretch and therefore uses Python3 exclusively. Support for building Python2 has been removed, the master branch only builds Python3. See https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html

The lava-dev package includes a helper script which is also present in the source code in lava-server/share/. The script requires a normal Debian package build environment (i.e. dpkg-dev) as well as the build-dependencies of the package itself. The helper checks for package dependencies using dpkg-checkbuilddeps which halts upon failure with a message showing which packages need to be installed.

The helper needs to know the name of the package to build and to be started from the directory containing the code for that package:

$ /usr/share/lava-server/debian-dev-build.sh -p lava-server

The packages will be built in a temporary directory using a version string based on the current git tag and the time of the build. The helper outputs the location of all the built packages at the end of a successful build, ready for use with $ sudo dpkg -i.

Note

the helper does not install the packages for you, neither do the packages restart apache, although the lava-server service will be restarted each time lava-server is installed or updated.

To install any package, including the developer build packages, the corresponding package must already be installed at the current production release version (or better), on the same machine. This ensures that all of the runtime dependencies already exist on the system.

Which branch to use for changes

Any and all changes for inclusion into a future release need to be based on the current git master branch and will need rebasing from time to time as master moves ahead.

All testing of the LAVA source code is based on the relevant master branch which is then merged into the staging branch for testing as a release candidate. The final release involves merging staging into the release branch. Git tags are based on the release branch.

When using existing git tags or the release branch, create a new local branch and commit your changes to ensure that a local version string is used.

There can also be new dependencies added by changes in master and staging before those changes are merged into release or uploaded as a production release. When these changes are merged into master, the packaging will also be updated.

Local version strings

The local version is built (using ./version.py) from these components:

  • package name

  • latest git tag name:

    $ git tag --sort -v:refname|head -n1
    2015.12
    
  • incremental revision list count:

    $ git rev-list --count HEAD
    5451
    
  • latest git hash:

    $ git rev-parse --short HEAD
    f9304da
    

The latest git hash is a reference to the latest commit. If you have not committed local changes (e.g. you are on a local branch based on a tag) then the short hash can be used to lookup the commit in the master branch, e.g.:

https://git.linaro.org/lava/lava-server.git/f9304da

Distribution differences

Always build packages on the suite you expect to use for installation.

Packages available from the LAVA repositories are built on the correct suite (using sbuild) using the lava-buildd scripts.

Example

The helper supports lava-server and lava-dispatcher:

$ sudo apt install lava-dev
$ git clone https://git.linaro.org/git/lava/lava-server.git
$ cd lava-server
$ /usr/share/lava-server/debian-dev-build.sh -p lava-server

$ git clone https://git.linaro.org/git/lava/lava-dispatcher.git
$ cd lava-dispatcher
$ /usr/share/lava-server/debian-dev-build.sh -p lava-dispatcher

lava-dispatcher has architecture-dependent dependencies. By default, the package is built for the native architecture and can only be installed on that architecture. To build for a different architecture, e.g. armhf, use:

$ /usr/share/lava-server/debian-dev-build.sh -p lava-dispatcher -a armhf

This does a binary build, so the source is not included, which allows these builds to be included in a local repository, e.g. using reprepro.

Helpers for other distributions may be added in due course. Patches welcome.

Developer build versions

LAVA uses git tags and the developer build adds a suffix to the tag for each local build - the suffix is formed from the git rev-list --count (to get a sequential, unique, identifier) and the git rev-parse --short hash to identify the latest git commit in the branch upon which this build is based. The git short hash can be looked up on the git.linaro.org site, irrespective of which release tag is the current. For example, build version 2015.07.5333.1521ddb-1 relates directly to https://git.linaro.org/lava/lava-server.git/1521ddb

From August 2015, LAVA uses git tags without a leading zero on the month number, in accordance with PEP440, so the git tag will be 2015.8 instead of 2015.07 used for the previous release tag.

Development using Python3

LAVA has moved to exclusive Python3 support as the final stage of the migration to V2. See <https://lists.linaro.org/pipermail/lava-announce/2017-June/000032.html>`_

Both lava-server and lava-dispatcher only support running the unit tests with Python3. All reviews must pass the unit tests when run with Python3.

Builds for Debian Jessie have ceased, support for Python2 has been dropped and only Python3 is be supported.

Python3 dependencies include:

python3-django (>= 1.8), python3-django-auth-ldap (>= 1.1.8), python3-django-restricted-resource (>= 2015.09), python3-django-tables2 (>=1.2), python3-docutils, python3-jinja2, python3-markdown, python3-psycopg2, python3-simplejson, python3-voluptuous (>= 0.8.8), python3:any (>= 3.3.2-2~), python3-configobj, python3-magic, python3-netifaces (>=0.10.0), python3-nose, python3-pexpect (>= 4.2), python3-pyudev (>= 0.21), python3-requests, python3-serial, python3-setproctitle (>= 1.1.8), python3-tz, python3-yaml, python3-zmq, python3-guestfs (>= 1.32.7)

Quick fixes and testing

The paths to execute LAVA python scripts have changed and developing LAVA based on packages has a different workflow.

Modified files can be copied to the equivalent python path. The current LAVA packages use python2.7, so the path is beneath /usr/lib/python2.7/dist-packages/ with sudo:

$ sudo cp <git-path> /usr/lib/python2.7/dist-packages/<git-path>

Warning

To fix failures in the Python3 unit tests, the same change will also need to be copied to /usr/lib/python3/dist-packages/.

Viewing changes

Different actions are needed for local changes to take effect, depending on the type of file(s) updated:

templates/*/*.html next browser refresh (F5/Ctrl-R)
device-types/*.jinja2 next testjob submission
devices/*.jinja2 next testjob submission
*_app/*.py $ sudo apache2ctl restart

Migrating postgresql versions

LAVA installs the postgresql package which installs the current default version of postgresql. When this default changes in Debian, a second package will be added to your system which will start with no actual data.

Caution

postgresql will disable database access during the migration and this will interfere with the running instance. There is typically no rush to do the migration, so this is usually a task for a scheduled maintenance window. Declare a time when all devices can be taken offline and put a replacement site in place of the apache configuration to prevent database access during the migration.

Determining the active cluster

The output of pg_lsclusters includes the port number of each cluster. To ensure that the correct cluster is upgraded, check the LAVA_DB_PORT setting in /etc/lava-server/instance.conf for the current instance. If multiple clusters are shown, postgresql will upgrade to the latest version, so ensure that any intermediate clusters are also stopped before starting the migration.

Performing the migration

Debian gives a notice similar to this when a new version of postgres is installed:

Default clusters and upgrading
------------------------------
When installing a postgresql-X.Y package from scratch, a default
cluster 'main' will automatically be created. This operation is
equivalent to doing 'pg_createcluster X.Y main --start'.

Due to this default cluster, an immediate attempt to upgrade an
earlier 'main' cluster to a new version will fail and you need to
remove the newer default cluster first. E. g., if you have
postgresql-8.2 installed and want to upgrade to 8.3, you first install
postgresql-8.3:

 apt install postgresql-8.3

Then drop the default 8.3 cluster:

 pg_dropcluster 8.3 main --stop

And then upgrade the 8.2 cluster to 8.3:

 pg_upgradecluster 8.2 main

Note

Upgrading a cluster combines pg_dump and pg_restore (making two copies of the database at one point). Ensure that you have enough available space on the disc, especially with a large database. If pg_upgradecluster is interrupted by the lack of disc space it will not harm the system and full rollback will be applied automatically.

See also https://askubuntu.com/questions/66194/how-do-i-migrate-my-postgres-data-from-8-4-to-9-1

Check your existing clusters:

$ sudo pg_lsclusters

Stop postgresql (stops both versions):

$ sudo service postgresql stop

Drop the main cluster of the NEW postgres as this is empty:

$ sudo pg_dropcluster 9.4 main --stop

Postgresql knows which version is the current default, so just tell postgresql which is the old version to migrate the data into the (empty) new one:

$ sudo pg_upgradecluster 9.3 main
Disabling connections to the old cluster during upgrade...
Restarting old cluster with restricted connections...
Creating new cluster 9.4/main ...
 config /etc/postgresql/9.4/main
 data   /var/lib/postgresql/9.4/main
 locale en_GB.UTF-8
 port   5433
Disabling connections to the new cluster during upgrade...
Roles, databases, schemas, ACLs...
Fixing hardcoded library paths for stored procedures...
Upgrading database postgres...
Analyzing database postgres...
Fixing hardcoded library paths for stored procedures...
Upgrading database lavapdu...
Analyzing database lavapdu...
Fixing hardcoded library paths for stored procedures...
Upgrading database lavaserver...
Analyzing database lavaserver...
Fixing hardcoded library paths for stored procedures...
Upgrading database devel...
Analyzing database devel...
Fixing hardcoded library paths for stored procedures...
Upgrading database template1...
Analyzing database template1...
Re-enabling connections to the old cluster...
Re-enabling connections to the new cluster...
Copying old configuration files...
Copying old start.conf...
Copying old pg_ctl.conf...
Stopping target cluster...
Stopping old cluster...
Disabling automatic startup of old cluster...
Configuring old cluster to use a different port (5433)...
Starting target cluster on the original port...
Success. Please check that the upgraded cluster works. If it does,
you can remove the old cluster with

 pg_dropcluster 9.3 main

Check that the instance is still running. Note that the port of the new postgresql server will have been upgraded to the port used for the old postgresql server automatically. Check that this is the case:

$ grep port /etc/postgresql/9.4/main/postgresql.conf
port = 5432

Drop the old cluster:

$ sudo pg_dropcluster 9.3 main

Now the old database package can be removed:

$ sudo apt remove postgresql-9.3

Javascript handling

Javascript has particular issues in distributions, often the version of a Javascript file is out of step with the version available in the distribution or not packaged at all. lava-server embeds javascript files in the static/js directories and maintains a list of files which are replaced with symlinks during a Debian package build. The list is in share/javascript.yaml and the replacement of matching files is done using share/javascript.py. Other distribution builds are invited to use the same script or provide patches if the paths within the script need modification.

After 2015.12 release, all of the .min.js files in the package are removed from VCS and minified files are created at build time. Templates in the system use only minified versions of the javascript files so after the release package rebuild will be mandatory.

Javascript and security

The primary concern is security fixes. Distributions release with a particular release of LAVA and may need to fix security problems in that release. If the file is replaced by a symlink to an external package in the distribution, then the security problem and fix migrate to that package. LAVA tracks these files in share/javascript.yaml. Files which only exist in LAVA or exist at a different version to the one available in the distribution, need to be patched within LAVA. Javascript files created by LAVA are packaged as editable source code and patches to these files will take effect in LAVA after a simple restart of apache and a clearing of any browser cache. Problems arise when the javascript files in the LAVA source code have been minified, resulting in a .min.js file which is not suitable for editing or patching.

The source code for the minified JS used in LAVA is provided in the LAVA source code, alongside the minified version. However, there is a lack of suitable tools to convert changes to the source file into a comparable minified file. If these files need changes, the correct fix would be to patch the unminified javascript and copy the modified file over the top of the minified version. This loses the advantages of minification but gains the benefit of a known security fix.

Javascript maintenance

Work is ongoing upstream to resolve the remaining minified javascript files:

  1. Identify the upstream location of all javascript not listed in share/javascript.yaml and not written by LAVA, specify this location in a README in the relevant js/ directory along with details, if any, of how a modified file can be minified or whether a modified file should simply replace the minified file.
  2. Replace the use of the remaining minified JS where the change to unminified has a negligible or acceptable performance change. If no upstream can be identified, LAVA will need to take over maintenance of the javascript itself, at which point minified files will be dropped until other LAVA javascript can also be minified.
  3. Monitor availability of packages for all javascript files not written by LAVA and add to the listing in share/javascript.yaml when packages become available.
  4. Maintain - only minify javascript written by LAVA if a suitable minify tool is available to be used during the build of the packages and to add such support to share/javascript.py so that minification happens at the same point as replacement of embedded javascript with symlinks to externally provided files.

Packaging changes

From time to time, there can be packaging changes required to handle changes in the LAVA upstream codebase. If you have write access to the packaging repository, changes to the packaging can be tested by pushing to a public branch and passing the -b option to debian-dev-build-sh:

$ /usr/share/lava-server/debian-dev-build.sh -p lava-server -b docs

Building for other architectures

lava-server is the same for all architectures but lava-dispatcher has a different set of dependencies depending on the build architecture. To build an armhf package of lava-dispatcher using the developer scripts, use:

$ /usr/share/lava-server/debian-dev-build.sh -p lava-dispatcher -a armhf

Debugging Django issues

When trying to investigate LAVA web pages generation we advise you to use django-debug-toolbar. This is a Django application that provide more information on how the page was rendered, including:

  • SQL queries
  • templates involved
  • HTTP headers

For instance, the toolbar is a really helpful resource to debug the Django ORM.

Installing

On a Debian system, just run:

$ apt-get install python-django-debug-toolbar

Configuration

Once the python-django-debug-toolbar package is installed, the toolbar needs to be enabled in the instance. Two settings are required in /etc/lava-server/settings.conf

  • "DEBUG": true,
  • "USE_DEBUG_TOOLBAR": true,

Note

settings.conf is JSON syntax, so ensure that the previous line ends with a comma and that the resulting file validates as JSON. Use JSONLINT

The toolbar can be disabled without disabling django debug but django must be in debug mode for the toolbar to be loaded at all.

Restart the django related services to complete the installation of the toolbar:

sudo service lava-server-gunicorn restart
sudo apache2ctl restart

Installation can be checked using lava-server manage shell:

>>> from django.conf import settings
>>> 'debug_toolbar' in settings.INSTALLED_APPS
True

In order to see the toolbar, you should also check the value of INTERNAL_IPS. Local addresses 127.0.0.1 and ::1 are enabled by default.

To add more addresses, set INTERNAL_IPS to a list of addresses in /etc/lava-server/settings.conf, (JSON syntax) for example:

"INTERNAL_IPS": ["192.168.0.5", "10.0.0.6"],

These value depends on your setup. But if you don’t see the toolbar that’s the first think to look at.

Apache then needs access to django-debug-toolbar CSS and JS files:

sudo su -
cd /usr/share/lava-server/static/
ln -s /usr/lib/python2.7/dist-packages/debug_toolbar/static/debug_toolbar .

In /etc/lava-server/settings.conf remove the reference to htdocs in STATICFILES_DIRS. Django-debug-toolbar does check that all directories listed in STATICFILES_DIRS exists. While this is only a leftover from previous versions of LAVA installer that is not needed anymore.

Once the changes are complete, ensure the settings are loaded by restarting both apache2 and django:

sudo service lava-server-gunicorn restart
sudo apache2ctl restart

Performance overhead

Keep in mind that django-debug-toolbar has some overhead on the webpage generation and should only be used while debugging.

Django-debug-toolbar can be disabled, while not debugging, by changing the value of USE_DEBUG_TOOLBAR in /etc/lava-server/settings.conf to false or by changing the ̀DEBUG level in /etc/lava-server/settings.conf to DEBUG: false.

Ensure the settings are reloaded by restarting both apache2 and django:

sudo service lava-server-gunicorn restart
sudo apache2ctl restart