Release management

Working in a large collaboration is always hard. Especially at software front. One of the projects I carried on was a release management. It was required to deploy 800+ software packages among different sites in a fast and efficient manner. Code should compile on multiple architectures, including OSF, Solaris and Linux.

The following operating systems are officially supported in CLEO:

• Tru64/OSF1
• SunOS 2.6 and Solaris 2.8, 9, 10
• Linux, RedHat 9, RedHat Enterprise 3, 4, Scientific Linux

Most of the problems we experienced with was a cross-platform incompatibilities and their resolution mostly evolved fixing or tuning a common utilities used on particular platform. For example, GNU utilities, such as gmake, was used on all platforms.

First approach we choose was building package RPMs. It leads to overhead on both sides: the server, where RPMs were built and the client, where software packages were rebuild from the source RPMs due to our requirements. The system was very heavy and hard to maintain. But it gave us a very usefull set of tools to build package inter-dependencies.

Therefore, a new approach was proposed. It was based on usage of CVS as software revision system, secure connection from client to the server (using scp or kerberos) for software deployment and well established Makefile rules. The new system was easily written as a set of Python modules and deployed to off-sites where system administrator used on demand to install new release.

We created a customizable tool to deploy and install CLEO releases. The release was splitted into three main categories:

• other sources, a set of third party packages used by CLEO software, e.g. readline, python, tcltk, etc. We decided to keep our own version of those packages to prevent versioning and dependency problems.
• Common, a set of common software packages used by On-line and Off-line systems.
• Offline, a set of Off-line software packages.

Each of those categories were versioned to allow cross inclusion among themselves, e.g. the same version of other sources was allowed to be used by different Offline releases. Such division allowed us to maintain 800+ packages in very efficient way and apply various patches to different components of the system. The remote client tool, based on CVS, was able to detect, deploy and rebuild necessary updates through secure connection to the main release server (either by kerberos authentication or ssh).

In this example we demostrate a part of release structure used by release management toolkit:

20050120_FULL
|-- bin
|-- cleo
|   |-- cvssrc
|   |   |-- cleog
|   |   |-- cleoii
|   |   |-- dvlib
|   |   |-- mcseq
|   |   |-- mctr
|   |   |-- qqlib
|   |   |-- seq
|   |   |-- svcal
|   |   `-- trkm
|   `-- lib
`-- cleo3
    |-- Common
    |   |-- CVS
    |   |-- rel
    |   `-- scripts
    |-- Offline
    |   |-- InfoFiles
    |   |-- cache
    |   |-- rel
    |   |-- scripts
    |   `-- util
    |-- cern
    |   `-- rel
    `-- other_sources
        `-- pkg