Omnibus

Serve your dish on all the tables

Balasankar C

https://balasankarc.in/talks/fossasia

Me

• Balasankar "Balu" C
• Jr. Distribution Engineer @ GitLab
• Debian Developer
• GNOME Foundation Member & Mozillian

Agenda

• Why?
• Introduction
• DSLs
• Process
• Output
• Caching
• omnibus-gitlab walkthrough

Why?

• Automate everything!
• Safeguard from dependency version mismatch
  • Eg: libssl1.0 v/s libssl1.1 ;)
• OS Package Managers
  • Familiar install/upgrade/removal process
• Reliability in what you provide to users
  • Guaranteed to work

Introduction

• Full stack installers
• (OpsCode) Chef Software, Inc.
• https://github.com/chef/omnibus
• Apache 2.0 License
• Ruby-like DSL

Project DSL

name            "gitlab-ce"
maintainer      "GitLab Inc."
homepage        "https://gitlab.com"
replaces        "gitlab"
install_path    "/opt/gitlab"
build_version   Omnibus::BuildVersion.new.semver
build_iteration 1

dependency "preparation"
dependency "postgresql"
dependency "gitlab-cookbooks"
dependency "gitaly"

exclude "\.git*"

Software DSL

name 'bzip2'
default_version '1.0.6'

license 'BSD-2-Clause'
license_file 'LICENSE'

dependency 'zlib'
dependency 'openssl'

version '1.0.6' do
  source md5: '00b516f4704d4a7cb50a1d97e6e8e15b'
end
source url: "http://www.bzip.org/#{version}/#{name}-#{version}.tar.gz"

relative_path "#{name}-#{version}"

build do
...
end

Software DSL (Build Process)

build do
  env = with_standard_compiler_flags(with_embedded_path)

  # Avoid warning where .rodata cannot be used when making a shared object
  env['CFLAGS'] << ' -fPIC'

  # The list of arguments to pass to make
  args = "PREFIX='#{install_dir}/embedded' VERSION='#{version}'"

  patch source: 'makefile_take_env_vars.patch', env: env
  patch source: 'soname_install_dir.patch', env: env if mac_os_x?

  make args.to_s, env: env
  make "#{args} -f Makefile-libbz2_so", env: env
  make "#{args} install", env: env
end

Process

• Read sources definitions
• Get sources
• Build software
• Perform checks
• Generate output files

Output

• Platform specific packages
  • Deb on Debian/Ubuntu/Mint etc.
  • RPM on RedHat/CentOS/openSUSE
• Package metadata file
  • Complete information of package and included softwares
• Version Manifest file
  • JSON format
  • Proper versions of all software - git commit SHAs

Caching

• Software source cache
  • S3 bucket backed
• Build cache
  • Software definitions, for faster subsequent builds
  • Based on lightweight git tags
  • Cache invalidation - a change in
    • Order of build
    • Software definition
    • Version

Advanced Features

• GPG Signing of packages
• Changelog generation
• Multi-level debugging
• Publishing to different backends like Artifactory, S3, etc.

Summary

• Full stack installers are a great idea.
• Product developers, take a look at Omnibus
• Ruby based, easy-to-understand and rich DSL
• Single, general process to build your product for multiple operating systems
• Platform specific build process with caching

Thanks & Credits

• GitLab, especially the Distribution team
• Chef, Inc. for Omnibus and the documentation at https://github.com/chef/omnibus
• Seth Chisamore and Christopher Maier for Omnibus and the presentation - https://speakerdeck.com/schisamo/eat-the-whole-bowl-building-a-full-stack-installer-with-omnibus
• Petar Milošević for the cover image. Released under CC-BY-SA license at Wikimedia Commons

Thank You

This presentation is licensed under Creative Commons Share Alike license. Logos of software presented are trademark of respective owners.

balasankar@gitlab.com

@cbalasankar