Discussion:
Taking ruby packaging to the next level
Marcus Rueckert
2014-09-09 15:18:18 UTC
Permalink
Taking ruby packaging to the next level
=========================================

Table of Content
------------------

1. TL;DR
2. Where we started
3. The basics
4. One step at a time
5. Rocks on the road
6. "Job done right?" "Well almost."
7. Whats left?

TL;DR
-------

* we are going back to versioned ruby packages with a new naming scheme.
* one spec file to rule them all. (one spec file to build a rubygem for
all interpreters)
* macro based buildrequires. %{rubygem rails:4.1 >= 4.1.4}
* gem2rpm.yml as configuration file for gem2rpm. no more manual editing
of spec files.
* with the config spec files can always be regenerated without losing
things.


Where we started
------------------

A long time ago we started with the support for building for multiple
ruby versions, actually MRI versions, at the same time. The ruby base
package was in good shape in that regard. But we had one open issue -
building rubygems for multiple ruby versions. This issue was hanging for
awhile so we went and reverted to a single ruby version packaged for
openSUSE again.

While this might work for openSUSE, it can become really challenging for
SLES with its long life cycle. Even in the openSUSE world we have
Evergreen for 11.4 still alive. That was released in March 2011 and
still no sign that Evergreen support for it is ending. We really want
multiple ruby versions here.

With that in mind we were wondering how much work would be left to do,
so we can actually fully support multiple ruby versions or even multiple
ruby interpreters.


The steps we need to take
---------------------------

When asking around for "If we support multiple versions again what would
you expect in regard to rubygems packaging?", we got a few important
points.

* only 1 spec file for all. python currently uses 1 spec file for each
version
* avoid re-packaging in d:l:r:e [1]
* system ruby should always be /usr/bin/ruby.


The basics
------------

The first step was to restore the versioned ruby packaging and add at
least one extra version. As a start we used a snapshot package for MRI
2.2.

All the common bits needed for all ruby interpreters moved into
ruby-common. The unversioned ruby package became a wrapper to pull in
the system ruby. As mentioned above /usr/bin/ruby should always point
to the system ruby. So in the new schema it is **not** a symlink
handled by update-alternatives, but a hardlink.

The same goes for %{_libdir}/libruby.so. It is actually a pity that we
still need this file, as any good program linking ruby should just query
RbConfig for the commands to link the libruby for the currently running
interpreter. Sadly some tools ask for the CFLAGS but then hardcode
-lruby. Ugh.

Our naming scheme is

```
mainpackage = "#{interpreter}#{major}.#{minor}"

jruby1.7
rubinius2.2
ruby2.1
ruby2.2
```

The interperter name is more or less the same except for rubinius. Their
binary is called rbx. So we get:

```
interpretername = "#{binaryname}#{major}.#{minor}"

jruby1.7
rbx2.2
ruby2.1
ruby2.2
```

As we wanted to support gems for more than one version, we also need to
express that in their package names now.

```
gempackage = "#{interpretername}-rubygem-#{gemname}#{gemsuffix}"

# gemsuffix is optional. we only need it for packages where we need more
# than one version

jruby1.7-rubygem-tzinfo
rbx2.2-rubygem-tzinfo
ruby2.1-rubygem-tzinfo
ruby2.2-rubygem-tzinfo
```

The gem binary naming is a bit longer now as well. But most of you
should not notice anything as update-alternatives covers this:

gembinary = "#{binaryname}.#{interpretername}-#{gemversion}"

```
$ ls -l /usr/bin/bundler*
lrwxrwxrwx 1 root root 25 25. Jul 19:04 /usr/bin/bundler -> /etc/alternatives/bundler
lrwxrwxrwx 1 root root 31 25. Jul 19:04 /usr/bin/bundler-1.6.5 -> /etc/alternatives/bundler-1.6.5
lrwxrwxrwx 1 root root 38 5. Sep 13:00 /usr/bin/bundler.rbx2.2 -> ../lib64/rubinius/gems/2.2/bin/bundler
lrwxrwxrwx 1 root root 33 25. Jul 19:04 /usr/bin/bundler.ruby2.1 -> /etc/alternatives/bundler.ruby2.1
-rwxr-xr-x 1 root root 504 25. Jul 18:53 /usr/bin/bundler.ruby2.1-1.6.5
lrwxrwxrwx 1 root root 33 8. Sep 19:19 /usr/bin/bundler.ruby2.2 -> /etc/alternatives/bundler.ruby2.2
-rwxr-xr-x 1 root root 504 5. Sep 15:57 /usr/bin/bundler.ruby2.2-1.6.5
```

Last but not least we have the ruby(abi). In the past it used to be just
the "version" number you had in your ruby paths. Now the define also
contains the ruby interpreter. If we take our example list from above:

```
jruby:1.7
rubinius:2.2
ruby:2.1.0
ruby:2.2.0
```

While this works nicely for our packaging needs. It is actually tricky
for cases where a gem wants at least a certain ruby version. The gemspec
only has spec.required_ruby_version which is a version number. While in
MRI this number is compared with the ruby version number, in the case of
rubinius/jruby it is compared with the version of ruby language standard
that is implemented.

One possible solution would be to add also mri(abi), jruby(abi),
rubinius(abi), which just have a numerical abi comparison.

```
Provides: jruby(abi) = 20100
Provides: rubinius(abi) = 20100
Provides: mri(abi) = 20100
Provides: mri(abi) = 20200
```

One step at a time
--------------------

Many of our gems nowadays build without having a buildrequires to their
runtime dependencies. We only generate the dependencies into the package
meta. Though some packages still could require e.g. rspec so they could
run their testsuite at build time. At this point generating multiple
spec files seemed like an easier solution: we would generate build deps
for each ruby interpreter/version into the spec file. But we were asked
not to, so we stick with a single spec file.

Fortunately rpm gives us macros, which on the other hand also need
support in the buildservice. Good thing is the guy to make that happen
for rpm and the OBS is the same person. :D

```
BuildRequires: %{ruby}
BuildRequires: %{rubydevel}
BuildRequires: %{rubygem somegem >= someversion}
```

Without going into too many details here [2], in home:darix:ruby this
expands to:

```
BuildRequires: ruby2.1 ruby2.2 rubinius2.2
BuildRequires: ruby2.1-devel ruby2.2-devel rubinius2.2-devel
BuildRequires: rubygem(ruby:2.1.0:somegem) >= someversion rubygem(ruby:2.2.0:somegem) >= someversion rubygem(rubinius:2.2:somegem) >= someversion
```

Most of our machinery for installing and cleaning up things was hidden
in macros/shell scripts already so adding a loop in those places was
trivial.

Suddenly we have multiple ruby interpreter/versions in the same build
root and our gems build with each of them.

So far nothing that would break all that horribly when we push it onto
d:l:r:e.


Rocks on the road
-------------------

The problems started when we came to the %files sections in the spec
files. Until now we had been generating them into the spec file. That
means for every new ruby interpreter/major branch we would need to
regenerate all spec files.

That is not really a viable solution going forward. So we replaced the
static files section with a macro too.

```
%gem_packages
```

But this flexibility comes at a price. If building the gem fails for one
ruby interpreter/version, it will break the build for all. But you can
control which interpreters are even pulled into the build environment.

```
%define rb_build_versions %{rb_default_ruby}
BuildRequires: %{rubydevel}
BuildRequires: %{rubygem cheetah}
```

In home:darix:ruby this would expand to:

```
BuildRequires: ruby2.1-devel
BuildRequires: rubygem(ruby:2.1.0:cheetah)
```

The valid values for %rb_build_versions can be found on the in the
macros files of each interpreter package. We would have loved to use the
package names as the macro values but those values are passed into
macros again and macro names can not contain dots. For easier reading
you can look at the prjconf of home:darix:ruby. [2]

"Job done right?" "Well almost."
----------------------------------

There were all those little bits and pieces that creeped in now. We had
gems with %pre/%post scriptlets, because the gem was actually a somewhat
better tarball for a service. Rubygems also lacked a way to express
native buildrequires.

As it became clear that we will need to regenerate all the spec files in
d:l:r:e, we aimed for a solution that allowed us regenerating the spec
files at any time.

**No more manually editing spec files**

So we looked through a huge selection of spec files and checked what
things, we actually modified. Once that list was compiled, we created a
config file.

The config file is named gem2rpm.yml. Each field in the config file [3]
has a matching hook in the spec file template or files section template
[4] (via %gem_packages).

gem2rpm.yml was patched to support the config file. Right now you have
to manually pass the config file, but there is a small shell wrapper [5]
that checks if the config file exists and adds the option automatically.

With that in place you can regenerate all spec files without worrying to
lose manual edits. There wont be any. And yes ... in the development
process this has been done multiple times after fixes to the spec file
template.

The final result?

```
#
# spec file for package rubygem-tzinfo-0
<snip>
# This file was generated with a gem2rpm.yml and not just plain gem2rpm.
# All sections marked as MANUAL, license headers, summaries and descriptions
# can be maintained in that file. Please consult this file before editing any
# of those fields
#

Name: rubygem-tzinfo-0
Version: 0.3.37
Release: 0
%define mod_name tzinfo
%define mod_full_name %{mod_name}-%{version}
%define mod_version_suffix -0
BuildRoot: %{_tmppath}/%{name}-%{version}-build
BuildRequires: ruby-macros >= 5
BuildRequires: %{ruby}
BuildRequires: %{rubygem gem2rpm}
BuildRequires: %{rubygem rdoc > 3.10}
Url: http://tzinfo.rubyforge.org/
Source: http://rubygems.org/gems/%{mod_full_name}.gem
Source1: gem2rpm.yml
Summary: Daylight-savings aware timezone library
License: MIT
Group: Development/Languages/Ruby

%description
TZInfo is a Ruby library that uses the standard tz (Olson) database to provide
daylight savings aware transformations between times in different time zones.

%prep

%build

%install
%gem_install \
--doc-files="CHANGES LICENSE README" \
-f

%gem_packages

%changelog
```

As you see this spec uses a gem2rpm.yml, which looks like this:

```
---
:version_suffix: '-0'
```

With that spec file the build generates for me:

```
rbx2.2-rubygem-tzinfo-0-0.3.37-9.4.x86_64.rpm
rbx2.2-rubygem-tzinfo-doc-0-0.3.37-9.4.x86_64.rpm
rbx2.2-rubygem-tzinfo-testsuite-0-0.3.37-9.4.x86_64.rpm
ruby2.1-rubygem-tzinfo-0-0.3.37-9.4.x86_64.rpm
ruby2.1-rubygem-tzinfo-doc-0-0.3.37-9.4.x86_64.rpm
ruby2.1-rubygem-tzinfo-testsuite-0-0.3.37-9.4.x86_64.rpm
ruby2.2-rubygem-tzinfo-0-0.3.37-9.4.x86_64.rpm
ruby2.2-rubygem-tzinfo-doc-0-0.3.37-9.4.x86_64.rpm
ruby2.2-rubygem-tzinfo-testsuite-0-0.3.37-9.4.x86_64.rpm
```

This has already been used extensively to build
[Discourse](http://discourse.org) and
[GitLab](https://about.gitlab.com/) for SLE 12.

What is left?
---------------

If you look just at the things that are packaged using the new way, we
are done. But it would be nice to also support the versioning pattern
that we have used up to 13.1. Maybe even support building against
unversioned ruby versions. (Yes, in theory that is possible.)

Once we have taken those steps, we have to recreate all spec files in
d:l:r:e. Yes, you read correctly ... recreate all spec files. While we
wrote all macros in a way to work as a drop in replacement, as soon as
we want to build for more than one ruby version, we need the
%gem_packages usage. I really hope enough people step up to the effort
so that the number of packages each person has to touch is kept in the
low double digits. Most of the work will be extracting the manual bits
into gem2rpm.yml files and then regenerate the spec file with the new
template. A more detailed mail will be sent later.

Last but not least we can improve the templates (move duplicated code
into gem2rpm e.g.)

Building non gem based ruby libraries is not covered by this new
packaging at all. For once many of the non rubygem based libraries are
bindings build within a larger build process. We would need to manually
do the loop for all ruby interpreter in those. I think in many cases we
only need those libraries for our system ruby. (like yast e.g.) For the
other cases we should work with upstream to switch the ruby bindings to
rubygems.

Also not covered yet, but it would be really useful: All the ruby
scripts shipped as part of the distro should use a shebang line pointing
to the versioned binary. You might ask "why? /usr/bin/ruby is a
hardlink!" While this is true ... people can still replace it. Our
scripts/programs should **not** break in that situation. E.g. yast is
very critical in that regard. Again a gem based distribution makes it
easier as rubygems does it for us in that case. But it shouldn't be much
work to add a small script to fix shebang lines and an rpmlint check
that finds all shebang lines that are unversioned.

Footnotes
-----------

[1]: https://build.opensuse.org/project/show/devel:languages:ruby:extensions
[2]: The interested reader can check
https://build.opensuse.org/project/prjconf/home:darix:ruby It involves
recursive macro calls and other fun things. You have been warned.
[3]: http://files.nordisch.org/ruby-packaging-next/gem2rpm.yml
[4]: http://files.nordisch.org/ruby-packaging-next/sles12.spec.erb
http://files.nordisch.org/ruby-packaging-next/sles12.gem_packages.spec.erb
[5]: the little wrapper called "g2r"
```
#!/bin/sh
if [ -e gem2rpm.yml ] ; then
cfg="--config gem2rpm.yml"
fi
exec gem2rpm $cfg -t /usr/share/doc/packages/rubygem-gem2rpm/sles12.spec.erb -o *spec *gem
```
--
openSUSE - SUSE Linux is my linux
openSUSE is good for you
www.opensuse.org
Jordi Massaguer Pla
2014-09-09 15:44:17 UTC
Permalink
Post by Marcus Rueckert
What is left?
---------------
If you look just at the things that are packaged using the new way, we
are done. But it would be nice to also support the versioning pattern
that we have used up to 13.1. Maybe even support building against
unversioned ruby versions. (Yes, in theory that is possible.)
Once we have taken those steps, we have to recreate all spec files in
d:l:r:e. Yes, you read correctly ... recreate all spec files. While we
wrote all macros in a way to work as a drop in replacement, as soon as
we want to build for more than one ruby version, we need the
%gem_packages usage. I really hope enough people step up to the effort
so that the number of packages each person has to touch is kept in the
low double digits. Most of the work will be extracting the manual bits
into gem2rpm.yml files and then regenerate the spec file with the new
template. A more detailed mail will be sent later.
Once we release SLE12 I'll be happy to help :-)
Marcus Rueckert
2014-09-30 12:00:40 UTC
Permalink
Post by Marcus Rueckert
Last but not least we have the ruby(abi). In the past it used to be just
the "version" number you had in your ruby paths. Now the define also
```
jruby:1.7
rubinius:2.2
ruby:2.1.0
ruby:2.2.0
```
While this works nicely for our packaging needs. It is actually tricky
for cases where a gem wants at least a certain ruby version. The gemspec
only has spec.required_ruby_version which is a version number. While in
MRI this number is compared with the ruby version number, in the case of
rubinius/jruby it is compared with the version of ruby language standard
that is implemented.
One possible solution would be to add also mri(abi), jruby(abi),
rubinius(abi), which just have a numerical abi comparison.
```
Provides: jruby(abi) = 20100
Provides: rubinius(abi) = 20100
Provides: mri(abi) = 20100
Provides: mri(abi) = 20200
```
As it turns out we have to go with the 2nd solution. in the first
solution the interpreter part after the "=" was interpreted as epoch by
rpm.

so we will have

```
Provides: jruby(abi) = 2.1.0
Provides: rubinius(abi) = 2.1.0
Provides: ruby(abi) = 2.1.0
Provides: ruby(abi) = 2.2.0
```

the changes are live in factory and sle12 already.

darix
--
openSUSE - SUSE Linux is my linux
openSUSE is good for you
www.opensuse.org
Loading...