March 19, 2009

Easy packaging of ruby gems for openSUSE, part II

I was in the need of learning about osc plugins, because I want to rewrite some scripts that I use to track maintenance updates to the software management stack. Learning osc plugins meant learning python at the same time.

I was inspired by the osc gnome todo functionality, which shows something like this:

vuntz@lyon ~/work/opensuse/tmp/>osc gnome todo
Downloading data in a cache. It might take a few seconds...
Package  | openSUSE:Factory | GNOME:Factory    | Upstream
---------+------------------+------------------+-------------
anjuta   | 2.23.91          | 2.23.91          | 2.24.0.1 (r)
at-spi   | 1.23.92          | 1.23.92          | 1.24.0
gdm      | 2.23.92          | 2.23.92 (s)      | 2.24.0 (s)
gedit    | 2.23.93          | 2.23.93          | 2.24.0

So, in my last post, I wrote how to easily package gems for openSUSE, saving at least 99% of the work. However, how do you know that you need to package a new gem? Well, here is osc ruby todo, in its first 0.0.0.0.1 release:

duncan@tarro:/osc-plugins> osc ruby todo
126 gems in build service projects
Retrieving upstream gem information...
+ libxml-ruby upstream: 1.1.2 bs: 0.9.5
+ radiant upstream: 0.7.1 bs: 0.6.9
+ launchy upstream: 0.3.3 bs: 0.3.2
+ hpricot upstream: 0.7 bs: 0.6.161
+ net-ssh upstream: 2.0.11 bs: 2.0.8

So, it connects to gems.rubyforge.org and tell you which gems are outdated. Do you want to keep openSUSE a nice ruby platform? help me. The following features are waiting:

  • track ruby package versions
  • track other gem repositories, like the rails one or github
  • also compare it with version in factory
  • nice tables
  • caching
  • Use gem2rpm and the openSUSE template to automatically create the specs for outdated gems

You can download it from git.opensuse.org, just symlink the .py file to ~/.osc-plugins. It requires rpm-python and Python 2.6 or newer.

My ultimate goal is to get a similar one for zypp/yast that allows to compare our Head repositories with Factory and maintenance repos, show diffs and make submit requests.

So how does it works? How to create your own osc plugin? Look at the source code of the osc-* projects at git.opensuse.org, You will realize it is as easy as:

  • Open a .py file
  • Create a do_something method, adding the documentation and checking input and arguments
  • Use the meta_ API and urllib to get data from the build service
  • Use makeurl to build the right authenticated urls to do the requests.
March 17, 2009

Easy packaging of ruby gems for openSUSE

Every distribution has its own conventions to package scripting languages addons (perl modules, ruby gems, etc) as native packages. This allows packages to depend on those addons honoring the package dependencies, and at the same time, look like the addon was installed the scripting language tool (cpan, gem).

openSUSE keeps a repository of gems, thanks to Marcus Rueckert.

For ruby, David Lutterkort (augeas lead developer) created a nice tool to help converting gems into spec files.

I created a template for openSUSE-like rubygem-* packages, and David committed it upstream.

So, to create a package for a gem:

Fetch it:

gem fetch foo

Convert it:

gem2rpm -t opensuse.spec.template ./foo-1.1.gem > rubygem-foo.spec

Build and tweak it. Make sure everything is alright. Some gems work out of the box, some not, but still gem2rpm saves 90% of the effort. Consider submitting it to the project if you are willing to keep it up to date ;-)

March 3, 2009

ZYpp 6.2.1, no mirror will stop you

Two weeks ago I posted about making aria2c the default transfer mechanism for our software management stack.

With libzypp 6.1.0 in Factory, this is already the case. With the first community round of testing, we got quite a lot of bug reports and feedback. Some stuff did not work because the implementation was wrong, while others just showed us that aria2 worked differently as we thought.

The upcoming 6.2.1 addresses the following issues:

  • implemented speed indicators (bnc#475506). You should see transfer speed in applications that report it (like zypper).
  • implemented progress indicators correctly (bnc#473846). You should see progress reporting now in a consistent basis.

  • fixed broken pipe when looking for aria2c which caused a fallback to curl. (bnc#480930)

  • implement saving and reading mirror stats data in /var/cache/zypp/aria2.stats. This means libzypp now learns which mirrors are faster and more reliable for you and remembers it. Every usage adds more information to the “learning”.

  • handle failover correctly (bnc#481115). This means libzypp tries harder to get the file from alternate mirrors.

  • various improvements in error and report handling. General code review and minor fixes.

How to test?

Peter added an interesting “test feature”. If a request happens with a header X-Broken-Mirrors then the openSUSE redirector will send you broken mirrors. So edit your root’s aria2.conf:

cat ~/.aria2/aria2.conf
header=X-Broken-Mirrors: true

And observe zypper.log. Even if errors are reported, libzypp should try other mirrors and get the file successfully.

You can go more extreme, and use “X-Broken-Mirrors: only”, which will give you only bad servers. Even in this case you will see libzypp try hard to get the file before giving up. How hard to try is configurable, see last post for more information.

Big thanks to Peter Poeml and Tatsuhiro Tsujikawa for all the feedback and support, and of course

And with this, feature 302923 is done ;-) and the rest is tracked as bugs.