openSUSE is a gem for ruby development

In this post I would like to show what openSUSE has to offer to ruby enthusiasts.

The latest openSUSE version ships with ruby 1.8.7, rubygems 1.3.1 and rails 2.3.2. The latest two being not so recent, here is where the openSUSE project shines. Say hello to devel:languages:ruby and devel:languages:ruby:extensions build service projects.

The first is a project containing a more recent 1.8.7 ruby (p249 vs p72 in 11.2). However, as a build service project, it is built on top of multiple targets so you can add this repository not only to 11.2 but also to SLE and older openSUSE releases.

In this project you will also find a ruby19 package which is nothing else than ruby 1.9.1 p376 installable in parallel with 1.8.7.

The devel:languages:ruby:extensions contains ruby libraries and gems. For example you can find rails 2.3.5 there. Libraries are usually packaged as ruby-something and gems are packaged as rubygem-something. Gem packages have some nice attributes:

• They are visible by the software management stack (rpm ZYpp, and all the ZYpp integrated tools: PackageKit, YaST, etc)
• They are visible to the gem tool. The rpm is installed where the gem tools expects to find it
• They are the best choice if you want distribute a fully packaged application, or an appliance using SUSE Studio
• You can easily create them from a standard gem using the gem2rpm-opensuse script included in the rubygem-gem2rpm package

To add the gem repository to your 11.2 system, just do (as root):

zypper ar \
http://download.opensuse.org/repositories/devel:/languages:/ruby:/extensions/openSUSE_11.2 rubygems

Living on the edge

While the ruby environment provided by openSUSE is great, you may want to go one step further. What comes to my head:

• If you have both ruby and ruby19, you would need to have a different package for each ruby interpreter
• Sometimes you want to try a really experimental virtual machine, however openSUSE offers no package for it (eeer, and experimental VMs make packaging difficult, especially if the developers are MacOS users whose build script download prebuilt binaries of LLVM over the internet eeeek!)
• If you run or develop applications in the same machine, you may want to isolate the gem environment for each applications
• You may want quickly to test a program with different interpreters.
• You may not have root access! And sadly quacking like an administrator won’t give you superpowers.

Enter rvm, the ruby version manager

rvm is a nice tool that can quickly compile ruby interpreters from source and switch between them, all without root access (you can also set the current interpreter to the “system” one). Gems you install for one interpreter are isolated from the other interpreters, and you can create “gem sets”.

While rvm can be installed really easily:

bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )

for the HEAD version in github or

bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-latest )

You can also install it the “openSUSE way”, if you added the devel:languages:ruby:extensions repository:

zypper install rubygem-rvm

Once you have it installed, you will need to run rvm-install (as user) and edit your shell profile so that the environment is set correctly.

Then you can start using it. To see all the interpreters available in your system:

rvm list

To see all the ones known to rvm:

rvm list --all

Build 1.9 from source:

rvm install ruby-1.9.2-head

Build JRuby:

rvm install jruby

Use jruby:

rvm use jruby

Then you can “double check”:

ruby -v

jruby 1.4.0 (ruby 1.8.7 patchlevel 174) (2009-11-02 69fbfa3) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_20) [amd64-java]

Combine rvm with the powerful bundler

Of all the new ruby tools, the ones I find most useful is the bundler, which was created as a rails independent solution to make ruby applications specify the gems they need in their environment.

You can create a Gemfile in the top level directory of your application:

Something like.

source 'http://gemcutter.org'
gem "rails", "3.0.0.beta3"
gem "runt", :git => "http://github.com/tevio/runt.git"

As you can see, you can specify gems from git repositories, or use specific branches or versions. This is really nice for deployment, as you can get the working environment really easy.

bundle install

Would install the missing gems. It would reuse system gems, etc. If you are using rvm, it would install them in the interpreter specific gems.

The bundler has more advanced features, like the ability to take all the gems and “bundle” them directly “in” the application, but I leave that as an exercise to the reader.

Conclusions

So that finishes this post. I hope you could see how openSUSE looks like for a ruby developer.

I would like to first thanks Marcus Rückert (a.k.a darix) as he was the main brain behind the rubygem packages. Also big thanks to the YaST, Klaus, SUSE Studio and Build Service teams, whose development resulted in many package updates and contributions.

Reading a commit from DHH I found out a syntax I was not aware of.

Instead of doing:

[1,2,3,4,5,6,7,8,9,10].select { |x| x.even? }

You can do:

[1,2,3,4,5,6,7,8,9,10].select(&:even?)

When I asked in #rubyonrails irc channel about this syntax, I got enlightened: The first part is obvious: last argument starting with & is interpreted as a Proc and equivalent to passing a block, so it is not a &: operator but & and a symbol. How does the symbol work?

According to the people in the channel, it is rails magic. When I said that it works for me in irb I was told I must be running ruby 1.9, which I am not. After some discussion we realized this magic is in ruby 1.8.7 too.

The magic is of course, just Symbol#to_proc method

I was very happy that Rails Engines got merged in Rails 2.3. However I never liked that engines did not support overlaying static assets like they do with controllers, models and views. Even worse, the discussion was around “mirroring” the engine’s assets into the application public/ directory (like you can see in this railcast).

I imagined that doing something more transparent would be non-trivial. But I never asked myself why. When mentioning this limitation to Daniel when explaining how we use engines in our code, he asked the magical question: “Why?”.

I have been looking at Rails’s Rack support recently. I love Rack. And if you see how Rails builds its complex pieces over such simple components based on Rack, you realize how good Rack is for the Ruby community as a base component.

So, I tried the very first though: Create a Rake middleware called StaticOverlay that is initialized with various overlay directories. When the middleware is called it will look for the file in the overlays, and if it finds it in one, it will serve it (using Rack::File), if not, it just call the application (or next middleware in the chain).

Once I put the rack middleware together it worked out of the box, so I started a thread in rails core list to find out whether this would be a valid approach to hack a real patch and add support for this in Rails itself.

I was in the need of learning about osc plugins, because I want to rewrite some scripts that I use to track maintenance updates to the software management stack. Learning osc plugins meant learning python at the same time.

I was inspired by the osc gnome todo functionality, which shows something like this:

vuntz@lyon ~/work/opensuse/tmp/>osc gnome todo
Downloading data in a cache. It might take a few seconds...
Package  | openSUSE:Factory | GNOME:Factory    | Upstream
---------+------------------+------------------+-------------
anjuta   | 2.23.91          | 2.23.91          | 2.24.0.1 (r)
at-spi   | 1.23.92          | 1.23.92          | 1.24.0
gdm      | 2.23.92          | 2.23.92 (s)      | 2.24.0 (s)
gedit    | 2.23.93          | 2.23.93          | 2.24.0

So, in my last post, I wrote how to easily package gems for openSUSE, saving at least 99% of the work. However, how do you know that you need to package a new gem? Well, here is osc ruby todo, in its first 0.0.0.0.1 release:

duncan@tarro:/osc-plugins> osc ruby todo
126 gems in build service projects
Retrieving upstream gem information...
+ libxml-ruby upstream: 1.1.2 bs: 0.9.5
+ radiant upstream: 0.7.1 bs: 0.6.9
+ launchy upstream: 0.3.3 bs: 0.3.2
+ hpricot upstream: 0.7 bs: 0.6.161
+ net-ssh upstream: 2.0.11 bs: 2.0.8

So, it connects to gems.rubyforge.org and tell you which gems are outdated. Do you want to keep openSUSE a nice ruby platform? help me. The following features are waiting:

• track ruby package versions
• track other gem repositories, like the rails one or github
• also compare it with version in factory
• nice tables
• caching
• Use gem2rpm and the openSUSE template to automatically create the specs for outdated gems

You can download it from git.opensuse.org, just symlink the .py file to ~/.osc-plugins. It requires rpm-python and Python 2.6 or newer.

My ultimate goal is to get a similar one for zypp/yast that allows to compare our Head repositories with Factory and maintenance repos, show diffs and make submit requests.

So how does it works? How to create your own osc plugin? Look at the source code of the osc-* projects at git.opensuse.org, You will realize it is as easy as:

• Open a .py file
• Create a do_something method, adding the documentation and checking input and arguments
• Use the meta_ API and urllib to get data from the build service
• Use makeurl to build the right authenticated urls to do the requests.

Every distribution has its own conventions to package scripting languages addons (perl modules, ruby gems, etc) as native packages. This allows packages to depend on those addons honoring the package dependencies, and at the same time, look like the addon was installed the scripting language tool (cpan, gem).

openSUSE keeps a repository of gems, thanks to Marcus Rueckert.

For ruby, David Lutterkort (augeas lead developer) created a nice tool to help converting gems into spec files.

I created a template for openSUSE-like rubygem-* packages, and David committed it upstream.

So, to create a package for a gem:

Fetch it:

gem fetch foo

Convert it:

gem2rpm -t opensuse.spec.template ./foo-1.1.gem > rubygem-foo.spec

Build and tweak it. Make sure everything is alright. Some gems work out of the box, some not, but still gem2rpm saves 90% of the effort. Consider submitting it to the project if you are willing to keep it up to date

Some random thoughts about the world, and the past and following years.

Digital society

2007 was quite active on the digital rights topic.

The inability of the industry to catch up with the current society has created a war on digital rights. Software patents on one side. Digital music transformed music labels into mafias and consumers into rival groups. Politicians trying to implement surveillance systems everywhere.

I have the feeling that we will see some progress on the music topic. Record labels will give up, but it will be too late, and if a bunch of major artists start to use some fair system. Something like amiestreet.com or direct selling comes to my mind.

I don’t think something will happen on patents.

It was funny, some weeks ago I got an idea about using gps to associate location to todo items. This plays well with the getting things done methodology where you organize by contexts and not categories. I started prototyping some stuff on android.

Sadly, I found out this simple idea was patented by Fujitsu. Not only that. But I found a program which does that, and the website dissapeared. Another news article about someone researching on the topic and developing a product on that also dissapeared from the map. However, I haven’t yet seen a product from Fujitsu on the topic (the patent is 7 years old). Software patents destroy innovation. Thanks to that stupid patent, you won’t see any product (unless free software) using that.

The web 2.0

Everybody is sick of the Web 2.0 buzz. The Web 2.0 exists.

It is normal that consultants/analysts start to invent new terms because their business depends on the next “big thing” that will “cut costs” and “save millions” to your company. They repeat the same year after year just replacing the term itself.

However the amount of services on the web is growing really fast, and they are all accessible by really standard protocols. Software is becoming just a support medium and the value is being transfered to services: information, storage, security, etc.

Now, there are new layers over that. Phones with gps will bring a new dimension of services based on our location. This is very important. The information we store on the web becomes more relevant if we map it over real-world dimensions: location, time, mood, energy, context. Open source fits here, you can see companies like Google taking advantage of it.

Question: How services will affect open source and/or free software itself? Google contributes quite a lot to open source software. But once you don’t distribute the software, you are not forced to publish modifications. Will other companies follow this path?

Amazon Web Services is another topic. The way they sell on demand “computing power”, “human processing”, “databases” and “storage” is simply amazing.

I would like to see more about “distributed” environments. I am disappointed on how I have to manage my information having 3 computers and one cell phone. There has to be something better than either being off-line and centralized or being online and ubiquitous (where network is available). I want to be ubiquitous, distributed, fail tolerant, and in a simple, pragmatic way. (I don’t want to setup a cluster on my devices).

What about the bubble?. Yes, there is a bubble. There are a bunch of companies that know what they are doing. And thousands of venture capital groups funding whoolalalhzuzu.com ajax websites which implement a calculator or whatever. Those dying is not a bubble, it is natural selection. Most people already know which ones will die after using them for 2 minutes.

I am really excited about the developments in this area and looking forward what is coming here. The direction is clear.

Software

• openSUSE / YaST

  I will leave this for a separate post.

• KDE 4.0

  4.0 is being released in a few days and you will see the most brave release of free software ever. A big bunch of new technologies and visions collected, cooked and packed inside a great community. And better, there is still no big place for politics in KDE, but technical arguments and user experience. Not that all desktops could say the same.

Software Development

Wow, what happened on 2007?

• Software configuration / Version Control

  The growing complexity of open source codebases, plus the need to maintain them for enterprise purposes, brought the topic of version control really hard on the blog sphere. Every blog and developer talked about git. Lot of talk about mercurial and bzr too. 3 version control systems being popular at the same time? The point is that being “distributed” is “the thing”. I personally switched to git, and it solved the “being distributed” part of working with 3 computers in different places. I want to see something like code.google.com with git support.

• Android

  Brilliant. I am waiting for the first phone. Some APIs are ugly. But still prettier than uggly guys that reinvent the wheel poorly, and worse, only on Windows.

Nothing that spectacular on other old topics:

• Java

  While Eclipse is a jewel. Sun is getting better but too slow to move. So slow that it is getting boring to watch.

• Ruby

  We saw the release of ruby 1.9 on December, a very important milestone. At the same time, JRuby is now fast and very compatible, and other implementations are also very active.

• C++

  Even more boring than Java eh?

Politics

• Chile

  The goverment of Michelle Bachelet whose goverment improvisation has made the country again miss the opportunity to develop quickly. Michelle has no strategy at all, so the hope for 2008 is that his sucking team don’t make more mistakes. The public transport system ( Transantiago ) has to start working somehow (both in operation and budget), because till now, it is a joke.

  The opposition hasn’t a good alternative. Nobody is willing to make the important change: universal free and good education, health and social care. Even Michelle, being a socialist, uses the private health and education system.

• Europa

  Spain’s election coming. Seems that Zapatero will be reelected, which seems reasonable. I am a little lost with german politics and I feel like living in a fantasy world. Time to change that. Still, Europe’s economy is going good and living here is awesome. I love it.

• USA

  Discussions on whether they should teach non-science on science class?. War. etc. Uhm… was I writing about a middle west country? I am sad, really sad to see a beautiful country being destroyed, destroying, hating and being hated by almost the entire world, and even worse, considered the biggest threat to the rest of the world.

  From latimes.com:

  36% of European poll respondents — who come from Italy, France, Germany, Britain, and Spain consider America as the No. 1 danger to world peace. Even 35% of American 16- to 24-year-olds identify their own country as the chief danger to peace. The poll was consistent with findings by the Pew Global Attitudes Project, which found that favorable ratings of the U.S. had declined in 26 of 33 countries over the last five years. Europeans next concerns are China, 19%; Iran 17%; Iraq 11%; North Korea 9%; Russia 5%.

  Elections aren’t this year. Lets see how it goes.

Richard Moore comment on Access Anxiety explains why Encapsulation is a good thing. I think both are missing the point. Nobody doubts that encapsulation is a good thing, and Michael Feathers tries to present ruby way as a more “relaxed” thing.

The author was comparing the ‘ruby style’ of direct access to member variables with the getter/setter pattern common in Java code

So what is the ruby style really?

The annoying thing about java setters is that they are functions. So you either start with them (which is overkill for simple things) or don’t use them and then you are lost when you want to add things like Richard showed.

Ruby on the other hand, makes it easy, because you can’t expose member variables, but you can very easily create default accessors.

Michael Feathers example is wrong. Usually you start by actually offering a simple default accessor.

class GlazeObject
  attr_accessor :store, :formatter

  def initialize
    @store = DefaultStore.new
    @formatter = DefaultFormatter.new
  end
end

Now, if you need something more special, like Richard example:

public void setFormatter(Formatter formatter) {
        this.formatter = formatter;
        this.expensive = doCalculation(formatter);
    }

You can just remove attr_accessor :formatter and add to the class:

class GlazeObject
  def formatter
    return @formatter
  end

  def formatter=(f)
    @formatter = f;
     @expensive = doCalculation(f);
  end
end

What is the difference? In ruby, for client code consuming this class, you always see obj.formatter for both cases. In java you have to start from the beginning copy-pasting repetitive accessor code if you don’t want to start changing obj.formatter to obj.getFormatter() and obj.formater = something to obj.setFormatter(something).

C# has a similar feature, called properties.

public class Person
{
  private int _age; 
  public int age
  {
   get
   {
     return _age;
   }
   set
   {
     _age = value;
   }
  }
} 

No matter if we start with a public variable called age, we can move it to setters and getters (which are the providers of encapsulation without having to change the operation mode from variables to functions.

After quite lot of hacking nights, I managed to get some yast2 ruby bindings improvements working.

You might know that ycp is not only a language, but a component technology. Every component using the YaST2 technology can be reused from other YaST components no matter what language they are written in, as long as there are bindings for that. That was the original intention of the ruby bindings.

So for example, there is a component call Storage, which is written in the ycp language. The component (well, it is really a namespace, but YaST sees it as a namespace component) is located in /usr/share/YaST2/modules/Storage.ycp (.ybc for the bytecode compiled version) and offers some medium level functions that YaST uses.

This component needs low level access to the system, for which there is another module called Libstorage ( located in /usr/share/YaST2/modules/LibStorage.pm ) which in this case is written in perl which is a language YaST supports as long as you follows some simple rules.

The reason is because libstorage is a C++ library, and there is no easy way to bind it to the YaST environment, but as YaST supports perl, you can generate perl bindings for the C++ library and all the library functionality will be available to other components in YaST.

So repeat after me. ycp is not only a language. It is a communication protocol, used by YaST component framework.

So, with the ruby bindings, you where able to do:

require 'yast'

m = YaST::Module.new("Storage")
dp = m.GetDiskPartition("/dev/sda1")
dp.each do | key, value |
  puts "#{key} #{value}"
end

But I was never happy with the syntax. The idea was that you construct a dynamic proxy for a YaST component and then call methods on it. This had some disadvantages:

• If the component did not exist, you did not get an error till the method call
• It was not possible to do introspection to the component, because the methods call where implemented using the method_missing hook. So basically the proxy responded to all methods and only raised an error if the equivalent call to the YaST side was not possible.

So I decided to look for a new approach. This is the result till now:

require 'yast'
require 'ycp/storage'

dp = YCP::Storage::GetDiskPartition("/dev/sda1")
dp.each do | key, value |
  puts "#{key} #{value}"
end

Oh yes, you notice. First, there is no need to instantiate the module, a classic require will do it. The bindings modify the built-in require method, so if you require something under ycp/, a module is created on the fly. A real module, a real symbol. You can also import it manually using YCP::import

Now, you can do this:

puts YCP::Storage.methods

When the module is imported, the symbols are declared in the module, so you can ask for them in the module. I still need to work on other symbol types, but it should be straight forward.

The last improvement is that I simplified the C part of the bindings a lot, and moved most of the magic to pure ruby code, which makes easier to work with.

Next improvement, the other way around, making nicer how to call ruby components from any other component.

Surprised to find out that Chris is writing Taskjuggler3 from scratch, and this time, in Ruby:

TaskJuggler 3.x is a from-scratch rewrite of TaskJuggler. It’s currently in its infancy and only supports a minimum feature set. It is not intended for users yet. Please join our developer forum if you are interested in contributing. In contrast to the 2.x version TaskJuggler 3.x is written in Ruby.

There is a git repository in

http://www.taskjuggler.org/git-repos/taskjuggler3.git

in case you want to give it a look.

I just commited support for using the user interface (yes, that is Qt, GTK or ncurses) from the ruby language:

require 'yast'
ui = YaST::Module.new("UI")
YaST::Ui::init("qt")
include YaST::Ui

t = HBox( Label("Welcome to Ruby!"), PushButton("Push me") )

puts "#{t.to_s} #{t.class}"

ui.OpenDialog(t)
ui.UserInput()

Result: