Duncan Mac-Vicar P.

Credits

The following cool stuff would have not been possible without the hard work of Jan Kupec, Michael Andres and Michael Calmer, who implemented and tested this stuff. The original concept dates back from ZLM and the Novell Customer Center.

Usecase #1: The project with multiple repositories and layers

Imagine the following usecase (with this one I am using some real request from our KDE guys)

The build service provides a KDE4 repository. Which in turn requires the Qt4 repository, because is built on openSUSE 11.0 + the new Qt4 repo.

When looking at this problem, repository dependencies is what comes to head in the first place. But forget about them. If package dependencies are complicated right now, imagine adding a secondary (and duplicated) layer of information. Packages already know their dependencies.

Now imagine our KDE guys can provide an URL, which you add to zypper. And this url returns a dynamic list of repositories. And zypper adds and remove repositories based on the information returned by this url on every refresh.

Usecase #2: Update repositories based on the customer

This is actually where services where originated. Services were present in Novell ZenWorks. How it works?

The service url nu.novell.com is added to the system. But in this url also a customer id is present as a http username. When you registered, Novell knows the product this customer is linked to, and can return a dynamic list of update repositories based on the customer preferences, products and entitlements and the customer does not need to keep them in sync.

Now that we don’t have Zenworks in the base system, we still want this cool functionality for our customers.

Technically, this even allows us to offer hotfixes to L3 supported customers on the fly!

Usecase #3: Dynamic repository collections

You are a build service user, and you have an account, and of course you have a list of watched projects you are interested to. What if you could keep your system repositores in sync with your watched project list.

Or what if the build service could offer a service based on keywords or other data: like http://build.opensuse.org/services/mostpopular/repo/repoindex.xml would contain dynamically the 15 most popular repositories. You add that service, and then ZYpp does the work for you of adding new popular repositories, and remove the old ones.

Quick test

For a quick test, I needed a service provider, and I have none. I could have used our Subscription Management Tool, which can act as a local proxy replacement of a Novell Update service provider, but I decided to implement a quick and dirty service that could be used as an example for other people that have nice ideas for this stuff.

So I created a rails application, a very simple one, that returns a dynamic repository index by searching for the “emacs” keyword using the opensuse-community search API.

require 'open-uri'
require 'rexml/document'
require 'uri'

include REXML

class RepoController < ApplicationController

  def index

    # do a package search on a keyword
    f = open('http://api.opensuse-community.org/searchservice/Search/Simple/openSUSE_110/emacs')
    buffer = f.read
    doc = Document.new(buffer)
    repourls = Set.new
    
    counter = 0
    doc.elements.each('ns2:packages/package/repoURL') do | repourl |
      # get all repourls
      repourls.add repourl.text.to_s
    end

    # create the repoindex
    outdoc = Document.new
    root = Element.new('repoindex')
    outdoc.add_element root
    counter = 0
    repourls.each do |repourl|
      al = URI.parse(repourl).path.gsub(/\/repositories\//, '').gsub(/[\:]/, '').gsub(/\//, '_')
      root.add_element 'repo'  , { 'url' => repourl,
        'alias' => al }
    end
    
    render :xml => outdoc.to_s
  end
end

And I need also to configure a route, as the service is searched in the service url + “repo/repoindex.xml”.

  map.connect ':controller/repoindex.xml', :action => 'index'

Requesting my new service http://localhost:3000/repo/repoindex.xml returns me the following.

<repoindex>
<repo url="http://download.opensuse.org/repositories/Education:/desktop/openSUSE_11.0" alias="Education_desktop_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/repositories/server:/mail/openSUSE_11.0" alias="server_mail_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/repositories/home:/tiwai/openSUSE_11.0" alias="home_tiwai_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/repositories/home:/sjcundy/openSUSE_11.0" alias="home_sjcundy_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/repositories/home:/hmacht/openSUSE_11.0" alias="home_hmacht_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/repositories/home:/marcinzalewski/openSUSE_11.0" alias="home_marcinzalewski_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/repositories/home:/beyerle/openSUSE_11.0" alias="home_beyerle_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/repositories/home:/dsteuer/openSUSE_11.0" alias="home_dsteuer_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/repositories/M17N/openSUSE_11.0" alias="M17N_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/repositories/home:/jefferyfernandez/openSUSE_11.0" alias="home_jefferyfernandez_openSUSE_11.0"/>
<repo url="http://download.opensuse.org/distribution/11.0/repo/oss/suse" alias="_distribution_11.0_repo_oss_suse"/>
<repo url="http://download.opensuse.org/repositories/home:/maw:/bzr/openSUSE_11.0" alias="home_maw_bzr_openSUSE_11.0"/>
</repoindex>

Adding your service can be done with zypper:

# zypper sa http://localhost:3000 myservice
Service 'myservice' has been successfully added.          

As you may note, services have their own commands. every repository not part of a service is also a service. So working with repositories at the service level basically groups all repositories that belong to the same service.

Now, when you refresh the service:

# zypper refs
Refreshing service 'myservice'.
Adding repository 'Education_desktop_openSUSE_11.0' [done]
Adding repository 'server_mail_openSUSE_11.0' [done]
Adding repository 'home_tiwai_openSUSE_11.0' [done]
Adding repository 'home_sjcundy_openSUSE_11.0' [done]
Adding repository 'home_hmacht_openSUSE_11.0' [done]
Adding repository 'home_marcinzalewski_openSUSE_11.0' [done]
Adding repository 'home_beyerle_openSUSE_11.0' [done]
Adding repository 'home_dsteuer_openSUSE_11.0' [done]
Adding repository 'M17N_openSUSE_11.0' [done]
Adding repository 'home_jefferyfernandez_openSUSE_11.0' [done]
Adding repository '_distribution_11.0_repo_oss_suse' [done]
Adding repository 'home_maw_bzr_openSUSE_11.0' [done]
Repository 'FATE' is up to date.
Repository 'KDE 4.1.x Packages (openSUSE_11.0)' is up to date.
Repository 'The (default) Factory KDE 4 desktop with extra apps (openSUSE_11.0)' is up to date.
Repository 'Current Qt 4.x packages (openSUSE_11.0)' is up to date.
Retrieving repository 'VirtualBox' metadata [done]
Building repository 'VirtualBox' cache [done]
Repository 'Latest YaST svn snapshots (openSUSE_11.0)' is up to date.
Repository 'Python and Python Modules (openSUSE_11.0)' is up to date.
Repository 'home:jnweiger' is up to date.
Repository 'PackageKit experiments (openSUSE_11.0)' is up to date.
Repository 'Network Utilities (openSUSE_11.0)' is up to date.
Repository 'openSUSE-11.0' is up to date.
Repository 'openSUSE-11.0-NONOSS' is up to date.
Repository 'openSUSE-11.0-Update' is up to date.
Repository 'openSUSE.org tools (openSUSE_11.0)' is up to date.
Repository 'outdated' is up to date.

Did you see how zypper automatically added the needed repositories? (it also would remove them if in the future the service does not list them).

Now, let list them:

Now if we list the services:

# zypper ls
#  | Alias                       | Name                                                                | Enabled | Refresh | Type
---+-----------------------------+---------------------------------------------------------------------+---------+---------+-------
1  | myservice                   | myservice                                                           | Yes     | No      | ris
2  | FATE                        | FATE                                                                | Yes     | Yes     | rpm-md
3  | KDE_KDE4_Factory_Desktop    | KDE 4.1.x Packages (openSUSE_11.0)                                  | Yes     | Yes     | rpm-md
4  | KDE_KDE4_Factory_Extra-Apps | The (default) Factory KDE 4 desktop with extra apps (openSUSE_11.0) | Yes     | Yes     | rpm-md
5  | KDE_Qt                      | Current Qt 4.x packages (openSUSE_11.0)                             | Yes     | No      | rpm-md
6  | Virtualization:VirtualBox   | VirtualBox                                                          | Yes     | No      | rpm-md
7  | YaST_SVN                    | Latest YaST svn snapshots (openSUSE_11.0)                           | Yes     | Yes     | rpm-md
8  | devel_languages_python      | Python and Python Modules (openSUSE_11.0)                           | Yes     | Yes     | rpm-md
9  | home:jnweiger               | home:jnweiger                                                       | Yes     | Yes     | rpm-md
10 | home_dmacvicar_packagekit   | PackageKit experiments (openSUSE_11.0)                              | Yes     | No      | rpm-md
11 | network_utilities           | Network Utilities (openSUSE_11.0)                                   | Yes     | Yes     | rpm-md
12 | openSUSE-11.0               | openSUSE-11.0                                                       | Yes     | No      | yast2
13 | openSUSE-11.0-NONOSS        | openSUSE-11.0-NONOSS                                                | Yes     | No      | yast2
14 | openSUSE-11.0-Update        | openSUSE-11.0-Update                                                | Yes     | No      | rpm-md
15 | openSUSE_Tools              | openSUSE.org tools (openSUSE_11.0)                                  | Yes     | No      | rpm-md
16 | outdated                    | outdated                                                            | Yes     | No      | rpm-md
17 | packman                     | packman                                                             | Yes     | Yes     | rpm-md
18 | rpms                        | rpms                                                                | Yes     | No      | rpm-md
19 | ruby                        | Ruby                                                                | No      | No      | rpm-md
20 | zypp_svn                    | ZYPP SVN Builds (openSUSE_11.0)                                     | Yes     | Yes     | rpm-md

Do you see that all the added repositories are “hidden” behind the added service.

If you want to see them, then you need to list the repositories instead:

# zypper lr
#  | Alias                               | Name                                                                | Enabled | Refresh
---+-------------------------------------+---------------------------------------------------------------------+---------+--------
1  | Education_desktop_openSUSE_11.0     | Education_desktop_openSUSE_11.0                                     | No      | Yes
2  | FATE                                | FATE                                                                | Yes     | Yes
3  | KDE_KDE4_Factory_Desktop            | KDE 4.1.x Packages (openSUSE_11.0)                                  | Yes     | Yes
4  | KDE_KDE4_Factory_Extra-Apps         | The (default) Factory KDE 4 desktop with extra apps (openSUSE_11.0) | Yes     | Yes
5  | KDE_Qt                              | Current Qt 4.x packages (openSUSE_11.0)                             | Yes     | No
6  | M17N_openSUSE_11.0                  | M17N_openSUSE_11.0                                                  | No      | Yes
7  | Virtualization:VirtualBox           | VirtualBox                                                          | Yes     | No
8  | YaST_SVN                            | Latest YaST svn snapshots (openSUSE_11.0)                           | Yes     | Yes
9  | _distribution_11.0_repo_oss_suse    | _distribution_11.0_repo_oss_suse                                    | No      | Yes
10 | devel_languages_python              | Python and Python Modules (openSUSE_11.0)                           | Yes     | Yes
11 | home:jnweiger                       | home:jnweiger                                                       | Yes     | Yes
12 | home_beyerle_openSUSE_11.0          | home_beyerle_openSUSE_11.0                                          | No      | Yes
13 | home_dmacvicar_packagekit           | PackageKit experiments (openSUSE_11.0)                              | Yes     | No
14 | home_dsteuer_openSUSE_11.0          | home_dsteuer_openSUSE_11.0                                          | No      | Yes
15 | home_hmacht_openSUSE_11.0           | home_hmacht_openSUSE_11.0                                           | No      | Yes
16 | home_jefferyfernandez_openSUSE_11.0 | home_jefferyfernandez_openSUSE_11.0                                 | No      | Yes
17 | home_marcinzalewski_openSUSE_11.0   | home_marcinzalewski_openSUSE_11.0                                   | No      | Yes
18 | home_maw_bzr_openSUSE_11.0          | home_maw_bzr_openSUSE_11.0                                          | No      | Yes
19 | home_sjcundy_openSUSE_11.0          | home_sjcundy_openSUSE_11.0                                          | No      | Yes
20 | home_tiwai_openSUSE_11.0            | home_tiwai_openSUSE_11.0                                            | No      | Yes
21 | network_utilities                   | Network Utilities (openSUSE_11.0)                                   | Yes     | Yes
22 | openSUSE-11.0                       | openSUSE-11.0                                                       | Yes     | No
23 | openSUSE-11.0-NONOSS                | openSUSE-11.0-NONOSS                                                | Yes     | No
24 | openSUSE-11.0-Update                | openSUSE-11.0-Update                                                | Yes     | No
25 | openSUSE_Tools                      | openSUSE.org tools (openSUSE_11.0)                                  | Yes     | No
26 | outdated                            | outdated                                                            | Yes     | No
27 | packman                             | packman                                                             | Yes     | Yes
28 | rpms                                | rpms                                                                | Yes     | No
29 | ruby                                | Ruby                                                                | No      | No
30 | server_mail_openSUSE_11.0           | server_mail_openSUSE_11.0                                           | No      | Yes
31 | zypp_svn                            | ZYPP SVN Builds (openSUSE_11.0)                                     | Yes     | Yes

There you have them all. You should also note that service configuration is very similar to repository configuration. Only services.d intead of repos.d.

Conclusion

As you can see, services implement a kind of “black box” repository, that allows subscription to dynamic repository sets. The concept is simple, optional (you can not use them at all), and the potential is very big, if people start to offer them.