Duncan Mac-Vicar P.

Today I used some of the coolest openSUSE Build Service features: project layering, patches against linked packages and aggregates. I want to write about them.

I needed to test a feature in PackageKit. I am using openSUSE 11.0. However upstream PackageKit does not play nice with 11.0. Stefan Haas fixed this, taking our PackageKit sources, adding some patches, and building them in his home project, which results in a repository here.

My feature involved a patch against PackageKit and then testing it from a client application (to adapt it), so what I wanted to achieve was to use Stefan’s PackageKit plus my patch, packaged in a rpm.

I could just copy Stefan’s sources and patches to my home project, and add my patch, but that would be duplication. Any change Stefan does later would require to copy it again.

Luckily the build service has a feature call linking, so I can link Stefan’s package to my home project. You can do that from osc or from the web user interface (Link package from another project), and that would result in a new package called PackageKit in my project (I created a subproject home:dmacvicar:packagekit for this purpose) with a single file called _link.

This version of PackageKit only builds with Factory’s libzypp, so here we have various options:

• build on top of zypp:svn/openSUSE11.0 which is ZYpp svn built on openSUSE11.0
• link all zypp:svn packages to my project (too inefficient, Adrian would kill me if I start to rebuild ZYpp in every project )
• aggregate all ZYpp openSUSE_11.0 packages from zypp:svn in my project (aggregate explained later)

Because I already have the repo zypp:svn in my repo list, I chose building on top of zypp:svn. (This will mean that in order to use this modified PackageKit repo, you need to add zypp:svn as well )

So now, I can add my patch next to the _link file. However, How to make this patch appear in Stefan’s specfile? Do we need to patch the spec file too? No. The build service does it for you. Just edit the _link file which looks like this:

<link project='home:haass' package='PackageKit'>
    <patches/>
</link>

So it looks like:

<link project='home:haass' package='PackageKit'>

    <patches>
        <add name="pk-outdated-notification.patch" />
    </patches>
</link>

This will insert the patch in the spec file when building. That is real magic.

Sadly, this bleeding edge PackageKit version requires some libtar-devel not available in 11.0. Stefan had a backport to 11.0 in his home project too. Do we need to link the package to our project too? No. Linking here is unnecessary because we don’t really need to rebuild a copy of this package, we only need to make it available in our repository for people installing PackageKit from our project, who don’t have Stefan’s project added in their repository list, that will get libtar grabbed via dependencies, but we don’t care if the metadata grabs the final rpm from Stefan’s project.

That is called “aggregate”. To aggregate a package from another project, we create a new package (called libtar) and add to it a file called “_aggregate” (the web interface does not help you here). Edit that file so it looks like:

<aggregatelist>  
    <aggregate project="home:haass">  
        <package>libtar</package>  
        <package>libtar-devel</package>  
    </aggregate>  
</aggregatelist> 

This will map home:haass repositories 1:1 to our repositories. So if we build our project against openSUSE11.0, it will aggregate our package with home:haass/openSUSE11.0 too. If you don’t have a 1:1 mapping, look the Build Service Tips and Tricks to do more complex aggregates.

Then I am ready to install my new PackageKit:

sudo zypper in -r home_dmacvicar_packagekit PackageKit PackageKit-devel
Reading installed packages...

The following NEW packages are going to be installed:
  PackageKit-devel PackageKit


Overall download size: 1.1 M. After the operation, additional 5.1 M will be used.
Continue? [YES/no]: y

There are features that are present since some time, but people just don’t know about it. This time I will mention one that is very useful to developers.

Imagine you are a contributor to foo, which is also available to the distribution as foo.rpm. However, you install the distro from packages, but compile your foo source tree by hand. Compiling foo requires lot of packages.

zypper to the rescue. If foo is in the distribution, make sure you have the main distribution repository added, plus the source repository. Source packages have a nice feature: their Requires, are the BuildRequires of the binary package.

The source install command (si) has a -d option to install only the dependencies, and not the source package itself. So lets say we want to compile PackageKit, lets install all the compile dependencies:

# sudo zypper si -d PackageKit
Reading installed packages...

The following NEW packages are going to be installed:
  opensp openjade gtk-doc docbook_3 docbook-dsssl-stylesheets sqlite-devel

Then I can go and compile:

PackageKit> ./autogen.sh --libdir=/usr/lib64
  --prefix=/usr --enable-developer --with-security-framework=dummy 
  --sysconfdir=/etc --with-dbus-sys=/etc/dbus-1/system.d
  --enable-zypp --with-default-backend=zypp

Note, you do need the source repository added in order for this to work, otherwise zypper will not be able to find a matching src.rpm package to read the dependencies from.

Federico posts about git interactive rebase to squash commits with build errors together with the fix, which is a nice feature.

However, git design of “commit often” does not mean you need to commit whenever you modify one line, that is a bit overkill.

The git index can be used to incrementally add your work to the point it is really worth committing, and actually you can also interactively pick stuff to be added to the index for later committing just as you can interactively rebase.

I recommend reading the article Embrace the Git Index to know more. Also this blog post mentions the topic.

On the same topic, does anyone knows if it is possible to install gitorious or its fork gitlab on a separate server to the machine actually hosting the repositories (and therefore having the web application push the configuration from the database to the other server via ssh or something? ).

I never expected to see Kopete running natively on Windows, but the KDE-Windows guys have done enormous progress:

See the rest of the article on how to run KDE applications on Windows.

Long time since I haven’t blogged.

The server where I have my blog suffered a big crash, and after they had reenabled most services, the status of the databases was unclear. For some days, my blog was gone, forever.

I had some backup, and also experimented recreating it from google cache using a html parser using hpricot. I was successful in recreating an atom feed from there, but had no much luck with the comments (because atom comments are not very standard). Well, I learned a lot on the way

At the end, the sysadmins were able to recover everything, and my blog came to life again, except for some encoding issue that crashed wordpress, which was solved by the sysadmins without any intervention (thanks guys).

So here I am, back. I will be posting some stuff I had on the queue.

I always have the problem remembering in which branch I am, and typing git branch all the time sucks, so just add this to your profile:

if [ -e /etc/bash_completion.d/git ] ; then
  source /etc/bash_completion.d/git
  export PS1='$(ppwd \l)\u@\h:\w$(__git_ps1 "(%s)")> '
fi

And your prompt will look like this if you are in a git checkout:

dmacvicar@piscola:/space/git/suse/zypp(master)>

DistroWatch praising zypper:

During my usage of Zypper over the past couple of weeks, I’ve learnt to appreciate the convenience it provides to the end user. There is not doubt in my mind that Zypper is the most powerful package management utility available today

Even further goes CNET with their review.

I’d have to say that next to Mac, this is probably the most beautiful installer program I have ever seen. Beauty isn’t just skin deep, however — a lot of time and work has been invested to make the SUSE installer faster and easier to use.

You can read more about the installer and a short interview with Stephan Kulow here.

And the last great paragraph:

If Ubuntu is a Volkswagen, then OpenSUSE is a Mercedes-Benz.

I got some feedback from users about managing software.

Raúl Moratalla. who has been blogging about using 11.0 ZYpp/YaST in 10.3, suggests two changes:

• Updated packages filter.

Adding that feature is really easy. However, 11.0 is almost out.

• New packages filter

This shouldn’t be impossible, but I don’t have an idea on how to do it. I think we don’t have the information on when a package appears in a repository, and even worse, we don’t keep information about the previous refresh when finishing one.

John Tomas asks and suggests an easier way to remove applications. I think the use case is valid, but I don’t like the exact proposal, specifically, I don’t like user interfaces with trees. And adding more check boxes for repo removal clutters the interface more. Moreover, I think this usecase is just part of a more generic one, how to install applications, without worrying about “packages”. This problem is already being solved by, for example, Ubuntu Appinstaller.

This problem could be solved in various way.

• Packages could be “tagged” as such.

However that requires changing the current world, therefore it won’t work.

• Do a .desktop file -> package mapping.

Ubuntu appinstaller does this by scanning the complete distribution archive, and generating special metadata from it. This metadata is distributed as a package called appinstaller-data. This method is sane, but the approach of scanning the distribution archive won’t work with 3rd party repositories.

Stephan Binner has a prototype for the same concept on openSUSE, called app-installer.

Any other ideas on how to solve this problem?

During the development of openSUSE 11.0, we have been reporting in real time cool improvements like the fast installation, how YaST became sexy, how YaST/ZYpp/zypper became fast, how YaST/ZYpp/zypper performs better than others and even that our solver is also really smart.

However, there is something else…

Interoperability

Background

One of the features of our stack is the availability of patches and patterns. The first provide updates in a sense of “fix for a problem” (which can mean various, or none updated packages), while patterns are intelligent groups that can recommend, require and suggest packages in order to make certain functionality available, without being too strict in the specific packages to install.

Unlike in other systems where groups and updates are handled as special entities, ZYpp patterns and patches are just objects with dependencies like packages, and the solver threats them in the same way.

Because rpm installed packages database does not know about patterns and patches, in openSUSE 10.x (and SLE10) those objects are installed in a separate database, only viewable to libzypp. This is hidden to the users, but does not allow for easy management using 3rd party tools.

In addition to that, the patch metadata format is our own extension to the metadata handled by yum, the tool used by Fedora and Centos. That means, even if other distribution provide similar concepts, they will mostly ignore our extended metadata.

This is sad, if we share 90% of the metadata format, why not go further?. Sometimes it is no worth to wait that others do steps in becoming more interoperable with you, so what about doing those steps ourselves?

At this time, Fedora was implementing updates metadata by using a yum plugin and a updateinfo.xml description. Metadata for deltarpm availability is handled via the yum-presto plugin.

Sharing tools and data, a step for Interoperability

Metadata format

In openSUSE 11.0, ZYpp reads patches from updateinfo.xml too! (check 11.0 update repo!). Not only that, our delta rpm availability metadata will be in the same format yum-presto (with some modifications agreed with its author).

How will this benefit you?

• You will be able to use yum stack with updates out of the box with updates and deltarpms, and they will just work.
• You will be able to generate custom patches for openSUSE using existing tools like Bodhi, from Fedora, or generate custom deltarpm information using the yum-presto included tools.
• We are working hard to get the ZYpp/YaST stack to build on Fedora (and other distros), in a near future, you will be able to enjoy ZYpp performance and features on Fedora with their own repositories!
• We decoupled the delta-rpm information from patches, so we may start adding delta-rpm to normal factory packages and it will start to work out of the box!
• Much more!

Handling of patches and patterns

As we mentioned, in 10.x codebase we used to install patterns and patches in a special database. This is no longer the case.

Patterns and patches are no longer installed, which means your system is rpm only! Patches are shown now as (un)satisfied (and (un)relevant). Which means you have all the requirements to consider them present.

All the information of patches and patterns (and products) is extra information that openSUSE applications use to add more value to you. So if you for example remove a repository offering patches, then you just lose the information about which patches do you have, the real information is the rpm packages you have installed. When you re-add the update repository, and you can immediately see which patches published affect you, which ones are irrelevant, and which ones are relevant but you don’t need because your packages are up to date.

Patterns and patches become “advice” and “value”, not extra non-compatible information.

How will this benefit you?

• Simpler system.
• No conflicts because using 3rd party tools, “rpm by hand” or our native tools.

openSUSE, PackageKit enabled

PackageKit is the new actor in the package management world. It is a thin layer that provides applications access to the package management system as a DBUS service. You may heard about it because Fedora 9 is coming PackageKit enabled. How it benefits you?

• Role based (non-root) package management, via PolicyKit.
• Sharing of upstream tools across distributions.
• Gives the desktop the chance to integrate with software manipulation.

So, openSUSE 11.0 is fully PackageKit enabled. You will be able to use all PackageKit compatible applications on openSUSE and they will use the ZYpp stack underneaths. Not only that it is enabled, but our hackers Scott Reeves and Stefan Haas did an amazing job on the backend, I would dare to say it is one of the most robust backend implementations, and it fully benefits from the ZYpp speed and features.

Future

All this improvements are available now. May be you are already enjoying them in Factory. However this opens the door for new possibilities, just a few examples come to mind:

• The openSUSE Build Service, the great software building platform from our openSUSE team, builds packages for all major distributions since long time. The build service could allow to enter and generate patch information for fixed bugs and the update/patch information will be compatible across yum/Fedora and openSUSE. Same with deltarpm information.
• We could extend ZYpp parser to understand Fedora groups stored in comps.xml and threat them as patterns.
• Do you have more ideas?

Community involvement

We welcome any help on creating more interoperability possibilities, especially about building the ZYpp stack and YaST on Fedora, Mandriva and others. There are already some packages building in the build service, but we still have a long way to go.

In my last post, I showed how sat-solver (solver’s ZYpp uses) would solve correctly the “case 2″ included in smart’s README, an exercise smart uses to claim its “smartness” (because apt and yum can’t handle them).

How does it with case 3?

That’s another interesting case which was tested with APT-RPM and YUM.

In this case, there’s a package A version 1.0 installed in the system, and there are two versions available for upgrading: 1.5 and 2.0. Version 1.5 may be installed without problems, but version 2.0 has a dependency on B, which is not available anywhere.

In this case, the best possibility is upgrading to 1.5, since upgrading to 2.0 is not an option.

Here both yum and apt fail:

Just like APT, YUM selected version 2.0 and didn’t consider the availability of an intermediate version.

But smart does the right thing:

Smart correctly selects the intermediate version 1.5, which is the only viable possibility given the current options.

So, we setup a repository packages.xml with A version 1.5 and 2.0 ( 2.0 requiring an non-existing B) and a system.xml representing the installed packages, with only A 1.0 installed. We put all together in a test.xml file, and select a “upgrade” action.

We run the already introduced deptestomatic:

# deptestomatic test.xml
>!> Solution #1:
>!> upgrade A-1.0-1.noarch => A-1.5-1.noarch[test]
>!> !unflag A-2.0-1.noarch[test]
>!> installs=0, upgrades=1, uninstalls=0

And we confirm, that ZYpp (sat-solver) would also select A 1.5 as expected.