Duncan Mac-Vicar P.

During the development of openSUSE 11.0, we have been reporting in real time cool improvements like the fast installation, how YaST became sexy, how YaST/ZYpp/zypper became fast, how YaST/ZYpp/zypper performs better than others and even that our solver is also really smart.

However, there is something else…

Interoperability

Background

One of the features of our stack is the availability of patches and patterns. The first provide updates in a sense of “fix for a problem” (which can mean various, or none updated packages), while patterns are intelligent groups that can recommend, require and suggest packages in order to make certain functionality available, without being too strict in the specific packages to install.

Unlike in other systems where groups and updates are handled as special entities, ZYpp patterns and patches are just objects with dependencies like packages, and the solver threats them in the same way.

Because rpm installed packages database does not know about patterns and patches, in openSUSE 10.x (and SLE10) those objects are installed in a separate database, only viewable to libzypp. This is hidden to the users, but does not allow for easy management using 3rd party tools.

In addition to that, the patch metadata format is our own extension to the metadata handled by yum, the tool used by Fedora and Centos. That means, even if other distribution provide similar concepts, they will mostly ignore our extended metadata.

This is sad, if we share 90% of the metadata format, why not go further?. Sometimes it is no worth to wait that others do steps in becoming more interoperable with you, so what about doing those steps ourselves?

At this time, Fedora was implementing updates metadata by using a yum plugin and a updateinfo.xml description. Metadata for deltarpm availability is handled via the yum-presto plugin.

Sharing tools and data, a step for Interoperability

Metadata format

In openSUSE 11.0, ZYpp reads patches from updateinfo.xml too! (check 11.0 update repo!). Not only that, our delta rpm availability metadata will be in the same format yum-presto (with some modifications agreed with its author).

How will this benefit you?

• You will be able to use yum stack with updates out of the box with updates and deltarpms, and they will just work.
• You will be able to generate custom patches for openSUSE using existing tools like Bodhi, from Fedora, or generate custom deltarpm information using the yum-presto included tools.
• We are working hard to get the ZYpp/YaST stack to build on Fedora (and other distros), in a near future, you will be able to enjoy ZYpp performance and features on Fedora with their own repositories!
• We decoupled the delta-rpm information from patches, so we may start adding delta-rpm to normal factory packages and it will start to work out of the box!
• Much more!

Handling of patches and patterns

As we mentioned, in 10.x codebase we used to install patterns and patches in a special database. This is no longer the case.

Patterns and patches are no longer installed, which means your system is rpm only! Patches are shown now as (un)satisfied (and (un)relevant). Which means you have all the requirements to consider them present.

All the information of patches and patterns (and products) is extra information that openSUSE applications use to add more value to you. So if you for example remove a repository offering patches, then you just lose the information about which patches do you have, the real information is the rpm packages you have installed. When you re-add the update repository, and you can immediately see which patches published affect you, which ones are irrelevant, and which ones are relevant but you don’t need because your packages are up to date.

Patterns and patches become “advice” and “value”, not extra non-compatible information.

How will this benefit you?

• Simpler system.
• No conflicts because using 3rd party tools, “rpm by hand” or our native tools.

openSUSE, PackageKit enabled

PackageKit is the new actor in the package management world. It is a thin layer that provides applications access to the package management system as a DBUS service. You may heard about it because Fedora 9 is coming PackageKit enabled. How it benefits you?

• Role based (non-root) package management, via PolicyKit.
• Sharing of upstream tools across distributions.
• Gives the desktop the chance to integrate with software manipulation.

So, openSUSE 11.0 is fully PackageKit enabled. You will be able to use all PackageKit compatible applications on openSUSE and they will use the ZYpp stack underneaths. Not only that it is enabled, but our hackers Scott Reeves and Stefan Haas did an amazing job on the backend, I would dare to say it is one of the most robust backend implementations, and it fully benefits from the ZYpp speed and features.

Future

All this improvements are available now. May be you are already enjoying them in Factory. However this opens the door for new possibilities, just a few examples come to mind:

• The openSUSE Build Service, the great software building platform from our openSUSE team, builds packages for all major distributions since long time. The build service could allow to enter and generate patch information for fixed bugs and the update/patch information will be compatible across yum/Fedora and openSUSE. Same with deltarpm information.
• We could extend ZYpp parser to understand Fedora groups stored in comps.xml and threat them as patterns.
• Do you have more ideas?

Community involvement

We welcome any help on creating more interoperability possibilities, especially about building the ZYpp stack and YaST on Fedora, Mandriva and others. There are already some packages building in the build service, but we still have a long way to go.

In my last post, I showed how sat-solver (solver’s ZYpp uses) would solve correctly the “case 2″ included in smart’s README, an exercise smart uses to claim its “smartness” (because apt and yum can’t handle them).

How does it with case 3?

That’s another interesting case which was tested with APT-RPM and YUM.

In this case, there’s a package A version 1.0 installed in the system, and there are two versions available for upgrading: 1.5 and 2.0. Version 1.5 may be installed without problems, but version 2.0 has a dependency on B, which is not available anywhere.

In this case, the best possibility is upgrading to 1.5, since upgrading to 2.0 is not an option.

Here both yum and apt fail:

Just like APT, YUM selected version 2.0 and didn’t consider the availability of an intermediate version.

But smart does the right thing:

Smart correctly selects the intermediate version 1.5, which is the only viable possibility given the current options.

So, we setup a repository packages.xml with A version 1.5 and 2.0 ( 2.0 requiring an non-existing B) and a system.xml representing the installed packages, with only A 1.0 installed. We put all together in a test.xml file, and select a “upgrade” action.

We run the already introduced deptestomatic:

# deptestomatic test.xml
>!> Solution #1:
>!> upgrade A-1.0-1.noarch => A-1.5-1.noarch[test]
>!> !unflag A-2.0-1.noarch[test]
>!> installs=0, upgrades=1, uninstalls=0

And we confirm, that ZYpp (sat-solver) would also select A 1.5 as expected.

YaST ideas and research

We moved the summer of code ideas, and other projects to the YaST/Research page. We would like to find all the experiments that are lying around in the tmp and experimental branches and put them together in that page so previous attempts and knowledge is not lost.

Installation Branding

Some information on branding the installation is available on YaST/Tips#Brandingtheinstallation.

Package groups

The first thing that caught my attention in the rpm world was package groups. It is a 4 deep level tree which has become pretty useless with time. They are also not consistent across rpm distributions. I agree that they are interesting metadata (because they contain keywords), but nothing you want to display in a user interface.

So I started to ask people around. Do you use the groups tree?. “YES” was the first reply. I was surprised. Were my initial guesses wrong?. “I use it to select the zzzAll group, which is the only way to select all packages”. Interesting point, any effort to change this should take that into account.

Luckily all the answers were the same. The famous “zzzAll” group was the only reason why people used that filter.

So when looking for a way to make that screen more easy to the eyes, PackageKit groups seemed like the way to go, as they are already in use in some applications.

That is how package groups looked like before:

After matching the rpm groups to PackageKit groups, this is the result. Easy.

Notice that “All packages” is still there. Also there is a recommended and suggested groups. The interesting thing is, that if you look at other applications, like Gnome PackageKit install, which is a braindead application (no features) compared to YaST, but still, now they look pretty consistent with each other:

Before you cry for your old 4 level deep tree, you can still reach the groups via search. Search will look into the groups field and also the generated keywords in the metadata (tags).

Package search

Now uses the sat solver APIs to query the solv files, which is much faster and we have transparent access to more attributes.

openSUSE 11.0 beta is coming. This means that all the pieces of the update stack have to be in place by then.

While reading Review: Hat Trick For Fedora 9 Beta, it caught my attention that the author says “Fedora continues to wear the innovation hat” because some of the changes described there. I did not see any killer feature openSUSE 11.0 does not have too, and a few ones are worth to highlight:

• KDE 4 integration: No distro will ever beat openSUSE here you know that. openSUSE is a multi desktop distribution and both the Gnome and the KDE team are unbeatable. Period. (I am objective today )
• The installation experience and look essentially remained the same from Fedora 8: Sorry guys, but YaST installer looks far cooler than in our last release. The experience is also much different.
• except there is now support for resizing ext2, ext3, and NTFS partitions from the installer: zzzzzzz. YaST resizes partitions before Linux was invented.
• The installer also checks for password strength for the root account: YaST in the 80’s ?
• The new package management solution, PackageKit, is another interesting feature: openSUSE 11.0 will have native PackageKit support (the backend is upstream) and it works really well. The yum backend has no good reputation. The ZYpp backend in 11.0 inherits all the unbeatable speed of ZYpp 4.x and it is robust at the same time.

Sorry guys, the innovation hat is green. Ok, enough with articles. Lets back to 11.0 beta.

We talked about package management speed, we talked about new looks and features already. However our work around patches and patterns was still missing.

During the last weeks, we have been working on this and now all the pieces start to fall together. Click on any image to see it in full size. Also note that ugly scrollbar in the disk usage is was also fixed already.

You may remember the pattern selector:

New pattern selector:

If you go to the repository view, it was a little boring. openSUSE has the Build Service, which has generated a big community of repositories. Visually, we want to make a difference to the eye if a repo is a normal repository, or the home project of a friend, or a well known repository, or may be a update repository. Result?

The old patch view was confusing. The category column never had space for itself.

So we introduced categories just like in the patterns view.

There are still some details. In that view the reboot patch should not be shown in the default filter, because I don’t have the package the patch fixes installed. For some reason isRelevant() is not working there. Same with the security patch. It shows correctly the fact that the patch is satisfied, but therefore it should be hidden.

In a next post I will write a bit about how patterns, products and patches are handled now, plus other features.

Package management is the most used task by our users according to the YaST survey.

Not happy with the super speed and making other similar tools eat dust, our hackers continue to check in cool stuff:

Katarina showed the new face of our text-mode package selector:

Ladislav checked in a cool change. If you look at the package selector main screen. You will notice there is an option in the menu to go to the repository configuration.

Clicking there brings you to the repository editor without needing to switch modules:

And now from there, you can go to edit which keys you trust for repositories:

Because the Google Summer of Code deadline was extended, I just added two projects I consider interesting for YaST:

Automatic generation of YCP bindings

Right now, if you want to access low level libraries from YCP code, you have two options:

• Create manual bindings, may be with help of /usr/share/YaST2/data/devtools/bin/generateYCPWrappers script. (used by package-bindings and libyui ycp bindings )
• Wrap your library using swig, and then generate perl bindings from that, and then use the module using YaST perl bindings (libstorage is accessed this way).

This problem could be solved in various ways:

• Create a SWIG module that allows ycp glue as output.

This approach would get all the swig parsing magic for free.

• Extend generateYCPWrappers so it could wrap any C library in a easy and automatic way. C++ is not required, as YCP is not object oriented.

Expected result:

Get package bindings or libstorage output which don’t depend on perl bindings and are automaticaly updated if the target API changes.

• Required knowledge: C++, YaST2, languages
• Skill level: medium
• Willing mentors: me!

Build YaST using cmake

YaST builds using autotools and a automated layer of scripts over that (called y2conf, y2make, etc).

Some modules as libzypp and friends, yast2-qt and ruby bindings build using cmake.

It would be great to build complete YaST using cmake. This would bring benefits like:

• Access to the CTest test framework
• out of the box out of source builds
• Easier and more intuitive.

This task means:

• making yast2-core to compile using cmake
• create devtools skeletons for modules and agents using cmake
• add support for devtools utilities which may break
• port all modules. (porting 3 would be a challenge, the others is mostly mechanical work once core and some problematic ones compile).

Automating the conversion of modules would also be a goal.

• Required knowledge: cmake, building, packaging
• Skill level: easy
• Willing mentors: me!

If these projects aren’t for you, there are more ideas in the wiki page. Check it out!.

This week arstechnica has a really nice selection of articles:

• Komodo Edit 4.3 was released, which is the first version of this scripting editor released as opensource. The editor is based on Mozilla XUL technology, which talks very good about its extensibility. I could not find a tarball with the source, only svn repository and binary tarballs.

• Nice article about filesystems.

• Article about Firefox 3 memory consumption .

Keeping a permanent build of svn on the build service motivated me to try to build it on other non-SUSE based distros. On the way I discovered not only simple things as different package names, but linking problems, compile errors, etc.

In the past it did not make much sense to try libzypp outside of SUSE as we were trying to catchup with other tools speed. But now that libzypp speed outperforms any other tool in speed, while keeping it complete set of features, we may start thinking about taking over the world. May be other distros want to use libzypp, or why not, the small and powerful sat-solver library alone.

So, since today I got the first successful build of sat-solver/libzypp/zypper on Fedora 8. Mandriva is also close, but I still get a compiler error I should not get when compiling the rpm backend.

At the beginning, it may be a consuming effort, but once it is a continuous process, our software should adopt a more agnostic view of the world and just compile there out of the box. If anyone has the time to test it, I would be interested in the results

By the way, our colleagues at internal tools finished processing the FOSDEM talk’s videos we had in the openSUSE Developer’s room. You can find the ogg video files here. Also you can watch on google video here.

I am not good at keyboard shortcuts. I started using mice very early. Therefore anytime I post a video recording my screen, I get more comments about my stupid ways to clear the screen or move the cursor, than about the content itself.

However, I like to change my habits to be more efficient and I keep trying new stuff, and old stuff too.

I tried learning vim, and I was successful in using it during the last years for console usage and basic file editing. However, I was never able to remember more than opening/saving files and deleting lines. I was using Kate for most of my coding needs. I tried switching from Kate to vim for coding like three or four times without any success.

During this trip, and brought some emacs tutorials with me, in order to give it a try. I was expecting something horrible, but surprise, after a few hours, I could do much more than I was able to do with vim, and I had no big problems with the ctrl-something key shortcuts (I have yet to try on my Microsoft Natural keyboard though).

I am now in the process of setting up some packages I need (code browsing and completion) and till now I am very happy. There are a couple of things that really suck:

• Setting up the environment sucks because you have to do it from scratch. There are no sane defaults for anything. No thing that ask you what do you do, and setup something that makes sense to start with.

• XEmacs and Emacs features.They forked one from the other and at some point XEmacs had more features, but now both have things the other hasn’t and you can’t tell exactly what it is.

• XEmacs and Emacs user interface. I discarded XEmacs because it has this horrible tcl/tk look and feel, like using its own widgets over xlib. It had a experimental gtk user interface based on 1.x, but its status is not really clear. At least our package does not enable it. There are screenshots of Emacs running on OS X that look very sweet. Sad that all efforts of bringing Qt to emacs are more or less dead.

For now, I choose GNU Emacs, just because I have some sense of aesthetics and usability. XEmacs is and look like 1970. With GNU Emacs I only have to suffer of the gtk “original” file dialog but I suffer from it in lot of gtk-infected programs (starting with Firefox) so it is not very serious. I can live with it, and I am sure I can use KDE file dialog by using some lisp magic and the kdialog command.

Other pending issues:

• setting up cedet/ecb to get auto completion and code browsing.

• setting up git mode and learn how to use it

• integrate cmake mode and a sane way to use the compile commands out of the source tree

May be next time I record a video, you will see me using ctrl-L instead of typing “clear”

One of the steps to build more community around YaST is to allow people to try it. Not every user uses factory but lot of people want to try one or more punctual recent features. This is not easy, because developers can’t invest much time making sure the latest developments can work seamlessly outside factory. But we can do more. If it compiles and (available) tests run, then we are more or less sure it should run.

In the last week we have been playing with the YaST and build systems in order to organize some YaST and zypp repositories abandoned in the openSUSE build service.

I introduce to you the new layout:

YaST:SVN are YaST packages submited from our code repository sources on a frequent basis. I am trying to make this as automated as possible, so I have internal scripts that submit packages when the repository compiles. Right now, I use this repo to test code that hasn’t been committed to svn yet, like build fixes on other distributions. YaST:SVN builds on top on another repository, called zypp:svn, which is the zypp stack from svn. The svn series is build for Factory, released versions and also I added Fedora and Mandriva, even if it does not build right now. I think making the stack to work in other distributions is a place where the community can make a big contribution. I made satsolver library to build on Mandriva with about one hour of trial-error work. The value is that all spec file modifications are now in the svn repository and not as patches floating around there.

On the other hand, we have the Backport series. YaST:Backport and zypp:Backport (the first builds on top of the second). These repositories are just package linked to the Factory ones, but the repository builds for 10.3, and I am trying to make it build in other Novell editions (SLE, 10.2, etc) and Mandriva/Fedora as well. We don’t need to update this repository, as it rebuilds when Factory versions are submited, which makes it more stable than the svn series of the repositories.

In the future, If we get enough help from the community to build all packages on something different than factory, I would like to create a Experimental series, which are packages linked to the svn series, plus patches from the community that are made upstream after testing them. That way the svn series of the repositories could turn themselves in being pure svn code, and submitted in a automatic way.

YaST is yours, so your feedback and help is welcome. Right now you can find there a zypp and YaST stack from svn and Factory, built for 10.3. YaST modules are missing yet. If you want to help and have submit access to this repositories, we would gladly do it, just ask for it in our yast-devel or zypp-devel mailing lists.