Tilting at windmills

Duncan Mac-Vicar P.'s blog

fast package management


I haven’t blogged since some time. We have been working hard to get package management stack changes in so we can have them in one of the next alphas. I wanted to make a small pause and write about it because communication is (sometimes) as important as coding ;-).

See it yourself:


### History

You know how was the package management stack on 10.1. On 10.3 we reached a point where we were fast enough, but with really bad performances in some cases, especially cold starts, and big lists of repositories, which is usually the usecase where the people complaining about speed are: factory users. Some of this bad performance was caused by design mistakes (like not using 1 db per repository), and others because sqlite horrible performance after fragmentation occurs.

This started sometime last year, when Michael Schröder started to show some people a [new solver][2] which was much faster than current libzypp solver. At the same time, he designed some memory and cpu efficient structures and file formats to deal with the huge amounts of metadata. The solver operated later over this optimized pool.

One of the features of the solver is its simplicity. Current libzypp operates over a dependency problem at the resolvable level while the sat solver converts the pool into a set of rules and then operates the problem as a [boolean satisfiability problem][3], which makes it codebase concentrate only on this, making it much smaller and simpler. The error messages on conflicts are supposed to be more human friendly too.

### The Road

During the YaST workshop times, [coolo][4] started to try the libzypp testsuite with the solver, by creating tools to use the existing testcases with it, and making sure it could serve as our solver.

Once coolo came and said, “ok, all testcases pass”. So we met and agreed on some milestones to see the feasibility of integration with libzypp. One of the milestones was to use the new solver without replacing all the infrastructure libzypp had. That is, taking libzypp pool, converting the transactions into rules, solve, convert them back. Stefan Schubert did a great work in this area, and actually if you are using factory you are already using the sat solver.

To familiarize myself with the solver at that time, I ported it to build with cmake and implemented swig bindings (focusing on ruby ones), which later [Klaus][5] started to enhance and experiment new concepts on top of them.

There was one part the sat solver did not take into account and it was the huge amount of non-solvable data you need to have cached. It is easy to claim to be fast if you ignore 95% of the data, so Michael Matz started to enhance the sat library by designing a special attribute store for repository metadata. The final concept and implementation was submited some weeks ago and it is really fast.

During January, Michael Andres got rid of thousands of lines of code in libzypp, keeping the nice APIs but trying to build everything on top of the sat library. I ported some classes like RepoManager, and later this week the final pieces were connected when Michael connected our Pool class with the sat pool which allows to “see” the packages “world” to applications. I ported zypper and YaST package bindings and Michael Matz told us yesterday he actually installed packages using zypper.

The whole stack is now designed as unix tools, so this will allow our repositories to ship solv files, which can in turn be downloaded by libzypp. So you don’t need to generate a cache locally if the solv file is already available on the server. This is not implemented yet.

There also other open issues. We don’t parse translations and only the suse media parser stores attributes like descriptions. Patches and packages are not installed (but this has also another reason I will talk about it later). The package management user interface won’t compile (I haven’t tried) yet.

This new level of performance would allow us to bring package management on openSUSE not to the smart or yum level, but a complete new generation ahead. And this will open the door for smooth integration with CIM and PackageKit and will bring fresh air during the road to the next generation of SLES and SLED.

If you are an openSUSE contributor, we do need help with testing, and there are lot of “mechanical” jobs to do which don’t require much experience with the platform (but compiling). If you want to see this in 11.0 and can contribute, please [contact us][6] .

[1]: http://www.youtube.com/watch?v=XB3o4Skka5Q
[2]: http://en.opensuse.org/Libzypp/Sat_Solver
[3]: http://en.wikipedia.org/wiki/Boolean_satisfiability_problem
[4]: http://www.kdedevelopers.org/blog/124
[5]: http://kkaempf.blogspot.com
[6]: http://lists.opensuse.org/zypp-devel/

About these ads

27 thoughts on “fast package management

  1. Hello!That looks really great! I’ll test it as soon as alpha 2 is released… don’t want to download alpha 1 in a vm and then update to alpha 2 when it’s released the next day ^^Hope to see these improvements in 11.0.. It’s going to rock

  2. This is not yet in alpha2, expect a very broken alpha3 ;-)

  3. too bad :-( anyway… is it in factory already? (don’t know but the package management in alhpa 2 seems to run faster too, compared to 10.3 (and I’m running the alpha in a vm on a single core system!!))

  4. No, the integration of the pieces is happening on the build service (zypp:svn)

  5. This sounds (and looks) really sweet! Go, suse, go!

  6. [...] Ubuntu Hardy Heron to be the hotness Found this link on the latest openSUSE weekly news: Duncan Mac-Vicar P. Blog Archive fast package management If this package management is really this fast on ver 11 and they offer the latest KDE 3.x as an [...]

  7. Will this run in 10.3?

  8. Great work!!this speed issue has been a paine and I’m very happy that you have improved it.Is this in alfa2 or will it be a part of alfa3?Congratulations for a work well done.

  9. In the past (open)SUSE’s slowness of its packet manager has always been a showstopper for me ;(it looks like this has changed to a big degree !time seems ready for a testride of openSUSE 11 :)please also improve mirror management for the repositories (if there’s anything to improve, I haven’t used / tested openSUSE for a long time)thanks,you guys rock ! :)

  10. [...] Duncan Mac-Vicar P. homepage « fast package management [...]

  11. [...] permalink I also was a long time SUSE/KDE user but I got hooked on Ubuntu/GNOME’s clean and simple approach. I also think the RPM based packaging/updates are slower than debs, at least on openSUSE its slow. Now I am on Ubuntu both in office and at home Although there was a blog about upcoming openSUSE 11 package solver being super fast now, so I might actually switch back because I prefer KDE 3.x (dont like KDE4)… See Duncan Mac-Vicar P. Blog Archive fast package management [...]

  12. [...] highlight of course is the reworked ZYpp framework, which gives you a much faster package management experience. This release also contains a new and [...]

  13. Wonderful! I love it! Congratulations!

  14. [...] alla precedente alpha, sono stati introdotti: ZYpp framework, il kernel Linux 2.6.25-rc5, GCC 4.3, GNOME 2.22, KDE 4.0.2 e 3.5.9, Compiz 0.7.2, CUPS 1.3.6, [...]

  15. [...] permalink yes darkstar, i agree the opensuse menu choices are vast and confusing. one of the reasons i stayed with ubuntu is gnome’s simplicity. i still like/prefer kde’s look-n-feel (dont like KDE4) and all the apps like amarok/ktorrent/k3b etc, so i will give opensuse 11 a try if it comes with kde3 or has a theme for kde4 to make it look like classic kde3 another reason why i dropped opensuse 10.3 is because during install if you check the "check for update/repositories" boxes (something like that) then it takes HUGE amount of time to download package lists on my 256k connection. also the package management/solver etc are much slower than apt/deb of ubuntu. but both fedora and opensuse are claiming they have improved packaging speeds so i will also give opensuse 11 a try again See Duncan Mac-Vicar P. Blog Archive fast package management [...]

  16. It is sooooooooooooo fast. Very nice done. It’s….just amazing. I am using an 8 years old computer and before Alpha3 I really have to think about what packages I wanted to install before clicking on “Finish’ cause starting yast/zypper took so long. Now I just do it while I am working cause it’s done in 20 second anyway (10 repo’s).SUPERRR!ps. I have seen the new artwork (proposals) on the designwiki-page and the new installer-theme and it blasted me away. A new graphical tool by beineri(?) to install some usefull software and KDE4 runs very nice allready. This release is a HUGE step forward for openSUSE!!!

  17. [...] highlight of course is the reworked ZYpp framework, which gives you a much faster package management experience. This release also contains a new and [...]

  18. [...] sido retrabajado el ZYpp framework, el cual te da un manejo mas eficiente de los paquetes. Esta Release tambien contiene un nuevo y [...]

  19. [...] happy with the super speed and making other similar tools eat dust, our hackers continue to check in cool [...]

  20. will the new zypper be available for opensuse 10.3 as well?

  21. [...] talked about package management speed, we talked about new looks and features already. However our work around patches and patterns was [...]

  22. I respect the hard work that has gone into improving OpenSuSE’s package management speeds. However, I wonder how much of this is reinventing the wheel? The dpkg / apt-get based package management system is quite fast, well supported, and resolves dependencies with a high rate of success. If the OpenSuSE team is at the point of rewriting zypper and removing “thousands of lines of code”, would this have not been a great opportunity to make the switch away from RPM? Perhaps I do not completely understand zypper and am comparing apples and oranges?Red Hat (http://www.press.redhat.com/2008/04/16/whats-going-on-with-red-hat-desktop-sy… suggest that it will not be continuing a consumer based desktop, so that leaves OpenSuSE (and categorically Novell SLED / SLES) as the last big supporter of RPM. Seems like a great time to consolidate packaging as Ian Murdock suggests…

  23. I switched from OpenSUSE to Ubuntu because of the horrendous speed (lack of) primarily in the package management. The impressive speedups in libzypp may well lure me back, esp. if Ubuntu bluetooth support keeps lagging. Thanks for the update, cheers!

  24. hey dudenice videobut rather then deleting or backspacing u, do u know u can type ctrl u? to delete backwards from the cursor to the prompt and ctrl w to delete back word by word?enjoy linux!

  25. I’ve been running 11 beta 2, and I have to say I am impressed with the huge increase in speed

  26. [...] been reporting in real time cool improvements like the fast installation, how YaST became sexy, how YaST/ZYpp/zypper became fast, how YaST/ZYpp/zypper performs better than others and even that our solver is also really [...]

  27. [...] Duncan Mac Vicar membuat beberapa posting blog mengenai kualitas instalasi berbasis pattern disini, sana, sono dan situ [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.