With libzypp 6.1.0 in Factory, this is already the case. With the first community round of testing, we got quite a lot of bug reports and feedback. Some stuff did not work because the implementation was wrong, while others just showed us that aria2 worked differently as we thought.

The upcoming 6.2.1 addresses the following issues:

• implemented speed indicators (bnc#475506). You should see transfer speed in applications that report it (like zypper).

• implemented progress indicators correctly (bnc#473846). You should see progress reporting now in a consistent basis.

• fixed broken pipe when looking for aria2c which caused a fallback to curl. (bnc#480930)

• implement saving and reading mirror stats data in /var/cache/zypp/aria2.stats. This means libzypp now learns which mirrors are faster and more reliable for you and remembers it. Every usage adds more information to the “learning”.

• handle failover correctly (bnc#481115). This means libzypp tries harder to get the file from alternate mirrors.

• various improvements in error and report handling. General code review and minor fixes.

How to test?

Peter added an interesting “test feature”. If a request happens with a header X-Broken-Mirrors then the openSUSE redirector will send you broken mirrors. So edit your root’s aria2.conf:

cat ~/.aria2/aria2.conf
header=X-Broken-Mirrors: true

And observe zypper.log. Even if errors are reported, libzypp should try other mirrors and get the file successfully.

You can go more extreme, and use “X-Broken-Mirrors: only”, which will give you only bad servers. Even in this case you will see libzypp try hard to get the file before giving up. How hard to try is configurable, see last post for more information.

Big thanks to Peter Poeml and Tatsuhiro Tsujikawa for all the feedback and support, and of course

And with this, feature 302923 is done and the rest is tracked as bugs.

Since a couple of weeks the ZYpp project repository is now hosted on git.opensuse.org. Please read the official announcement here.

Now you can fork and develop much easily without needing access to the “official repository”. Developers can work disconnected, enjoy the git features to handle merges and branches, oh, and you can keep your forks in git publishing sites like the cool GitHub

We updated the following pages to reflect the move:

• ZYpp Development page
• sat-solver homepage

Those also link to the new pages, written as a starting point:

• General git information and useful links
• git hosting (including git.opensuse.org) information

One of the challenges of the migration was continuous integration. We needed to replicate the automated building/testing process, which is a core part of the ZYpp team development model.

The process has been migrated from cruisecontrol.rb to Hudson ( Extensible continuous integration engine, https://hudson.dev.java.net/) which provides better job handling support, plugins and it is not tied to svn.

Some new features:

• We now build zypper automatically over the rest of the ZYpp stack.
• We publish the last successful build that passes the testsuite automatically to zypp:Head project. This one will obsolete zypp:svn.
• We are working to build other pieces of the stack automatically, like PackageKit.
• We are working to also build and test the whole YaST stack, so it can also be published automatically. So expect a YaST:Head project soon too, to obsolete YaST:SVN (outdated).

I would like to thank R. Tyler for his help with Hudson plugins, Jens Daniel for his original automation scripts we used, which are still the base for the Hudson build scripts , Stano for adding on-request features to y2makeall, and the Hudson developers for such a amazing piece of software.

Since today, there is developer documentation generated on every svn build from our cruise control system (continuous build), available for trunk and branches.

ZYpp documentation has already qute some articles and the API is very good documented, still, it will be enhanced in the following months to take advantages of having the documentation always available. SP2 branch documentation will be available soon.

For SAT solver, we have moved the existing documentation into the Related pages of the generated documentation (which already contains very useful articles) and we will do every possible effort to document the API too (which is almost empty right now :-/ ).

Did not start the day in a good way. I buy mobicards every month so I have full access to the public transport system. I buy them via the internet, so I get an email when it is about to run out, and with one click I get the form to order another one by mail (even with the date range pre filled).

Today I took the straßenbahn (tram), and suddenly the guy next to me transformed itself in 1 second from a civilian to a public transport control employee, and asked me for my ticket. I took my mobicard from my pocket and showed it to him. He nods but after a second he comes back and ask “let me see it again”. He looks again and points out it starts on 12.11 and today was 11.11. No problem, last month one is behind that one (the whole year cards are there). He looks the old one and points out “This one ends at 10.11″. I could not believe it, I did not even realize. Usually I just click OK-OK when buying them. I got a ticket. Luckily the checkbox marked was not “pay 40€” but “Go to the headquarters to clarify” (which still means they could fine me). Of course that means wasting half a morning not counting the time I lost already over the tram (not being able to get out in the right station). If I get fined after buying mobicards every month for years it would be really ironic.

ZYpp

In the office we had the typical “first media” blocker. Luckily Michael found the bug really quickly. Later I wanted to test the fix using linuxrc magic driverupdate tricks I have learned in the past weeks. However it did not work: Linuxrc can use rpms as a container format, but still expects the driverupdate directory structure. Bah, world was not that cool as I thought. Met with Michael and Steffen about it. Steffen realized that it would not be a bad idea to try the rpm first as a driverupdate, and if no driverupdate is there just unpack it in the system (and promised to implement it) That sounds good, easy testing. Happy.

YaST

autoYaST

Katarina wrote a really nice tutorial that explains how use autoYaST to apply a configuration to a running system. Uhm. Looks like an XML API. Wow. Couldn’t we use it for our YaST web service?. Talked with schubi about it. While the command line interface has higher level, exposing this interface would make sense too, because it will offer automatically almost all autoYaST power.

Partitioner

The new YaST partitioner, on which Katarina, Arvin, Matthias and Martin worked really hard during the Code11 cycle got a pretty nice review here. Give it a look. And give the guys feedback. I am sure lots of things can still be improved!.

YaST releases

Stano is reviving the discussion about whether to make independent of openSUSE YaST releases. I talked about this topic with Zonker just after he started as community manager.

I think it would be a pretty good move. Also a big challenge. It would mean a challenge not only for us the YaST teams but also for other internal and external stakeholders.

Right now a new YaST package submission (same with libzypp) is more or less tied with the development of features or some project manager sitting next to your chair to get a blocker fixed for an openSUSE release. Things would change if the distribution’s team would need to take the last released YaST and take patches, forward ports, backports, etc.

Also it would be a challenge for us. Most of YaST testing happens on a openSUSE release. We have lot of ideas on how to improve our automated testing and this would require develop them further. Also I think an approach like this would facilitate making YaST less distro-dependent.

Another point I like of the approach is that it would force us to have a better process to define our core-platform (ir order to be able to release it separately). Right now I see it a bit chaotic: APIs are born in modules, and as soon as they are used by more than one module, they land in the wildcard yast2 package. Something I really don’t find very convenient. APIs should be reviewed with more care and packaged by the area the API covers.

Still and interesting topic and material to innovate!

made possible by Scout.

I’ve been intrigued by http://en.opensuse.org/Standards/OneClickInstall for some time now. (That’s a way to provide a one-click web install experience for .deb/.rpm/.psi etc. packages, implemented as a mime type handler that parses a simple .xml file pointing to the package/repository appropriate for each distro.) When this idea was brought up on the Packagekit mailing list, it generated lots of negative feedback. The summary at http://packagekit.org/pk-faq.html#1-click-install gives a bunch of non-central objections, followed by the central objection that one cannot trust third party repositories: “Allowing to easily add third party repositories and install third party software without a certification infrastructure is like opening the gates to hell”

One Click Install is only a simple description to add a repository and some software, but the security is not a property of this simple description, but of the package manager. A malicious one click install file can point ZYpp to “hell”, but “hell:

• Either is signed with a non trusted key and the user needs to trust it.
• Is signed with the distribution key, which is already in the trusted keyring therefore it just works.

This is a real problem. Here are a couple risks:

No, it is not. It depends on the package manager you are using. Basic security here is independent of one click install.

1) users might click on malware sites and add completely malicious sites to their repository lists

Again, this is not true for ZYpp, and I guess not all package managers allows to download metadata from a repository without trusting it first.

2) a compromised third-party repository might update system packages maliciously.

If you trusted it, there is nothing you can do.

3) several genuinely well-intentioned repositories might include conflicting versions of a commonly needed package not provided by the system repositories.

This has nothing to do with one click install. Packman repository does it already. This is a dependency resolving problem and vendor management: ZYpp for example, does not allow implicit vendor jump on update (only on dist-upgrade). And vendor jump is a conflict you need manually to approve. Only package managers that threat all packages with the same name as the same package suffer here.

After mulling this problem over for a long time, two ideas came to mind: 1) Since the distribution is trusted, it could decide to trust some third-party repositories. For instance, it might decide to trust Adobe’s hypothetical repository so that people could get flash and air updates straight from the source.

This is possible now:

• You can import Adobe’s key into the distribution together with the distribution key, so it goes automatically to the trusted keyring
• You can create metadata in the distribution side, linking to Adobe’s packages, and sign the metadata with the distribution key
• You can do nothing and let the user trust explicitly the repository

This idea of using the distribution as arbiter of trust for third party repositories could be extended to games publishers, etc. This could provide a partial solution to the first threat listed above; if the “good” third-party repositories are already known to the distribution, there’s less need for users to be doing something dangerous like deciding on their own to trust a random third-party repo. This addresses the first threat identified above.

I think repository trusting and the required package manager infrastructure has nothing to do with one click install. It just needs to be there.

2) A simple way to keep repositories from updating packages they shouldn’t is to have package managers enforce some sort of namespacing. e.g. Adobe’s repository could be allowed to only update packages whose names start with “adobe-”. (System repositories would continue to be able to update any package at all.) This addresses the second and third threats identified above.

Why inventing a new thing? You already have the vendor tag there. A package should be considered update candidate only for packages that are from the same vendor.

Imagine you have mediaplayer-1.0 from the distribution, without mp3 support. Then a 3rd party repo offers a compiled 2.0 version with mp3 support but without other features, and then the distribution offers 3.0. You would be getting and losing features all the time as you update. Same package names does not means it is the same package. If your package manager does that, it is a bug. If there is no solution to the dependency solving than jumping vendor, do a conflict and make the user explicitly switch. Once he does, the update candidate would be the same vendor packages.

I think something like this is going to be needed before we can have a thriving — and safe – ecosystem of ISVs providing easily-downloaded-and-installed binary packages for Linux. What do people think about the package namespacing idea? – Dan

I don’t think reinventing the vendor tag and repository signatures is a good idea.

The problem is not at this level, IMHO the only thing we can do is improve the usability of the security chain. For the user is meanless to trust a repository with gpg key 0×32432432 and they will probably just click “yes”. The pending task is to make the cryptographic chain something that makes sense for the user.

The new feature is updateinfo metadata generation (patches) support, in a simple but very automatic way, designed specially for testing purposes, but it may be sufficient for people wanting to generate patches for their own repositories.

Using the –generate-update pkg1,pkg2.. option, enhancerepo will look in your repository (and additionally in another base directory), and look for all packages. If a package has multiple versions available (including the base directory) it will create update metadata in the repoparts directory. If you run enhancerepo with the –updates option (either in the same run or not) it will take all repoparts and index them in the updateinfo.xml. This allow to manually edit the patches before indexing them, or to mix automatically generated ones with hand-crafted ones.

As additional coolness it will look for bugs to generate descriptions/references, update type (for example changes containing CVE references or vulnerabilities are tagged with security automatically).

For example, I have 2 amarok packages in my repo, plus one in the 11.0 base directory. This command will look the 3 rpms, and start looking backwards till it finds some changes in the changelog. You can specify multiple packages per “patch”.

# enhancerepo --generate-update amarok --updates --updates-base-dir /space/repo/11.0 /space/repo/duncan2
generating update...
3 versions for 'amarok'
Found change amarok-1.4.10-17-i586.i586 and amarok-1.4.9.1-53-i586.i586.
'amarok' has 1 entries (68/67)
Saving update part to '/space/repo/duncan2/repoparts/update-amarok-1.xml'.
Adding update /space/repo/duncan2/repoparts/update-amarok-1.xml
Saving /space/repo/duncan2/repodata/updateinfo.xml.gz ..
Adding /space/repo/duncan2/repodata/updateinfo.xml.gz to 
       /space/repo/duncan2/repodata/repomd.xml index
repodata/updateinfo.xml.gz already exists. Replacing.
Saving /space/repo/duncan2/repodata/repomd.xml ..

The resulting updateinfo.xml:

<?xml version="1.0" encoding="UTF-8"?>
<updates>
<update status="stable" from="dmacvicar@piscola" type="security" version="1">
  <title>Untitled updatesecurity update 1 for amarok</title>
  <id>amarok</id>
  <issued>1223293050</issued>
  <release>no release</release>
  <description>
    - update to 1.4.10: fix tmp file vulnerability in the Magnatune
    database parsing. Secunia#SA31418 / CVE-2008-3699 / bnc#417232
  </description>
  <references>
    <reference href="http://bugzilla.novell.com/417232" 
       title="bug number 417232" type="bugzilla" id="417232"/>
    <reference 
       href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-3699" 
       title="CVE number 2008-3699" type="cve" id="2008-3699"/>
  </references>
  <pkglist>
    <collection>
      <package name="amarok" arch="i586" version="1.4.10" release="17">
        <filename>amarok-1.4.10-17.i586.rpm</filename>
      </package>
    </collection>
  </pkglist>
</update>
</updates>

Some important notes:

• If you generate an update, and then you don’t add new packages or changes, enhancerepo will generate the same update again, but with a new version.

The new feature is deltarpm metadata generation support and also some kind of smart deltarpm generation.

This means, enhancerepo can look which package has several versions in the repository, and generate delta rpms for N steps to the older versions. The default is one step, that is, only a delta rpm to the newest to the previous one. And then it can generate the metadata for you too (and add it to the index and such).

Example run:

# enhancerepo --disk-usage --keywords --eulas --create-deltas 2 --deltas -- /space/repo/duncan2
Adding eula: /space/repo/duncan2/zapping-0.9.6-72.eula to zapping-0.9.6-72-i586
Adding eula: /space/repo/duncan2/zaptel.eula to zaptel-1.2.10-70-i586
Adding keyword: /space/repo/duncan2/zaptel-debuginfo.keywords to zaptel-debuginfo-1.2.10-70-i586
Preparing disk usage...
Creating delta - amarok-1.4.10-3-i586 -> amarok-1.4.10-17-i586 (1/2)
Creating delta - amarok-1.4.9.1-53-i586 -> amarok-1.4.10-17-i586 (2/2)
Saving /space/repo/duncan2/repodata/susedata.xml.gz ..
Adding /space/repo/duncan2/repodata/susedata.xml.gz to /space/repo/duncan2/repodata/repomd.xml index
repodata/susedata.xml.gz already exists. Replacing.
Saving /space/repo/duncan2/repodata/deltainfo.xml.gz ..
Adding /space/repo/duncan2/repodata/deltainfo.xml.gz to /space/repo/duncan2/repodata/repomd.xml index
repodata/deltainfo.xml.gz already exists. Replacing.
Saving /space/repo/duncan2/repodata/repomd.xml ..

Some important notes:

• It now requires ruby-rpm, which is available on the devel:languages:ruby:extensions repository.
• Be careful with running createrepo on top of a directory with deltarpms. createrepo will index them incorrectly as packages (So clean deltarpms, run createrepo, and then generate deltas with enhancerepo on top).
• I did not test this release as much as the latest

So Peter mentored Gerard Farràs during the last Summer of Code, and the result was a ZYpp media handler that uses aria2 to download files, instead of curl. (May be aria uses curl for http then, I have no idea ).

aria2 is a utility for downloading files. The supported protocols are HTTP(S), FTP, BitTorrent (DHT, PEX, MSE/PE), and Metalink. It can download a file from multiple sources/protocols and tries to utilize your maximum download bandwidth. It even supports downloading a file from HTTP(S)/FTP and BitTorrent at the same time, while the data downloaded from HTTP(S)/FTP is uploaded to the BitTorrent swarm. Using Metalink’s chunk checksums, aria2 automatically validates chunks of data while downloading a file like BitTorrent.

The benefits is that aria knows about bittorrent, metalinks, etc, so this open the door for much better download failover in the future.

As we are aready on beta phase and we can’t switch such an important component, I merged the code, but disabled. That means ZYpp still uses curl, unless you enable the environment variable ZYPP_ARIA=1. That way you can play with it while it matures for 11.2 or something. There should be lot of features missing in the aria2 handler, so please have that in mind.

Thanks to Peter and Gerard for the handler.