Duncan Mac-Vicar P.

Yesterday I sat together with Stefan Haas in front of the whiteboard and analyzed the specification of the DSP1023 (Software Inventory) and DSP1025 (Software Updates) CIM profiles, from the DMTF.

Our goal was to understand it, and therefore we tried to map the concept to the knowledge we already have, which is Linux software management (which in turn can be reduced almost to packages :-). Here are our notes:

(For simplicity, I am skipping the CIM_ prefix for all classes)

Packages

Packages are represented as usual NVRs (name, version, release) using the SoftwareIdentity class.

The url of the package is represented associating the SoftwareIdentity with url instances (SoftwareIdentityResources) trough a SAPAvailableForElement association. Products, subpackages and components can be modeled by using OrderedComponent class, which references the component and one member per instance.

One can associate hardware (or any managed element) with software using ElementSoftwareIdentity

Repositories

Repos are modeled by using SystemSpecificCollection and associated via HostedCollection to a system (that seems to be the local computer).

Packages are associated to the repository via the MemberOfCollection class.

Installed packages are represented by instances of InstalledSoftwareIdentity (which references the ComputerSystem where it is installed).

Groups, Patterns

Just another software identity (for the group itself), and references the grouped identities using OrderedComponent instances (which has two fields, the GroupComponent and PartComponent, which reference the respective instances).

Dependencies

If the target software does not have and instance, a SoftwareIdentity is created and isEntity set to false (kind of named dependency) For dependencies that exists software collection, then OrderedDependency is instantiated for each dependency and Antecedent and Dependent is filled)

For installing the packages, a SoftwareInstallationService is needed. Packages can be installed calling installFromSoftwareIdentity() (name, version release instance) or installFromByteStream() (like an rpm package).

To define an installation service, the SoftwareInstallationService class need to specify what it supports by instantiating SoftwareInstallationServiceCapabilities (which has properties like the supported URI schemes), and then this capabilities are associated back to the service using ElementCapabilities.

To define whether a package is compatible with a given target, one uses the TargetTypes array in the SoftwareIdentity class.

If a SoftwareIdentity is available and there is some SoftwareInstallationService that is compatible (or capable) of installing it, a instance of the ServiceAffectsElement needs to be instantiated.

Before actually installing or updating, the client checks if a SoftwareIdentity can be installed on a element, using the CheckSoftwareIdentity() method on the SoftwareInstallationService. You give this method the software itself, the collection and the target element. You get back a InstallCharacteristics with the details (for example if you need to reboot after installing or not).

Once you start the operation using InstallFromSoftwareIdentity(), you get back a Job instance that represent the task. Also true for InstallFromByteStream(), but here you pass an Image instance instead of a SoftwareIdentity. There is a similar InstallFromURI operation too. (jobs are only returned if the service has async capabilities).

The client can pass options to the install operation using InstallOptionValues.

TODO: figure out ElementSoftwareStatus more.

Additionally, you can see how HP has implemented the inventory profile with rpm. It has some useful information about which rpm tags belong to which classes properties for example.

Today I used some of the coolest openSUSE Build Service features: project layering, patches against linked packages and aggregates. I want to write about them.

I needed to test a feature in PackageKit. I am using openSUSE 11.0. However upstream PackageKit does not play nice with 11.0. Stefan Haas fixed this, taking our PackageKit sources, adding some patches, and building them in his home project, which results in a repository here.

My feature involved a patch against PackageKit and then testing it from a client application (to adapt it), so what I wanted to achieve was to use Stefan’s PackageKit plus my patch, packaged in a rpm.

I could just copy Stefan’s sources and patches to my home project, and add my patch, but that would be duplication. Any change Stefan does later would require to copy it again.

Luckily the build service has a feature call linking, so I can link Stefan’s package to my home project. You can do that from osc or from the web user interface (Link package from another project), and that would result in a new package called PackageKit in my project (I created a subproject home:dmacvicar:packagekit for this purpose) with a single file called _link.

This version of PackageKit only builds with Factory’s libzypp, so here we have various options:

• build on top of zypp:svn/openSUSE11.0 which is ZYpp svn built on openSUSE11.0
• link all zypp:svn packages to my project (too inefficient, Adrian would kill me if I start to rebuild ZYpp in every project )
• aggregate all ZYpp openSUSE_11.0 packages from zypp:svn in my project (aggregate explained later)

Because I already have the repo zypp:svn in my repo list, I chose building on top of zypp:svn. (This will mean that in order to use this modified PackageKit repo, you need to add zypp:svn as well )

So now, I can add my patch next to the _link file. However, How to make this patch appear in Stefan’s specfile? Do we need to patch the spec file too? No. The build service does it for you. Just edit the _link file which looks like this:

<link project='home:haass' package='PackageKit'>
    <patches/>
</link>

So it looks like:

<link project='home:haass' package='PackageKit'>

    <patches>
        <add name="pk-outdated-notification.patch" />
    </patches>
</link>

This will insert the patch in the spec file when building. That is real magic.

Sadly, this bleeding edge PackageKit version requires some libtar-devel not available in 11.0. Stefan had a backport to 11.0 in his home project too. Do we need to link the package to our project too? No. Linking here is unnecessary because we don’t really need to rebuild a copy of this package, we only need to make it available in our repository for people installing PackageKit from our project, who don’t have Stefan’s project added in their repository list, that will get libtar grabbed via dependencies, but we don’t care if the metadata grabs the final rpm from Stefan’s project.

That is called “aggregate”. To aggregate a package from another project, we create a new package (called libtar) and add to it a file called “_aggregate” (the web interface does not help you here). Edit that file so it looks like:

<aggregatelist>  
    <aggregate project="home:haass">  
        <package>libtar</package>  
        <package>libtar-devel</package>  
    </aggregate>  
</aggregatelist> 

This will map home:haass repositories 1:1 to our repositories. So if we build our project against openSUSE11.0, it will aggregate our package with home:haass/openSUSE11.0 too. If you don’t have a 1:1 mapping, look the Build Service Tips and Tricks to do more complex aggregates.

Then I am ready to install my new PackageKit:

sudo zypper in -r home_dmacvicar_packagekit PackageKit PackageKit-devel
Reading installed packages...

The following NEW packages are going to be installed:
  PackageKit-devel PackageKit


Overall download size: 1.1 M. After the operation, additional 5.1 M will be used.
Continue? [YES/no]: y

You may know, that rpm supports (is that the right name?) “named” dependencies. That is, you can require arbitrary strings other packages provide, no matter if that arbitrary string is a package or not.

Take curl as an example, lets see what curl provides:

 # rpm -q --provides libcurl4
 libcurl.so.4()(64bit)
 libcurl4 = 7.16.4-16.2

You see the package provides the libcurl.so.4()(64bit) symbol. Now lets explore the requires:

# rpm -q --requires libcurl4
curl-ca-bundle
/sbin/ldconfig
/sbin/ldconfig
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1
libc.so.6()(64bit)
libc.so.6(GLIBC_2.2.5)(64bit)
libc.so.6(GLIBC_2.3)(64bit)
libc.so.6(GLIBC_2.3.4)(64bit)
libc.so.6(GLIBC_2.4)(64bit)
libcrypto.so.0.9.8()(64bit)
libdl.so.2()(64bit)
libdl.so.2(GLIBC_2.2.5)(64bit)
libidn.so.11()(64bit)
libssl.so.0.9.8()(64bit)
libz.so.1()(64bit)
rpmlib(PayloadIsBzip2) <= 3.0.5-1

curl depends on zlib. But it is not needed to have the package zlib installed. Any package “providing” the symbol “libz.so.1()(64bit)” will fulfill the requirement. If you have a megapackage-bundle rpm which includes lot of libraries, it could also fulfill curl’s requirement.

Advantages:

• They are generated automatically from elf binaries, perl modules, etc. AFAIK Debian depend only on packages. If the depending package stops providing a library, you will not know until you get an error.

• You don’t need to maintain the requirements manually and the spec file only lists what you need to build the package.

• The requirements are more granular. If you depend on a big package, you are not really needed all the stuff. Therefore there is no need to split packages in thousands of smaller pieces.

• Don’t couple the libraries with package names. AFAIK Debian does not support this. And a virtual package called “zengine” or whatever would need to be created so packages need to depend on it.

Disadvantages:

• More noise. Metadata gets bigger.

• More processing needed by the solver, as there are much more provides and requires.

This technique is not only used with native libraries, but with other languages. Look at mono:

# rpm -q --provides mono-web
mono-web-forms
mono-web-services
mono-remoting
mono(Mono.Http) = 1.0.5000.0
mono(Mono.Http) = 2.0.0.0
mono(System.Runtime.Remoting) = 1.0.5000.0
mono(System.Runtime.Remoting) = 2.0.0.0
mono(System.Runtime.Serialization.Formatters.Soap) = 1.0.5000.0
mono(System.Runtime.Serialization.Formatters.Soap) = 2.0.0.0
mono(System.Web) = 1.0.5000.0
mono(System.Web) = 2.0.0.0
mono(System.Web.Services) = 1.0.5000.0
mono(System.Web.Services) = 2.0.0.0

Did I mention that they are also used in kernel modules to match driver compatibility? The kernel package provides certain interface names, with a version which is actually some kind of hash of the signature of those interfaces. So a driver will depend on those.

# rpm -q --provides kernel-default
kernel-default-nongpl
kernel = 2.6.22.9-0.4
k_deflt
k_numa
k_smp
smp
kernel-smp
kernel(drivers_ata) = 8c8c26cd48be2c29
kernel(drivers_char_tpm) = c2f46bb4192faaf6
kernel(fs_nfsd) = 5302e5e83fcef713
kernel(drivers_media_dvb_frontends) = 6c12beb7312724c0
kernel(drivers_video_matrox) = 9d4717fb264df90d
kernel(drivers_block_paride) = c3185e6447e90578
...

Now, lets get back to Java. I tried to package jruby, microemu and other small things. The amount of jars you need to build, which are not yet available in the distribution is amazing. You give up after three iterations. And the binary tarball with the jars is there as a temptation.

However, binary jars sometimes include everything you need to run the program. Including many java packages (namespaces) from external projects. For example, microemu includes nanoxml and asm.

However, the information that current java packages carry is minimal:

# rpm -q --provides ant
apache-ant
ant = 1.7.0-30

Even source packages, usually include the full source, but all binary jar’s required to build the source in a lib directory. We don’t want to include these jars in the rpm package (our goal is to include only what we compiled). But if these jars are not available in the distribution, we also don’t know the name of the package providing those jars.

However, we could build the source using those jars, and then make the rpm require the java namespaces the jar requires by examining the resulting jar. And therefore also the spec file could remove the namespaces included in the resulting jar which don’t belong to that package, and in that way we don’t provide the rpm package version’s version of those namespaces.

So, I discussed this with Pascal. He hacked a bash line to find the provides quicker than me

for jar in *.jar; do 
  jar tf "$jar"|grep '\.class$'|sed 's|/[^/]*\.class$||'|sort -u|while read package; do 
    echo "Provides: java(${package//\//.}) = %{version}-%{release}"
  ; done
; done

Which for ant, results in:

Provides: java(org.apache.tools.ant) = %{version}-%{release}
Provides: java(org.apache.tools.ant.dispatch) = %{version}-%{release}
Provides: java(org.apache.tools.ant.filters) = %{version}-%{release}
Provides: java(org.apache.tools.ant.filters.util) = %{version}-%{release}
Provides: java(org.apache.tools.ant.helper) = %{version}-%{release}
Provides: java(org.apache.tools.ant.input) = %{version}-%{release}
Provides: java(org.apache.tools.ant.listener) = %{version}-%{release}
... much more

However, we still need the requires information. We discussed about using jarjar. But Pascal hacked a better way using jdepend. The output looks like:

Provides: java(org.springframework.dao)
Requires: java(org.springframework.core)
Provides: java(org.springframework.dao.support)
Requires: java(org.apache.commons.logging)
Requires: java(org.springframework.beans.factory)
Requires: java(org.springframework.dao)
Requires: java(org.springframework.util)
Provides: java(org.springframework.transaction)
Requires: java(org.springframework.core)
Provides: java(org.springframework.transaction.annotation)
Requires: java(org.springframework.transaction.interceptor)
Provides: java(org.springframework.transaction.interceptor)
Requires: java(org.aopalliance.aop)
Requires: java(org.aopalliance.intercept)
Requires: java(org.apache.commons.logging)
Requires: java(org.springframework.aop)
Requires: java(org.springframework.aop.framework)
Requires: java(org.springframework.aop.framework.adapter)
Requires: java(org.springframework.aop.support)
Requires: java(org.springframework.aop.target)
Requires: java(org.springframework.beans.factory)
Requires: java(org.springframework.beans.propertyeditors)
...

The missing piece would be stripping unwanted namespaces from the final compiled jar. This can be done using either jarjar or manually by unpacking the jar, deleting some directories and repacking it.

This would allow for much easier building of java packages from source without needing the full dependency tree prepackaged. And build those later one by one, and in a much more granular way. It would also help when building directly from binary jars, resulting in much more granular rpms.

What do you think? What other things can be improved when packaging java software?

Some random thoughts about the world, and the past and following years.

Digital society

2007 was quite active on the digital rights topic.

The inability of the industry to catch up with the current society has created a war on digital rights. Software patents on one side. Digital music transformed music labels into mafias and consumers into rival groups. Politicians trying to implement surveillance systems everywhere.

I have the feeling that we will see some progress on the music topic. Record labels will give up, but it will be too late, and if a bunch of major artists start to use some fair system. Something like amiestreet.com or direct selling comes to my mind.

I don’t think something will happen on patents.

It was funny, some weeks ago I got an idea about using gps to associate location to todo items. This plays well with the getting things done methodology where you organize by contexts and not categories. I started prototyping some stuff on android.

Sadly, I found out this simple idea was patented by Fujitsu. Not only that. But I found a program which does that, and the website dissapeared. Another news article about someone researching on the topic and developing a product on that also dissapeared from the map. However, I haven’t yet seen a product from Fujitsu on the topic (the patent is 7 years old). Software patents destroy innovation. Thanks to that stupid patent, you won’t see any product (unless free software) using that.

The web 2.0

Everybody is sick of the Web 2.0 buzz. The Web 2.0 exists.

It is normal that consultants/analysts start to invent new terms because their business depends on the next “big thing” that will “cut costs” and “save millions” to your company. They repeat the same year after year just replacing the term itself.

However the amount of services on the web is growing really fast, and they are all accessible by really standard protocols. Software is becoming just a support medium and the value is being transfered to services: information, storage, security, etc.

Now, there are new layers over that. Phones with gps will bring a new dimension of services based on our location. This is very important. The information we store on the web becomes more relevant if we map it over real-world dimensions: location, time, mood, energy, context. Open source fits here, you can see companies like Google taking advantage of it.

Question: How services will affect open source and/or free software itself? Google contributes quite a lot to open source software. But once you don’t distribute the software, you are not forced to publish modifications. Will other companies follow this path?

Amazon Web Services is another topic. The way they sell on demand “computing power”, “human processing”, “databases” and “storage” is simply amazing.

I would like to see more about “distributed” environments. I am disappointed on how I have to manage my information having 3 computers and one cell phone. There has to be something better than either being off-line and centralized or being online and ubiquitous (where network is available). I want to be ubiquitous, distributed, fail tolerant, and in a simple, pragmatic way. (I don’t want to setup a cluster on my devices).

What about the bubble?. Yes, there is a bubble. There are a bunch of companies that know what they are doing. And thousands of venture capital groups funding whoolalalhzuzu.com ajax websites which implement a calculator or whatever. Those dying is not a bubble, it is natural selection. Most people already know which ones will die after using them for 2 minutes.

I am really excited about the developments in this area and looking forward what is coming here. The direction is clear.

Software

• openSUSE / YaST

  I will leave this for a separate post.

• KDE 4.0

  4.0 is being released in a few days and you will see the most brave release of free software ever. A big bunch of new technologies and visions collected, cooked and packed inside a great community. And better, there is still no big place for politics in KDE, but technical arguments and user experience. Not that all desktops could say the same.

Software Development

Wow, what happened on 2007?

• Software configuration / Version Control

  The growing complexity of open source codebases, plus the need to maintain them for enterprise purposes, brought the topic of version control really hard on the blog sphere. Every blog and developer talked about git. Lot of talk about mercurial and bzr too. 3 version control systems being popular at the same time? The point is that being “distributed” is “the thing”. I personally switched to git, and it solved the “being distributed” part of working with 3 computers in different places. I want to see something like code.google.com with git support.

• Android

  Brilliant. I am waiting for the first phone. Some APIs are ugly. But still prettier than uggly guys that reinvent the wheel poorly, and worse, only on Windows.

Nothing that spectacular on other old topics:

• Java

  While Eclipse is a jewel. Sun is getting better but too slow to move. So slow that it is getting boring to watch.

• Ruby

  We saw the release of ruby 1.9 on December, a very important milestone. At the same time, JRuby is now fast and very compatible, and other implementations are also very active.

• C++

  Even more boring than Java eh?

Politics

• Chile

  The goverment of Michelle Bachelet whose goverment improvisation has made the country again miss the opportunity to develop quickly. Michelle has no strategy at all, so the hope for 2008 is that his sucking team don’t make more mistakes. The public transport system ( Transantiago ) has to start working somehow (both in operation and budget), because till now, it is a joke.

  The opposition hasn’t a good alternative. Nobody is willing to make the important change: universal free and good education, health and social care. Even Michelle, being a socialist, uses the private health and education system.

• Europa

  Spain’s election coming. Seems that Zapatero will be reelected, which seems reasonable. I am a little lost with german politics and I feel like living in a fantasy world. Time to change that. Still, Europe’s economy is going good and living here is awesome. I love it.

• USA

  Discussions on whether they should teach non-science on science class?. War. etc. Uhm… was I writing about a middle west country? I am sad, really sad to see a beautiful country being destroyed, destroying, hating and being hated by almost the entire world, and even worse, considered the biggest threat to the rest of the world.

  From latimes.com:

  36% of European poll respondents — who come from Italy, France, Germany, Britain, and Spain consider America as the No. 1 danger to world peace. Even 35% of American 16- to 24-year-olds identify their own country as the chief danger to peace. The poll was consistent with findings by the Pew Global Attitudes Project, which found that favorable ratings of the U.S. had declined in 26 of 33 countries over the last five years. Europeans next concerns are China, 19%; Iran 17%; Iraq 11%; North Korea 9%; Russia 5%.

  Elections aren’t this year. Lets see how it goes.