Duncan Mac-Vicar P.

You may know, that rpm supports (is that the right name?) “named” dependencies. That is, you can require arbitrary strings other packages provide, no matter if that arbitrary string is a package or not.

Take curl as an example, lets see what curl provides:

 # rpm -q --provides libcurl4
 libcurl.so.4()(64bit)
 libcurl4 = 7.16.4-16.2

You see the package provides the libcurl.so.4()(64bit) symbol. Now lets explore the requires:

# rpm -q --requires libcurl4
curl-ca-bundle
/sbin/ldconfig
/sbin/ldconfig
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1
libc.so.6()(64bit)
libc.so.6(GLIBC_2.2.5)(64bit)
libc.so.6(GLIBC_2.3)(64bit)
libc.so.6(GLIBC_2.3.4)(64bit)
libc.so.6(GLIBC_2.4)(64bit)
libcrypto.so.0.9.8()(64bit)
libdl.so.2()(64bit)
libdl.so.2(GLIBC_2.2.5)(64bit)
libidn.so.11()(64bit)
libssl.so.0.9.8()(64bit)
libz.so.1()(64bit)
rpmlib(PayloadIsBzip2) <= 3.0.5-1

curl depends on zlib. But it is not needed to have the package zlib installed. Any package “providing” the symbol “libz.so.1()(64bit)” will fulfill the requirement. If you have a megapackage-bundle rpm which includes lot of libraries, it could also fulfill curl’s requirement.

Advantages:

• They are generated automatically from elf binaries, perl modules, etc. AFAIK Debian depend only on packages. If the depending package stops providing a library, you will not know until you get an error.

• You don’t need to maintain the requirements manually and the spec file only lists what you need to build the package.

• The requirements are more granular. If you depend on a big package, you are not really needed all the stuff. Therefore there is no need to split packages in thousands of smaller pieces.

• Don’t couple the libraries with package names. AFAIK Debian does not support this. And a virtual package called “zengine” or whatever would need to be created so packages need to depend on it.

Disadvantages:

• More noise. Metadata gets bigger.

• More processing needed by the solver, as there are much more provides and requires.

This technique is not only used with native libraries, but with other languages. Look at mono:

# rpm -q --provides mono-web
mono-web-forms
mono-web-services
mono-remoting
mono(Mono.Http) = 1.0.5000.0
mono(Mono.Http) = 2.0.0.0
mono(System.Runtime.Remoting) = 1.0.5000.0
mono(System.Runtime.Remoting) = 2.0.0.0
mono(System.Runtime.Serialization.Formatters.Soap) = 1.0.5000.0
mono(System.Runtime.Serialization.Formatters.Soap) = 2.0.0.0
mono(System.Web) = 1.0.5000.0
mono(System.Web) = 2.0.0.0
mono(System.Web.Services) = 1.0.5000.0
mono(System.Web.Services) = 2.0.0.0

Did I mention that they are also used in kernel modules to match driver compatibility? The kernel package provides certain interface names, with a version which is actually some kind of hash of the signature of those interfaces. So a driver will depend on those.

# rpm -q --provides kernel-default
kernel-default-nongpl
kernel = 2.6.22.9-0.4
k_deflt
k_numa
k_smp
smp
kernel-smp
kernel(drivers_ata) = 8c8c26cd48be2c29
kernel(drivers_char_tpm) = c2f46bb4192faaf6
kernel(fs_nfsd) = 5302e5e83fcef713
kernel(drivers_media_dvb_frontends) = 6c12beb7312724c0
kernel(drivers_video_matrox) = 9d4717fb264df90d
kernel(drivers_block_paride) = c3185e6447e90578
...

Now, lets get back to Java. I tried to package jruby, microemu and other small things. The amount of jars you need to build, which are not yet available in the distribution is amazing. You give up after three iterations. And the binary tarball with the jars is there as a temptation.

However, binary jars sometimes include everything you need to run the program. Including many java packages (namespaces) from external projects. For example, microemu includes nanoxml and asm.

However, the information that current java packages carry is minimal:

# rpm -q --provides ant
apache-ant
ant = 1.7.0-30

Even source packages, usually include the full source, but all binary jar’s required to build the source in a lib directory. We don’t want to include these jars in the rpm package (our goal is to include only what we compiled). But if these jars are not available in the distribution, we also don’t know the name of the package providing those jars.

However, we could build the source using those jars, and then make the rpm require the java namespaces the jar requires by examining the resulting jar. And therefore also the spec file could remove the namespaces included in the resulting jar which don’t belong to that package, and in that way we don’t provide the rpm package version’s version of those namespaces.

So, I discussed this with Pascal. He hacked a bash line to find the provides quicker than me

for jar in *.jar; do 
  jar tf "$jar"|grep '\.class$'|sed 's|/[^/]*\.class$||'|sort -u|while read package; do 
    echo "Provides: java(${package//\//.}) = %{version}-%{release}"
  ; done
; done

Which for ant, results in:

Provides: java(org.apache.tools.ant) = %{version}-%{release}
Provides: java(org.apache.tools.ant.dispatch) = %{version}-%{release}
Provides: java(org.apache.tools.ant.filters) = %{version}-%{release}
Provides: java(org.apache.tools.ant.filters.util) = %{version}-%{release}
Provides: java(org.apache.tools.ant.helper) = %{version}-%{release}
Provides: java(org.apache.tools.ant.input) = %{version}-%{release}
Provides: java(org.apache.tools.ant.listener) = %{version}-%{release}
... much more

However, we still need the requires information. We discussed about using jarjar. But Pascal hacked a better way using jdepend. The output looks like:

Provides: java(org.springframework.dao)
Requires: java(org.springframework.core)
Provides: java(org.springframework.dao.support)
Requires: java(org.apache.commons.logging)
Requires: java(org.springframework.beans.factory)
Requires: java(org.springframework.dao)
Requires: java(org.springframework.util)
Provides: java(org.springframework.transaction)
Requires: java(org.springframework.core)
Provides: java(org.springframework.transaction.annotation)
Requires: java(org.springframework.transaction.interceptor)
Provides: java(org.springframework.transaction.interceptor)
Requires: java(org.aopalliance.aop)
Requires: java(org.aopalliance.intercept)
Requires: java(org.apache.commons.logging)
Requires: java(org.springframework.aop)
Requires: java(org.springframework.aop.framework)
Requires: java(org.springframework.aop.framework.adapter)
Requires: java(org.springframework.aop.support)
Requires: java(org.springframework.aop.target)
Requires: java(org.springframework.beans.factory)
Requires: java(org.springframework.beans.propertyeditors)
...

The missing piece would be stripping unwanted namespaces from the final compiled jar. This can be done using either jarjar or manually by unpacking the jar, deleting some directories and repacking it.

This would allow for much easier building of java packages from source without needing the full dependency tree prepackaged. And build those later one by one, and in a much more granular way. It would also help when building directly from binary jars, resulting in much more granular rpms.

What do you think? What other things can be improved when packaging java software?

Some random thoughts about the world, and the past and following years.

Digital society

2007 was quite active on the digital rights topic.

The inability of the industry to catch up with the current society has created a war on digital rights. Software patents on one side. Digital music transformed music labels into mafias and consumers into rival groups. Politicians trying to implement surveillance systems everywhere.

I have the feeling that we will see some progress on the music topic. Record labels will give up, but it will be too late, and if a bunch of major artists start to use some fair system. Something like amiestreet.com or direct selling comes to my mind.

I don’t think something will happen on patents.

It was funny, some weeks ago I got an idea about using gps to associate location to todo items. This plays well with the getting things done methodology where you organize by contexts and not categories. I started prototyping some stuff on android.

Sadly, I found out this simple idea was patented by Fujitsu. Not only that. But I found a program which does that, and the website dissapeared. Another news article about someone researching on the topic and developing a product on that also dissapeared from the map. However, I haven’t yet seen a product from Fujitsu on the topic (the patent is 7 years old). Software patents destroy innovation. Thanks to that stupid patent, you won’t see any product (unless free software) using that.

The web 2.0

Everybody is sick of the Web 2.0 buzz. The Web 2.0 exists.

It is normal that consultants/analysts start to invent new terms because their business depends on the next “big thing” that will “cut costs” and “save millions” to your company. They repeat the same year after year just replacing the term itself.

However the amount of services on the web is growing really fast, and they are all accessible by really standard protocols. Software is becoming just a support medium and the value is being transfered to services: information, storage, security, etc.

Now, there are new layers over that. Phones with gps will bring a new dimension of services based on our location. This is very important. The information we store on the web becomes more relevant if we map it over real-world dimensions: location, time, mood, energy, context. Open source fits here, you can see companies like Google taking advantage of it.

Question: How services will affect open source and/or free software itself? Google contributes quite a lot to open source software. But once you don’t distribute the software, you are not forced to publish modifications. Will other companies follow this path?

Amazon Web Services is another topic. The way they sell on demand “computing power”, “human processing”, “databases” and “storage” is simply amazing.

I would like to see more about “distributed” environments. I am disappointed on how I have to manage my information having 3 computers and one cell phone. There has to be something better than either being off-line and centralized or being online and ubiquitous (where network is available). I want to be ubiquitous, distributed, fail tolerant, and in a simple, pragmatic way. (I don’t want to setup a cluster on my devices).

What about the bubble?. Yes, there is a bubble. There are a bunch of companies that know what they are doing. And thousands of venture capital groups funding whoolalalhzuzu.com ajax websites which implement a calculator or whatever. Those dying is not a bubble, it is natural selection. Most people already know which ones will die after using them for 2 minutes.

I am really excited about the developments in this area and looking forward what is coming here. The direction is clear.

Software

• openSUSE / YaST

  I will leave this for a separate post.

• KDE 4.0

  4.0 is being released in a few days and you will see the most brave release of free software ever. A big bunch of new technologies and visions collected, cooked and packed inside a great community. And better, there is still no big place for politics in KDE, but technical arguments and user experience. Not that all desktops could say the same.

Software Development

Wow, what happened on 2007?

• Software configuration / Version Control

  The growing complexity of open source codebases, plus the need to maintain them for enterprise purposes, brought the topic of version control really hard on the blog sphere. Every blog and developer talked about git. Lot of talk about mercurial and bzr too. 3 version control systems being popular at the same time? The point is that being “distributed” is “the thing”. I personally switched to git, and it solved the “being distributed” part of working with 3 computers in different places. I want to see something like code.google.com with git support.

• Android

  Brilliant. I am waiting for the first phone. Some APIs are ugly. But still prettier than uggly guys that reinvent the wheel poorly, and worse, only on Windows.

Nothing that spectacular on other old topics:

• Java

  While Eclipse is a jewel. Sun is getting better but too slow to move. So slow that it is getting boring to watch.

• Ruby

  We saw the release of ruby 1.9 on December, a very important milestone. At the same time, JRuby is now fast and very compatible, and other implementations are also very active.

• C++

  Even more boring than Java eh?

Politics

• Chile

  The goverment of Michelle Bachelet whose goverment improvisation has made the country again miss the opportunity to develop quickly. Michelle has no strategy at all, so the hope for 2008 is that his sucking team don’t make more mistakes. The public transport system ( Transantiago ) has to start working somehow (both in operation and budget), because till now, it is a joke.

  The opposition hasn’t a good alternative. Nobody is willing to make the important change: universal free and good education, health and social care. Even Michelle, being a socialist, uses the private health and education system.

• Europa

  Spain’s election coming. Seems that Zapatero will be reelected, which seems reasonable. I am a little lost with german politics and I feel like living in a fantasy world. Time to change that. Still, Europe’s economy is going good and living here is awesome. I love it.

• USA

  Discussions on whether they should teach non-science on science class?. War. etc. Uhm… was I writing about a middle west country? I am sad, really sad to see a beautiful country being destroyed, destroying, hating and being hated by almost the entire world, and even worse, considered the biggest threat to the rest of the world.

  From latimes.com:

  36% of European poll respondents — who come from Italy, France, Germany, Britain, and Spain consider America as the No. 1 danger to world peace. Even 35% of American 16- to 24-year-olds identify their own country as the chief danger to peace. The poll was consistent with findings by the Pew Global Attitudes Project, which found that favorable ratings of the U.S. had declined in 26 of 33 countries over the last five years. Europeans next concerns are China, 19%; Iran 17%; Iraq 11%; North Korea 9%; Russia 5%.

  Elections aren’t this year. Lets see how it goes.

I have spent quite some time studying and testing Android, the new mobile platform from Google.

It is hard to understand people complaining about Java fragmentation because Google’s technical decisions like not using a standard virtual machine.

Just at the same time, I was trying to create an application for Blackberry. Oh yes, Blackberry uses the standard J2ME plus some proprietary APIs. But they of course did not think about Eclipse and instead offer a crappy self-made Windows IDE that nobody likes. There are hacks to use the blackberry APIs (a .jar file) inside Eclipse or Netbeans, but things are not easy if Research in Motion (Blackberry makers) offers the files inside a Windows .exe setup. I did not even try to install the Sun stuff. It was not clear what I needed to download. Which VM? which Eclipse plugin? Which emulator? Where are the docs.

Forget about that. Enter Android.

After downloading the files and extracting them to a directory, I installed the Eclipse plugin adding an remote update site to the configuration. Then I configured the plugin to look for the Android SDK in the directory where I extracted it.

Working with Eclipse is a pleasure. I always become depressed when working with Eclipse because it reminds me how crappy is the life of a C++ programmer when it comes to decent tools. Real time compilation and completion would make me happy in the C++ world.

The documentation is good. Not at the level of Trolltech’s Qt docs but enough. There are some examples in the site, and the amount of Android related projects on code.google.com hosting is impressive. This makes easy learning by looking different examples.

I like the design of the framework, based on concepts such as Activity, Intent, Content (modeled after URIs), etc. The user interface API looks like a normal toolkit, but the XML way of creating layouts is not intuitive. One simple google search and I found Droiddraw, a tool to draw screens and generate XML files from them.

The android sqlite API sucks. I figured out how to wrap a database to expose a content provider url based API, by looking other examples, and at the end I was copy pasting code. Where is DRY ?. However the content provider API is really nice and a clever way to expose data across applications.

Other APIs are really good. Location services, Google apps views, and the way you can use XML files as resources (layouts, strings, etc) is also interesting. Android generates a Java class called R, which you use to access the resources, this allows you to autocomplete the resources in Eclipse.

The emulator rocks, I press “run” in Eclipse and it works, and debugging support is also complete. The emulator is based on qemu. It support different skins and can simulate network speeds and latencies.

In conclusion, Android provides a easy to use framework that work out of the box on your Linux box. It integrates with Eclipse, and the amount of existing code for the platform is enough to start learning.

I will post again as soon as I discover more

Nice surprise today:

• ITO time spent on YaST. No success yet with Wt. It does not provide something like Qt’s QSocketNotifier, or glib’s gioadd_watch which integrate themselves with the event loop. Did a hack with a standard select and a timeout, did not work. Even worse, Wt crashes on processEvents().

• Will hosted a hack session on Saturday. While I did not get any code done, I got motivated by the Kopete 4.x state to continue working on it at home. Yesterday I commited my chat window participants view code for Kopete. It simplifies the code and the signal battle a lot. I still have to fix some issues.

• The Android stuff is so cool. Read this post about Dalvik: how Google routed around Sun’s IP-based licensing restrictions on Java ME. The Activity, Services and Intents model seems natural for other scenarios, not only mobiles.