Source-package management

[piratePenguin]

I've had a few ideas (and I'm working on them).
Posted them to my ancient blog @ blogger.com:
http://piratepenguin.blogspot.com/2005/11/source-package-management.html

soooooooooo...... thoughts? Think it'd work well, if it was implemented?

[piratePenguin]

OK, here's a better XML file if anyone's interested in this thing, for irssi.

Code: [Select]

   irssi is a text-based IRC client.
   
      
   
   
      
         Enables use of irssi over an IPv6 network.
      
      
         Use a garbage collector.
         
            
         
      
              advice="recommended">
         Support for perl extensions for irssi.
         
            
         
         
            Build with perl support as a module.
         
         
            Build with perl support statically linked into the irssi binary.
         
         
            Build without perl support.
         
      
   

Just use your imagination for now to find out what exactly the client can do with this information.

Perhaps something like this could work. Here's what you would need to do, I think. Have your distribution system work like php. Then the source code to all these programs gets dropped into the database. When my computer running FC4 with stops by to pick up the latest release of transcode, the package manager looks at my system, and determines what flags are required to create a package custom-suited to my needs. The package manager then gives these requirements to the distro system, which produces a package custom-fit for me. It would also store a compressed copy of the package in the database, just in case someone else with similar requirements comes for the package.

source
Well worker, with this (the thing I was describing back then was nothing like this) system, such a system wouldn't be hard to do.

The hardest bit, by far, would be the configure bit of the XML files. For big packages with many options, that'd be one heck of a bitch.

Oh man. Gaping opportunity for an XUL(or HTML and friends, but that's like, old technology :p. The XUL app could easily be run remotely (what worker described) or installed locally (far easier than with HTML/PHP/whatever), and it'd look like a native application.)-built frontend to this thing.

[worker201]

It looks like what you are doing here is making the package do the work.  Meaning that the configuration files are package dependent.  Then your distroserver has to parse the files and then get down to business.  What you are basically doing is asking developers to do it your way, and not their way.  They don't like that.  If they could agree to a certain way of doing things, they probably would have already.

I think there is a better way.  It should be possible for the distroserver to run a ./configure just like any monkey, and then generate the configuration files by itself.  Once a config file was created, it could be stored, to make installation of that package faster for the next guy.

I think this would maybe make the whole system more efficient.  Because just about every program built with autoconf has --build-target=, --with-shared, --prefix=, and etc.  The system can prepare canned answers to each of these, based on the machine, so you get a custom package.  That leaves extra time and trouble for the optional components.

So, instead of going to the distroserver and asking for an FC3 package, the distro asks your computer what kind of package it wants.  Then it builds a custom rpm (as an example) that is designed for your computer, on the fly, while you wait.  This is time-consuming, but you can always use a little bandwidth to entertain during the process.

The reason to do it this way is that all you have to do are find the source tarballs.  Every open developer has them.  It wouldn't require anything more from anybody else than the current system (or lack thereof) already has.  All the burden is placed on you and your program.  Yeah, it will be harder for you.  But it will be completely unobtrusive to everyone else.  I mean, imagine what a pain in the ass it would be, if that sorta lazy programmer at ffmpeg didn't write an XML config file.  You'd have to write one for him, and since he doesn't release anything but CVS snapshots, that would maybe be once a week.  That's too much work, all because some lazy guy won't do things your way.  It would be so much easier to just grab the CVS tarball, and have the machine do all the work.

All the users would have to do is download the app and/or visit the website - everything else is taken care of.  It's like the way apt repos are done now, except there's no reason to keep extra binaries on hand for all the different possible flavors of Linux.  

And best of all, if there is an important option for the package that I want, I get it.  Example there is dvdauthor.  It requires png15-devel, but only if the --enable-png flag is set at build time.  Dag and Dries, in Fedora aptland, just didn't enable png, eliminating the devel dependency.  But then you can't use pngs in your dvd menus.  I had to download dvdauthor myself and configure it - the apt package was useless to me.

Just a few thoughts and experiences I have had that might help you design your system.

[piratePenguin]

the configuration files are package dependent.

Yea, just like the FreeBSD port's makefiles. Although these XML package files will be far more complicated. But whoever's writing the XML file doesn't have to cover every configure flag. Whatever they think is necessary, just. The frontend could produce the output of './configure --help', and the user could add their own options.

What you are basically doing is asking developers to do it your way, and not their way.

I'm not asking anyone to do anything.

This isn't a bid by piratePenguin to take over the world. It's not 2010 yet.

I think there is a better way.  It should be possible for the distroserver to run a ./configure just like any monkey, and then generate the configuration files by itself.  Once a config file was created, it could be stored, to make installation of that package faster for the next guy.

I dunno what different system you're talking about.

The XML files I posted are used by the package manager to sort out dependencies and configure options. It asks the user do they want to apply some_patch, do they want to use '--enable-png' and install libpng. Then it patches, installs dependencies (libpng), runs configure with the options the user wanted, and then compiles it.

So, instead of going to the distroserver and asking for an FC3 package, the distro asks your computer what kind of package it wants.  Then it builds a custom rpm (as an example) that is designed for your computer, on the fly, while you wait.  This is time-consuming, but you can always use a little bandwidth to entertain during the process.

All that was part of the plan.

imagine what a pain in the ass it would be, if that sorta lazy programmer at ffmpeg didn't write an XML config file.

I don't expect him to. I could probably set it up so old ffmpeg package files (that's the XML file) will work with new ones (although they won't know about e.g. --enable-something-new until the next update of the package file).

And best of all, if there is an important option for the package that I want, I get it.  Example there is dvdauthor.  It requires png15-devel, but only if the --enable-png flag is set at build time.

Code: [Select]

   
      Allow the use of PNGs in your DVD menus.
      
         
      
   

Now any frontend can and should (unless you configured the package manager (not the frontend) to automatically accept/reject recommended configure flags) ask you about '--enable-png' and you can decide if you want it or not. If you say yes, it'll check if you have libpng. If you don't have it, it'll install it. If you have an old version, it'll update it.

Dag and Dries, in Fedora aptland, just didn't enable png, eliminating the devel dependency.  But then you can't use pngs in your dvd menus.  I had to download dvdauthor myself and configure it - the apt package was useless to me.

Well in this source package manager (remember that's what it will be), you won't have to worry about that.

Just a few thoughts and experiences I have had that might help you design your system.

Thanks for your input.
A few of the bits I didn't understand (most of the bits I didn't quote/comment on here) - but I think that might be because you misinterpreted what I'm trying to do...

[worker201]

You're going to write the xml files for each package?  That's a shitload of work.  I thought your plan was to have the devs write the xml files.  But if you want to do the work yourself, then that's cool, I guess.

[piratePenguin]

You're going to write the xml files for each package?  That's a shitload of work.  I thought your plan was to have the devs write the xml files.  But if you want to do the work yourself, then that's cool, I guess.

Yea I was always gonna write the XML files. Only question is how much info (configure stuff mainly) I shoulld put in them...

After I have XML files for the (relatively) few packages required for a usable system (that doesn't mean graphical stuff... But Xorg and some WMs, I wouldn''t mind doing) (and the package manager written, of course) I could make a distro that uses them.

[worker201]

It would probably be easier in the long run to create a script that dynamically creates the xml file.  All the appropriate information could be pulled straight from the configure file, given the right kind of reader.  Then all you have to do is feed a full tarball to the script, and it would build the package for you.

[piratePenguin]

It would probably be easier in the long run to create a script that dynamically creates the xml file.  All the appropriate information could be pulled straight from the configure file, given the right kind of reader.  Then all you have to do is feed a full tarball to the script, and it would build the package for you.

Good idea.

I think I know what way to do this now.
2 programs (actually they'll be libraries in the end, and then there will be a frontend) - one for the configuration, and one for everything else (solving dependencies, package management itself). Whenever the configuration program doesn't have an XML file, it'll notify the user and print the output of './configure --help', and the user can decide what configuration options to use. Or maybe pull the info like you said, and give that to the user.

[piratePenguin]

The idea so far:

The frontend is what the user sees and interacts with. The frontend interacts with two libraries - libdependencies and libconfigure (which I'm not so sure about).

libdependencies is used to sort out dependencies. It will be extensible - there will be extensions for it to support packages, users, and various other stuff. libdependencies asks the extensions "do you have X?" (X being a dependency - the name of a package or the location of a file, or whatever the plugin takes.), and the extension replies as appropriate.

If the extension replies with "no", then libdependencies checks it's configuration to see whether it should tell the extension to go fulfill the dependency or ask the user first. All dependencies that get to libdependencies are required (hence libconfigure (part of it's reason for existance is to act as a frontend to libdependencies), which just might be replaced with libsourcepackagemanagement, or something), so when libdependencies tells the frontend that a dependency isn't met right now, the user will only have a choice between having the dependency solved or cancelling installation (or, if it's a configure option that triggered the dependency, then just cancel the configure option).

All the extensions to libdependencies will actually be libraries (I think that's how extensability works most of the time). The extension that deals with packages, say it's called libdeppackages, will itself be extensible. You will be able to install an extension for RPM, which would check the RPM database for packages, Slackware, which would check /var/log/packages for packages, etc. That way, when some package asks for glib2, and you have it installed as an RPM, this package manager won't go get glib2 again, or anything. Another way that could be dealt with is searching for files a package installs, but that'd be too messy.

[Jack2000]

nice work pP

[piratePenguin]

nice work pP

I haven't actually done anything yet. Just thinking. And now I wanna implement this - but I needa learn quite a bit more programming first (actually I don't, and I probably won't (but I will be looking at code from XMMS and other random programs that are extensible), which means it's gonna be messy at first. I have no experience with this stuff.).

And I just might abandon it half way too, but hopefully not

[Master of Reality]

Sounds a bit like the Gentoo portage system

[piratePenguin]

Sounds a bit like the Gentoo portage system

Yep it would be, a bit. I've never used Gentoo, but I have used FreeBSD. What's the difference between their port collection stuff?

In my package management system, you'd just ask the frontend to get you e.g. irssi, and it'll go get it. A server would be contacted and the XML files downloaded and parsed, dependencies solved, configure flags sorted out (usually off the user), anything else that needs to be done, and finally it'll be compiled and installed. The XML files won't (normally) be stored on the local filesystem like they are in FreeBSD and I think Gentoo, they'd be on a server and downloaded as required.

[Master of Reality]

I've never really used Gentoo either, but i think im switching to it soon. Its package system is inspired by FreeBSD and the ports system. So its fairly close. According to the gentoo site the only thing it is lacking at all in is with binary packages. They are starting to support more, but it prabably does not have as much support as the BSD package system yet.

[piratePenguin]

According to the gentoo site the only thing it is lacking at all in is with binary packages.

I've had an idea for binary packages, but I haven't thought about it much. Any configure flags used, and patches applied, and packages required etc., would be stored in a file and saved in a tarball with the binaries/etc. A frontend (which could be a web-based interface) could poke into the tarballs and give the information to the user, and sort out the runtime dependences needed.